Grafana LabsRemote
Staff AI Engineer United States |Remote
United States of America
Full Time
4 weeks ago
$175,000 - $210,000 USD
H1B Sponsor Likely
Key skills
Large Language Models (LLMs)Multi-agent AI systemsPythonJavaScriptNode.jsGit workflowsPrompt engineeringRetrieval-augmented generation (RAG)Google Cloud Platform (GCP)BigQueryServerless computingContainerized servicesSalesforceCRM platformsWorkflow orchestration toolsAI-assisted development toolsAPI developmentMicroservicesObservability tools for AI systemsProblem diagnosisHigh-leverage problem identificationAILLMClaudeAnthropicRAGLangChainAgenticPineconeWeaviateAnalyticsReactGCPGoogle CloudServerlessCloud RunCloud FunctionsGrafanaGitGitHubSaaSCRMCI/CDCustomer SuccessSales
About this role
Grafana Labs is a remote-first, open-source powerhouse with over 20M users. They are seeking a Staff AI Engineer to own the AI agent infrastructure and automation platform, building multi-agent architectures and backend services that connect AI models to various data platforms.
Responsibilities:
- Own end-to-end development of multi-agent AI systems, from architecture and implementation through testing, deployment, and ongoing operation
- Build modular, composable agentic systems using orchestration frameworks (LangChain, CrewAI, Anthropic MCP, or similar) that operate 24/7 across teams
- Develop reusable agentic skills that agents invoke across interfaces (Slack, dashboards, internal apps, CLIs)
- Implement observability and feedback loops including logging, performance metrics, prompt iteration, model evaluation, and cost management
- Establish governance and compliance standards for AI workflows including access controls, audit trails, PII handling, and human-in-the-loop escalation paths
- Build MCP servers, APIs, CLIs, and microservices connecting AI models to business systems (BigQuery, Slack, Salesforce, email, calendars, analytics tools)
- Architect data flows for retrieval-augmented generation (RAG), connecting LLMs to internal knowledge bases, customer data, and real-time business context
- Build serverless or containerized services (GCP Cloud Functions, Cloud Run) that scale with usage and integrate with Grafana's cloud infrastructure
- Partner with RevOps, and Finance to build solutions with measurable business outcomes
- Design and deploy workflows using orchestration tools (n8n, Workato, or custom platforms) with CI/CD, testing, and production reliability standards
- Build systems designed for self-service with documentation, playbooks, and enablement materials that let partner teams operate independently
Requirements:
- 8+ years of software engineering experience with depth in backend development, systems integration, or data/analytics engineering
- 2+ years hands-on experience applying LLMs/AI to production workflows, not just prototypes
- Strong proficiency in Python and JavaScript/Node.js with Git-based workflows, code review practices, and testing discipline
- Hands-on experience with LLM frameworks and patterns including prompt engineering, RAG, function calling/tool use, structured output parsing, and evaluation
- Experience building and operating multi-agent systems at scale including agent decomposition, orchestration patterns (sequential chains, router/dispatcher, parallel fan-out), state management, and production monitoring
- You diagnose business problems before writing code. You think in workflows and outcomes, not just functions
- Deep familiarity with Google Cloud Platform, BigQuery, and serverless/containerized services (Cloud Functions, Cloud Run)
- Understanding of LLM failure modes and production mitigations including confidence thresholds, fallback logic, human escalation, and cost/latency management
- Proven ability to identify high-leverage problems, push back on low-impact requests, and deliver end-to-end with minimal direction
- Fluent with AI-assisted development tools (GitHub Copilot, Cursor, Claude Code). You use AI to build AI systems
- Clear technical communicator—you can explain complex systems in simple terms to both engineers and business stakeholders
- Familiarity with GTM platforms like Salesforce, HubSpot, Outreach, Gainsight, or similar CRM/sales engagement tools
- Experience with frontend frameworks & tooling (React, Slack Block Kit, dashboard components) to build user-facing interfaces for AI tools
- Experience with vector databases or retrieval pipelines (Pinecone, Weaviate, ChromaDB, pgvector, or similar)
- Prior work automating sales, customer success, or marketing workflows in a B2B SaaS environment
- Experience with workflow automation platforms like n8n, Prefect, Clay, PhantomBuster, Apify, Dust, or similar tools
- Familiarity with Model Context Protocol (MCP) or similar standards for connecting AI systems to data sources and tools
- Exposure to observability tools for AI systems (LangSmith, Weights & Biases, custom logging/evaluation frameworks)
- Experience working in Revenue Operations, GTM Analytics, or Sales Operations environments
- Previous experience in open source or developer-focused SaaS companies—Grafana is built on OSS and we value engineers who share that DNA