Senior AI Engineer | US | Remote

Stack

Responsibilities

• ▹Own end-to-end development of multi-agent AI systems, from architecture through testing, deployment, and ongoing operation.
• ▹Build modular, composable agentic systems using orchestration frameworks (LangChain, CrewAI, Anthropic MCP, or similar).
• ▹Develop reusable agentic skills that agents invoke across interfaces (Slack, dashboards, internal apps, CLIs).
• ▹Implement observability and feedback loops: logging, performance metrics, prompt iteration, model evaluation, and cost management.
• ▹Establish governance and compliance standards for AI workflows (access controls, audit trails, PII handling, human-in-the-loop escalation).
• ▹Build MCP servers, APIs, CLIs, and microservices connecting AI models to business systems (BigQuery, Slack, CRMs, email, calendars, analytics).
• ▹Architect data flows for retrieval-augmented generation (RAG) connecting LLMs to internal knowledge and real-time business context.
• ▹Build serverless or containerized services (GCP Cloud Functions, Cloud Run) that scale with usage.
• ▹Partner with RevOps, Demand Generation, Regional Marketing, and SDR teams to scope and build high-impact automation.
• ▹Design and deploy workflows using orchestration tools (n8n, Workato, or custom) with CI/CD, testing, and production reliability.

Requirements

• ▹8+ years of software engineering experience with depth in backend development, systems integration, or data/analytics engineering.
• ▹2+ years hands-on applying LLMs/AI to production workflows, not just prototypes.
• ▹Strong proficiency in Python and JavaScript/Node.js with Git-based workflows, code review, and testing discipline.
• ▹Hands-on experience with LLM frameworks and patterns: prompt engineering, RAG, function calling/tool use, structured output parsing, and evaluation.
• ▹Experience building and operating multi-agent systems at scale (decomposition, orchestration patterns, state management, monitoring).
• ▹Ability to diagnose business problems before writing code; thinking in workflows and outcomes.
• ▹Deep familiarity with Google Cloud Platform, BigQuery, and serverless/containerized services (Cloud Functions, Cloud Run).
• ▹Understanding of LLM failure modes and production mitigations (confidence thresholds, fallback logic, human escalation, cost/latency management).
• ▹Proven ability to identify high-leverage problems and deliver end to end with minimal direction.
• ▹Fluent with AI-assisted development tools (GitHub Copilot, Cursor, Claude Code).

Nice to have

• ▹Experience with vector databases or retrieval pipelines (Pinecone, Weaviate, ChromaDB, Qdrant, pgvector).
• ▹Familiarity with marketing or sales platforms (Salesforce, Customer.io, HubSpot, Marketo, Outreach).
• ▹Experience with frontend frameworks (React, Slack Block Kit) for AI tool interfaces.
• ▹Observability tooling for AI systems (LangSmith, Weights & Biases, custom evaluation frameworks).
• ▹Experience with workflow orchestration platforms (n8n, Temporal, Prefect, Airflow).
• ▹Familiarity with Model Context Protocol (MCP) or similar standards.
• ▹Prior work automating marketing, sales, or customer success workflows in B2B SaaS.
• ▹Active in open-source communities.

Soft skills

What we offer

• ▹Base compensation of $154,445–$185,334 (US), varying by level, experience, and skillset.
• ▹Restricted Stock Units (RSUs) for every team member.
• ▹100% remote, global culture.
• ▹30 days of annual leave per year (including Grafana Shutdown Days).
• ▹In-person onboarding with fellow new hires.
• ▹Career growth pathways and access to AI coding assistants (Claude Code, Gemini CLI, OpenAI Codex).

About the company

Grafana Labs, the company behind the open observability cloud, is founded on open source, open standards, open ecosystems, and open culture. More than 35 million users and 7,000+ customers — including Anthropic, Bloomberg, NVIDIA, Microsoft, and Salesforce — trust Grafana Labs. It is a 100% remote company with 1,600+ team members across 40+ countries.