← Back to list

Job
· Principal
Staff AI Engineer - Grafana AI/ML | Canada | Remote
AI / ML Engineer
• Principal
• Remote
• Full-time
•
Canada
Grafana Labs' AI team is hiring a Staff AI Engineer to build LLM- and GenAI-powered features that help users make sense of observability data and detect, triage and resolve incidents.
Responsibilities
- ▹Develop high-performance AI features to help users detect, triage and resolve incidents using observability data
- ▹Rapidly prototype, test and validate with real users in an iterative process, including shipping and evolving LLM- or agent-powered workflows
- ▹Collaborate cross-functionally with data analysts, product managers and designers to shape AI-driven product features
- ▹Use AI and automation tools to enhance both product functionality and personal development workflow
- ▹Take full ownership of developed AI solutions, ensuring they are scalable, maintainable and aligned with real user workflows
Requirements
- ▹Strong engineering skills building production software systems (backend and/or full-stack)
- ▹Experience with LLMs, prompt engineering, and building applications powered by GenAI
- ▹Proven track record of delivering software into production that is actively used
- ▹Exposure to cloud-native environments (e.g. AWS, GCP, Azure)
- ▹Using observability tools to understand and troubleshoot system behavior
Nice to have
- ▹Experience building or working with agent frameworks or multi-agent workflows
- ▹Familiarity with infrastructure/DevOps tooling: Kubernetes, Docker, Terraform or similar
- ▹Familiarity with model fine-tuning techniques
- ▹Experience building observability tooling
Soft skills
Self-driven, proactive attitudeExperimental, fast-iteration mindsetCollaborative approach with designers and product managersOpenness to feedbackComfort with ambiguity, ability to define scope
What we offer
- ▹Restricted Stock Units (RSUs)
- ▹Company-funded budget for AI tools (e.g. GPT-Codex, Claude Opus, Gemini Pro)
- ▹100% remote work in a global team spanning 40+ countries
- ▹30 days of annual leave, including 3 shared Grafana Shutdown Days
About the company
Grafana Labs is the company behind the open observability cloud, building Grafana Cloud. It serves more than 35 million users and 7,000+ customers – including Anthropic, Bloomberg, NVIDIA, Microsoft and Salesforce – with 1,600+ team members across 40+ countries, fully remote.
Similar jobs

Job
· Principal
Principal AI Engineer
PointClickCare
Istio
+9
€119,811–€133,123/yr
gross
🏢 On-site
Mississauga
🗣️ EN

Job
· Principal
Principal AI Engineer
PointClickCare
AI/MLBicep
+4
€119,811–€133,123/yr
gross
🏢 On-site
Mississauga
🗣️ EN

Job
· Principal
Principal Engineer, AI/ML Systems (Context Services)
Cognite - AI for Industry
AI/MLCppData Science
+11
💰 Salary: not specified
🏢 On-site
India (Bengaluru)
🗣️ EN
Job
· Principal
Principal Machine Learning Engineer, Foundation Models
Cambridge Mobile Telematics
AI/MLData Science
+10
$177,000–$221,300/yr
gross
🏢 On-site
Cambridge
🗣️ EN

Job
· Principal
Staff AI Engineer
Acquia, Inc.
Langchain
+2
💰 Salary: not specified
🌍 Remote
🗣️ EN

Job
· Principal
Staff AI Engineer
Acquia
Langchain
+2
💰 Salary: not specified
🌍 Remote
🗣️ EN