Senior Software Engineer, AI Evals

Stack

Responsibilities

• ▹Design and build robust evaluation frameworks to measure accuracy, reliability, regressions, and edge cases in AI systems
• ▹Create and curate high-quality datasets, golden test cases, and benchmarks grounded in real production data
• ▹Build automated test harnesses and metrics pipelines to continuously evaluate models, prompts, and agentic workflows
• ▹Partner closely with applied AI engineers and product leaders to define what 'good' looks like and translate it into measurable criteria
• ▹Own the evaluation lifecycle for major AI initiatives, from early experimentation through production monitoring

Requirements

• ▹Minimum 5+ years of professional experience with a Bachelor's degree in computer science, machine learning, or a related field
• ▹Experience building testing, evaluation, or data infrastructure for complex systems (AI/ML experience strongly preferred)
• ▹Comfort writing production-quality code (Python and TypeScript)
• ▹Experience working with structured and unstructured datasets, labeling workflows, or data quality pipelines
• ▹Familiarity with modern ML systems and evaluation techniques (e.g., offline metrics, online evaluation, regression testing for models or prompts)

Nice to have

• ▹Experience evaluating LLMs, agentic systems, or AI-assisted developer tools

Soft skills

What we offer

• ▹Incentive compensation
• ▹Equity grants
• ▹Paid time off
• ▹Group health insurance coverage

About the company

Sentry's application monitoring platform is trusted by 200,000+ organizations to help developers spot and fix errors and performance issues before users notice — the team is building its AI-native future.