Software Engineer, Agent Evaluation and Quality

Responsibilities

• ▹Designing and building a best-in-class AI evaluation system: curated datasets, offline replay, scorers/judges, regression alerts, dashboards
• ▹Designing feedback loops from real usage: collecting, cleaning and interpreting user signals to inform model and harness changes
• ▹Developing analysis tooling and workflows for debugging agent behavior: deep dives on failure modes, clustering themes, surfacing actionable insights
• ▹Improving reliability and guardrails by making quality measurable and operational: defining "good/bad/degraded" sessions, alerting, triage primitives

Requirements

• ▹Experience building and operating evaluation or measurement systems (e.g. AI evals, experimentation, ranking/relevance, search quality)
• ▹Ability to turn ambiguous "quality" questions into concrete metrics, pipelines and decisions
• ▹Strong data acumen and effective collaboration with data scientists and researchers
• ▹Strong software engineering fundamentals, enjoy shipping production systems

Nice to have

• ▹Taste and strong opinions on model and agent behaviors
• ▹Staying up to date with emerging research and industry trends

Soft skills

About the company

Cursor's mission is to automate coding. The organization is very flat, with a small, talent-dense team building the best tool for professional programmers by combining inventive research, design and engineering.