Product Lead, AI/ML (Evals)

Stack

Responsibilities

• ▹Drive product strategy and execution for the evals platform
• ▹Own the roadmap across the eval lifecycle and own outcomes against it
• ▹Build the shared measurement infrastructure that lets any pod define quality, run experiments, compare models and watch production
• ▹Own the standards for LLM judges, rule-based evaluators, human annotation and online monitoring, and be clear about where each belongs
• ▹Make model selection fast and routine: a repeatable way to evaluate a new frontier model within days of release and a defensible framework for when a post-trained specialty model is worth it over a prompted frontier one
• ▹Keep the eval system model-agnostic so it stays a neutral referee
• ▹Own the operating model across pods: eval gates from early build through GA and steady-state monitoring, which gates are hard versus advisory, and who owns non-negotiable floors like critical-error rates
• ▹Work with Data Engineering on the de-identification pipeline, Data Science on bootstrapping judge quality with less human annotation, Clinical Science on flagged production cases, and the agent platform team as workflows go agentic

Requirements

• ▹5+ years of product management experience with significant ownership of ML-powered products or platform systems
• ▹Deep understanding of how to measure and improve model quality, including evaluation frameworks, annotation pipelines and benchmark design
• ▹Strong technical fluency across ML, data pipelines and distributed systems
• ▹Experience working closely with ML researchers and engineers to drive impact in production
• ▹Ability to balance long-term architectural investments with near-term quality improvements
• ▹Strong communication skills and the ability to translate complex technical concepts into clear decisions and narratives
• ▹A track record of delivering high-quality products in domains where accuracy, reliability and trust are paramount
• ▹Ability to land solutions across pods you do not own, without formal authority

Nice to have

• ▹Experience building evaluation platforms, ML observability systems or quality measurement pipelines
• ▹Experience in clinical, healthcare or regulated environments with a high bar for accuracy and compliance
• ▹Experience with specialty-specific or domain-specific model adaptations
• ▹Experience with personalization systems, context ingestion frameworks or ambient intelligence products
• ▹Experience shipping large-scale ML products with human-in-the-loop workflows

Soft skills

What we offer

• ▹14 paid holidays and flexible PTO for salaried employees
• ▹Medical, dental and vision coverage for full-time employees and their families
• ▹Monthly company HSA contributions with a High Deductible Health Plan
• ▹Generous paid parental leave
• ▹Family forming benefits
• ▹401(k) matching
• ▹Tax-free personal device allowance
• ▹Pre-tax benefits (FSA and commuter benefits)
• ▹Monthly Lifestyle Wallet for fitness, professional development, coworking and more
• ▹Mental health support: therapy and coaching
• ▹Paid sabbatical leave after 5 years
• ▹Competitive compensation and equity grants for full-time employees

About the company

Abridge was founded in 2018 to power deeper understanding in healthcare. Its AI platform turns patient-clinician conversations into structured clinical notes in real time, with deep EMR integrations. It has offices in San Francisco, New York and Pittsburgh.