Senior Research Scientist, Model Evaluation

Stack

Responsibilities

• ▹Create ambitious new evaluation benchmarks that push the limits of what models can accomplish
• ▹Work on highly cross-functional teams to translate model feedback into trustworthy, repeatable evaluations
• ▹Conduct research to advance the state-of-the-art in LLM evaluation methods, including training LLM judges, refining LLM-based data synthesis pipelines and improving evaluation efficiency
• ▹Build scalable and reusable tools for digging into model performance

Requirements

• ▹Enjoy rapidly building prototypes that demonstrate the boundaries of LLM capabilities and developing resources to measure them
• ▹Have spent dozens of hours reviewing complex data and LLM outputs to ensure high data quality
• ▹Obsessive about rigorously measuring AI capabilities and ensuring measurements align with the capabilities you care about
• ▹Strong software engineering skills

Soft skills

What we offer

• ▹Weekly lunch stipend of $75/£75 (or local equivalent)
• ▹Full health and dental benefits, including a separate mental health budget
• ▹RRSP matching, 401K, Pension Scheme
• ▹100% Parental Leave top-up for up to 6 months for either parent
• ▹Annual enrichment benefits and an education & learning stipend
• ▹6 weeks of paid vacation (30 working days)
• ▹Travel budget to other offices for remote staff plus an annual company offsite
• ▹$500 home office stipend; co-working benefit for those not near an office