Senior Research Engineer, Model Evaluation

Responsibilities

• ▹Develop evaluation benchmarks, datasets and environments for measuring the bleeding edge of model capabilities
• ▹Conduct research to push the state-of-the-art in LLM evaluation methods, including training LLM judges, improving evaluation efficiency and scalably building high-quality datasets
• ▹Build scalable tools for investigating and understanding evaluation results used by all technical staff, leadership and the CEO
• ▹Learn from and work with the best researchers and engineers in the field

Requirements

• ▹Enjoy pushing the limits of LLM capabilities and have built high-quality evaluation resources (datasets, simulators, environments, etc.)
• ▹A track record of developing new methods and/or data to evaluate LLMs, e.g. publications at top-tier conferences or popular benchmarks
• ▹Deep experience building with and around LLMs and tools for analyzing their performance
• ▹Strong software engineering skills

Soft skills

What we offer

• ▹Weekly lunch stipend of $75/£75 (or local equivalent)
• ▹Full health and dental benefits, including a separate mental health budget
• ▹RRSP matching, 401K, Pension Scheme
• ▹100% Parental Leave top-up for up to 6 months for either parent
• ▹Annual enrichment benefits and an education & learning stipend
• ▹6 weeks of paid vacation (30 working days)
• ▹Travel budget to other offices for remote staff plus an annual company offsite
• ▹$500 home office stipend; co-working benefit for those not near an office