Member of Technical Staff (Data Scientist, Evals)

Stack

Responsibilities

• ▹Architect and maintain automated evaluation pipelines to assess answer quality across Perplexity's products
• ▹Design evaluation sets and methods specifically to measure the impact of tool calls (particularly web search retrieval) on final answer quality
• ▹Develop VLM-based solutions to programmatically evaluate how final answers render visually across platforms and devices
• ▹Continuously review public benchmarks and academic evaluations for applicability, adapting them into regular performance measurement
• ▹Operate in a small, high-impact team where evaluation metrics directly shape product changes, working closely with technical leadership

Requirements

• ▹PhD or MS in a technical field or equivalent experience
• ▹4+ years of experience in data science or machine learning
• ▹Strong proficiency in Python and SQL (production-grade code expected)
• ▹Experience building within a modern cloud data stack, specifically AWS and Databricks
• ▹Comfortable with agentic coding workflows and AI-assisted development tools

Nice to have

• ▹1+ years of experience working with LLMs at scale, specifically LLM-as-a-judge setups
• ▹Prior experience on customer-facing web products or consumer apps with real user traffic at scale
• ▹Strong research background applying research methods to real-world ML problems
• ▹Experience defining evaluation metrics (e.g., factual consistency, hallucination rate, retrieval precision) and building ground truth datasets

Soft skills

About the company

Perplexity serves tens of millions of users daily with reliable answers from its LLM-first search engine and specialized data sources, aiming to use the latest models even where public benchmarks don't cover its use cases.