User Researcher, AI Evaluations

Stack

Responsibilities

• ▹Establish clear, reusable evaluation frameworks and scoring guidance (helpfulness, trust, tone, control, transparency)
• ▹Run recurring longitudinal and feature-specific studies to measure experience quality over time
• ▹Anchor evaluations in real workflows (goal-setting, delegation, review, iteration) rather than isolated feedback
• ▹Identify failure modes and recovery behavior across the system
• ▹Operationalize evaluation with Product, Design, Engineering and Data Science, building scalable feedback loops

Requirements

• ▹5+ years of industry UX research experience
• ▹Ability to turn soft user expectations (trust, tone, usefulness) into concrete rubrics and metrics
• ▹AI fluency and systems thinking, with experience evaluating AI products (LLMs, agents)
• ▹Clear communication, connecting research to business outcomes
• ▹Strong quantitative and qualitative UX research craft (interviews, benchmarking, surveys, experiments)
• ▹Pragmatism in fast-moving environments, ruthless prioritization

Nice to have

• ▹Familiarity with LLM-as-judge methods, evaluator prompt design or golden dataset creation
• ▹Experience with AI research tooling (e.g. Dovetail, Listen Labs, Maze, Outset) and AI observability tooling (e.g. Braintrust)
• ▹Experience with data querying languages (SQL), scripting languages (Python) or statistical software (R, SAS, Matlab)
• ▹Master's or PhD in HCI, Psychology, Behavioral Science, Anthropology or Sociology

Soft skills

What we offer

• ▹Highly competitive cash compensation, equity and benefits

About the company

Notion is the collaborative AI workspace where teams and agents think together, bringing knowledge, projects, meetings and AI tools into one place. Millions of individuals, small teams and large companies run their work on Notion.