User Researcher, AI Evaluations

Stack

Responsibilities

• ▹Define clear, reusable evaluation criteria and rubrics (helpfulness, trust, tone, control, transparency)
• ▹Run recurring longitudinal and feature-specific studies to measure experience quality over time
• ▹Lead qualitative studies, side-by-side comparisons, and human-in-the-loop evaluations
• ▹Anchor evaluations in real workflows (goal setting, delegation, review, iteration) rather than isolated feedback
• ▹Identify failure modes and recovery behavior, turning insights into actionable guardrail guidance
• ▹Partner with Product, Design, Engineering, and Data Science to build scalable evaluation loops (human-in-the-loop review, longitudinal studies, calibrating automated/LLM-judge approaches)

Requirements

• ▹Ability to turn "soft" user expectations (trust, tone, usefulness, clarity) into concrete rubrics and metrics
• ▹AI fluency and systems thinking; experience evaluating AI-enabled products (LLMs, agents, generative UI/workflow automation)
• ▹Clear communication, ability to connect research to business outcomes
• ▹Strong UX research craft (quant + qual: interviews, benchmarking, surveys, experiments)
• ▹Pragmatism, comfort working through ambiguity
• ▹5+ years doing UX research in industry

Nice to have

• ▹Familiarity with LLM-as-judge methods, prompt design for evaluators, or golden dataset creation
• ▹Experience with AI research tooling (e.g. Dovetail, Listen Labs, Maze, Outset) and AI observability tooling (e.g. Braintrust)
• ▹Experience with data querying languages (e.g. SQL), scripting languages (e.g. Python), or statistical software (e.g. R, SAS, Matlab)
• ▹Master's or PhD in HCI, Psychology, Behavioral Science, Anthropology, or Sociology

Soft skills

What we offer

• ▹Competitive cash compensation, equity, and benefits
• ▹In-office collaboration 3 days a week (San Francisco or New York City, anchor days)

About the company

Notion is the collaborative AI workspace where teams and agents think together — one place for knowledge, projects, meetings, and AI tools.