← Back to list
Job · Principal

Principal Data Scientist - Agent Builder

Elastic
Data Engineer • Principal • On-site • Full-time • 📍 Portugal
📄 View original listing (EN)

Senior applied data science leadership role at Elastic's Search Conversational Experiences team, building the quality layer for the company's conversational and agentic search platform. You will set the technical direction for evaluating, improving, and scaling chat quality across RAG, agents, tools, retrieval, and citations on Elasticsearch data. The role spans evaluation strategy, prototyping, roadmap influence, and mentoring.

Stack

AI/MLdata scienceelasticsearchPandasPythonPyTorch

Responsibilities

  • ▹Define the evaluation strategy for conversational and agentic search (offline/online evaluation, golden datasets, rubrics, LLM-as-judge calibration, groundedness and citation checks, A/B testing)
  • ▹Lead the design of quality metrics and decision frameworks for RAG, agents, tools, model selection, agent routing, prompt behavior, and cost/latency trade-offs
  • ▹Build, compare, and guide improvements across retrieval and re-ranking (sparse and dense retrieval, vector search, query understanding, semantic rewrites, context enrichment)
  • ▹Turn experimental results into product and business decisions on models, request routing, exposed tools, and agent customization
  • ▹Partner with engineering to productionize evaluation pipelines, telemetry, dashboards, CI guardrails, and regression detection
  • ▹Mentor data scientists and engineers in experiment design, evaluation methodology, and statistical rigor

Requirements

  • ▹Deep expertise in LLM evaluation methodology and statistical rigor
  • ▹Experience with RAG, agents, tools, retrieval and re-ranking (sparse/dense retrieval, vector search)
  • ▹Ability to design quality metrics and decision frameworks for LLM-powered systems
  • ▹Experience productionizing evaluation pipelines, telemetry, and regression detection
  • ▹Strong stakeholder communication across product, engineering, and leadership

Nice to have

  • ▹Experience with multimodality in conversational systems
  • ▹Background in agent-driven visualizations on top of data platforms
  • ▹Experience with reasoning strategies and tool-selection policies

Soft skills

Technical leadership and direction-settingClear communication of trade-offsMentoring and coachingComfort with ambiguous, cutting-edge problems
Languages: angol