Senior Machine Learning Engineer - Orchestration

Stack

Responsibilities

• ▹Define and own the evaluation strategy for the AI agentic system: metrics, protocols, datasets and tooling.
• ▹Implement and maintain automated evaluation pipelines to monitor model quality, safety and alignment across iterations.
• ▹Run systematic experiments to assess reasoning, factuality, robustness and user experience.
• ▹Collaborate closely with model developers and research scientists to provide insights and drive iterative improvement.
• ▹Contribute to research and internal knowledge sharing on LLM evaluation methodologies and best practices.

Requirements

• ▹MSc or PhD in Computer Science, Machine Learning, Data Science or a related field.
• ▹7+ years of hands-on experience working with large language models (e.g. GPT, Claude, Llama, or BERT-like architectures).
• ▹Proven experience evaluating agentic or reasoning systems (autonomous agents, tool-using LLMs, dialogue systems, task-oriented assistants).
• ▹Strong track record in experiment design, metric definition and evaluation automation.
• ▹Ability to bridge research and production, influencing modeling and product decisions.
• ▹Excellent communication skills and a collaborative mindset.

Nice to have

• ▹Experience in the clinical or medical domain and sensitivity to ethical or regulatory challenges in healthcare AI.

Soft skills

What we offer

• ▹Free health insurance for you and your children.
• ▹Parent Care Program: one additional month of leave on top of legal parental leave.
• ▹Free mental health and coaching services via Moka.care.
• ▹For caregivers and workers with disabilities: adapted remote policy, extra days off for medical reasons, and psychological support.
• ▹Work from EU countries and the UK up to 10 days per year (flexibility days policy).
• ▹Works council subsidy toward sport club or creative class membership.
• ▹Up to 14 days of RTT.
• ▹Lunch voucher with Swile card.

About the company

Doctolib is on a mission to transform how healthcare is delivered by harnessing the power of AI, building an AI Health Companion for millions of patients and practitioners. Its platform is fully cloud-native with a stack including Rails, TypeScript, Java, Python, Kotlin, Swift and React Native. Permanent, full-time, hybrid position in Levallois.