Backend Software Engineer (Evals)

Stack

Responsibilities

• ▹Design eval pipelines that are reliable, reproducible, and extendable
• ▹Build the infrastructure for continuous eval monitoring frameworks (regression/drift monitoring, robust golden datasets) along with feedback loops that strengthen support automation
• ▹Design, build, and maintain backend services and APIs to support intelligent automation and knowledge systems
• ▹Integrate and structure data across internal platforms, transforming it into formats optimized for downstream systems and AI workflows
• ▹Collaborate closely with data, research, and engineering teams to integrate OpenAI models into high-leverage workflows
• ▹Own the full development lifecycle of new backend systems and internal platform capabilities
• ▹Build with scale and maintainability in mind while rapidly iterating on new ideas

Requirements

• ▹4+ years of backend engineering experience at product-driven companies (excluding internships)
• ▹Proficiency in backend technologies — the stack includes Python, FastAPI, and Postgres
• ▹Experience designing and scaling distributed systems, APIs, or data processing pipelines
• ▹Experience building AI agents or applications, including designing evals and improving performance through prompting or scaffolding
• ▹Familiarity with evaluation methods for LLMs, having worked with patterns like multi-agent workflows, tool use, or long context
• ▹Experience creating production evals and/or measuring performance of ML/LLM models at scale
• ▹A pragmatic mindset, comfortable shipping iteratively while building toward a long-term vision

Soft skills

About the company

The Support Automation team applies cutting-edge AI models to real-world challenges, automating and enhancing work across OpenAI — from customer operations to engineering. They build reusable solutions that create patterns applicable across diverse domains within OpenAI.