Staff Software Engineer, Model Infrastructure

Responsibilities

• ▹Lead the design and implementation of Harvey's Model Infrastructure platform
• ▹Build systems to ensure high availability, low latency, and operational excellence for AI inference
• ▹Design and improve the Unified Model Controller (UMC) and Model Selector platform to detect model degradations and intelligently route traffic
• ▹Develop systems for model provisioning, capacity management, failover, and traffic engineering across multiple AI providers
• ▹Integrate new model providers and maintain provider APIs and SDKs
• ▹Improve observability through health dashboards, alerting, token usage analytics, cost reporting, and end-to-end telemetry
• ▹Partner with Product Engineering to support model launches, experimentation, and production AI workload monitoring
• ▹Drive infrastructure efficiency through capacity planning, utilization optimization, and cost visibility
• ▹Collaborate with AI Research to build infrastructure for model evaluation, training, and deployment
• ▹Lead cross-functional technical initiatives and mentor engineers across the organization

Requirements

• ▹7+ years of software engineering experience building large-scale distributed systems
• ▹Experience designing and operating highly available production services
• ▹Strong programming skills in Go, Java, Python, Rust, or C++
• ▹Deep understanding of distributed systems, cloud infrastructure, networking, and observability
• ▹Experience leading technical projects across multiple engineering teams
• ▹Ability to balance long-term architecture with pragmatic execution
• ▹Strong communication and collaboration skills

Nice to have

• ▹Experience with AI infrastructure, LLM serving, or machine learning platforms
• ▹Experience with model routing, inference gateways, or policy-based serving systems
• ▹Experience with OpenAI, Anthropic, Azure OpenAI, Fireworks, Baseten, or open-source LLMs
• ▹Experience with Kubernetes, cloud infrastructure, and service mesh technologies
• ▹Experience with large-scale observability and SRE best practices
• ▹Experience with data infrastructure technologies such as Kafka, Spark, Flink, Airflow, or Iceberg
• ▹Familiarity with GPU infrastructure or model training platforms

Soft skills

About the company

Harvey builds an enterprise-grade, agentic AI platform for legal and professional services, serving 2,400+ customers in 70+ countries.