Machine Learning Infrastructure Engineer, Model Inference

Stack

Responsibilities

• ▹Design, deploy and maintain scalable Kubernetes clusters for AI model inference and training
• ▹Develop, optimize and maintain ML model serving infrastructure with high performance and low latency
• ▹Collaborate with ML and product teams to scale backend infrastructure for AI-driven products, focusing on model deployment, throughput optimization and compute efficiency
• ▹Optimize compute-heavy workflows and improve GPU utilization for ML workloads
• ▹Build a robust model API orchestration system
• ▹Work with leadership to define and implement strategies for scaling infrastructure as the company grows

Requirements

• ▹2+ years of experience building and deploying machine learning models in production environments
• ▹Deep understanding of container orchestration and distributed systems architecture
• ▹Expertise in Kubernetes administration, including custom resource definitions, operators and cluster management
• ▹Experience developing APIs and managing distributed systems for both batch and real-time workloads
• ▹Excellent communication skills, with the ability to interface between research and product engineering

Nice to have

• ▹Expertise with model serving frameworks such as NVIDIA Triton Server, vLLM and TRT-LLM
• ▹Expertise with ML toolchains such as PyTorch, TensorFlow or distributed training and inference libraries
• ▹Familiarity with GPU cluster management and CUDA optimization
• ▹Knowledge of infrastructure as code (Terraform, Ansible) and GitOps practices
• ▹Experience with container registries, image optimization and multi-stage builds for ML workloads
• ▹Experience orchestrating across ASR or LLM models to build GenAI applications

Soft skills

What we offer

• ▹14 paid holidays and flexible PTO for salaried employees
• ▹Medical, dental and vision coverage for full-time employees and their families
• ▹Monthly company HSA contributions with a High Deductible Health Plan
• ▹Generous paid parental leave
• ▹Family forming benefits
• ▹401(k) matching
• ▹Tax-free personal device allowance
• ▹Pre-tax benefits (FSA and commuter benefits)
• ▹Monthly Lifestyle Wallet for fitness, professional development, coworking and more
• ▹Mental health support: therapy and coaching
• ▹Paid sabbatical leave after 5 years
• ▹Competitive compensation and equity grants for full-time employees

About the company

Abridge was founded in 2018 to power deeper understanding in healthcare. Its AI platform turns patient-clinician conversations into structured clinical notes in real time, with deep EMR integrations. It has offices in San Francisco, New York and Pittsburgh.