Software Engineer, ML Serving

Stack

Responsibilities

• ▹Architect and implement Rime's TTS serving infrastructure, from GPU-backed inference engines to the API surface
• ▹Optimize models from single-node to disaggregated fleet serving
• ▹Ensure compatibility across NVIDIA hardware generations (Hopper to Blackwell and beyond) for on-prem and cloud deployments
• ▹Build CI/CD workflows for the model serving pipeline
• ▹Own site reliability: on-call rotation, monitoring, alerting, and observability across the serving stack
• ▹Manage resource provisioning and cost across the GPU fleet

Requirements

• ▹Hands-on experience with real-time, multi-node ML serving infrastructure
• ▹Experience with an ML serving framework: NVIDIA Dynamo/Triton, vLLM, SGLang, or equivalent
• ▹Experience with distributed or disaggregated model serving (Tensor Parallel, Pipeline Parallel, or equivalent)
• ▹Strong cloud infrastructure fundamentals: Linux internals, networking, containerization (Docker, Kubernetes)
• ▹IaC experience: Terraform, Packer, or comparable
• ▹Comfort with on-call and shared responsibility for production reliability

Nice to have

• ▹Experience with multi-node training (DDP, FSDP, etc.)
• ▹Experience with gRPC or other bidirectional binary streaming protocols
• ▹Experience with audio streaming (WebRTC, WebSockets, etc.)
• ▹Experience with a multilingual monorepo
• ▹Experience with multi-cloud infrastructure (AWS, GCP, OCI, etc.)
• ▹Comfort with configuration management tooling (Ansible, Chef, Puppet, or similar)
• ▹SRE, DevOps, or platform engineering background at a startup
• ▹Experience at an early-stage company

Soft skills

What we offer

• ▹Build the serving infrastructure behind a category-defining voice AI company from the ground up
• ▹Direct collaboration with inference, platform, and ML teams
• ▹Meaningful equity upside at an early stage
• ▹SF / Bay Area location

About the company

Rime builds voice AI for enterprises running high-volume customer experiences, with its own studio-quality speech corpus annotated by linguists.