Software Engineer- BIS (Baseten Inference Stack)

Stack

Responsibilities

• ▹Develop infrastructure and orchestration systems for deploying and managing large-scale distributed LLM inference
• ▹Work across the stack, from customer-facing features to low-level infrastructure components
• ▹Build platform capabilities related to routing, autoscaling, scheduling, observability, and runtime management
• ▹Improve the reliability, scalability, and usability of the inference stack
• ▹Collaborate closely with Model Performance engineers to make new inference optimizations broadly available and easy to configure
• ▹Help define best practices around testing, release automation, benchmarking, and operational excellence
• ▹Debug complex production systems spanning Kubernetes, distributed runtimes, networking, and GPU workloads
• ▹Make thoughtful engineering tradeoffs balancing performance, reliability, operational simplicity, and developer experience
• ▹Own projects end-to-end: from architecture and implementation through deployment, monitoring, and iteration

Requirements

• ▹Bachelor's, Master's, or Ph.D. in Computer Science, Engineering, or a related field
• ▹Strong background in distributed systems, backend infrastructure, or platform engineering
• ▹Experience building and operating production systems where reliability, latency, and scale are first-class concerns
• ▹Strong sense of developer experience — thinking about how systems are used, not just how they work
• ▹Motivated and willing to learn new languages, frameworks, and systems as needed
• ▹Ability to debug complex systems across multiple layers of the stack
• ▹Genuine interest in inference engineering, with willingness to learn even without prior hands-on experience
• ▹Excellent communication and collaboration skills

Nice to have

• ▹Experience with Kubernetes, including concepts like operators and custom resources
• ▹Prior work on Dynamo, vLLM, SGLang, TensorRT-LLM, or similar inference frameworks
• ▹Experience with distributed scheduling, autoscaling, or service orchestration
• ▹Experience operating GPU workloads in production
• ▹Familiarity with observability tooling, CI/CD systems, or release automation
• ▹Experience contributing to open-source infrastructure or ML systems

Soft skills

What we offer

• ▹Competitive compensation, including meaningful equity
• ▹100% coverage of medical, dental, and vision insurance for employee and dependents
• ▹Flexible PTO policy, including a company-wide Winter Break (offices closed from Christmas Eve to New Year's Day)
• ▹Paid parental leave
• ▹Fertility and family-building stipend through Carrot
• ▹Company-facilitated 401(k)
• ▹Exposure to a variety of ML startups, offering unparalleled learning and networking opportunities

About the company

Baseten powers mission-critical inference for some of the world's most dynamic AI companies, including Cursor, Notion, OpenEvidence, Abridge, Clay, Gamma and Writer. By uniting applied AI research, flexible infrastructure, and seamless developer tooling, Baseten helps companies operating at the frontier of AI bring cutting-edge models into production. The company is growing quickly and recently closed a $1.5B Series F round led by Altimeter Capital, Conviction Partners, and Spark Capital.