Software Engineer - Model Performance

Stack

Responsibilities

• ▹Implement, refine, and productionize cutting-edge inference techniques (quantization, speculative decoding, KV cache reuse, chunked prefill, LoRA)
• ▹Deep-dive into the codebases of TensorRT, PyTorch, TensorRT-LLM, vLLM, SGLang, CUDA, and other libraries to debug ML performance issues
• ▹Apply and scale optimization techniques across a wide range of ML models, particularly large language models
• ▹Collaborate with a diverse team to design and implement innovative solutions
• ▹Own projects from idea to production

Requirements

• ▹Bachelor's, Master's, or Ph.D. in Computer Science, Engineering, Mathematics, or a related field
• ▹Experience with a general-purpose programming language such as Python or C++
• ▹Familiarity with LLM optimization techniques (e.g. quantization, speculative decoding, continuous batching)
• ▹Strong familiarity with ML libraries, especially PyTorch, TensorRT, or TensorRT-LLM
• ▹Demonstrated interest and experience in LLMs
• ▹Deep understanding of GPU architecture

Nice to have

• ▹Proficiency enhancing the performance of software systems, particularly in the context of LLMs
• ▹Experience with CUDA or similar technologies
• ▹Deep understanding of software engineering principles with a track record of developing and deploying AI/ML inference solutions
• ▹Experience with Docker and Kubernetes

Soft skills

What we offer

• ▹Competitive compensation, including meaningful equity
• ▹100% coverage of medical, dental, and vision insurance for employee and dependents
• ▹Flexible PTO policy, including a company-wide Winter Break (offices closed from Christmas Eve to New Year's Day)
• ▹Paid parental leave
• ▹Fertility and family-building stipend through Carrot
• ▹Company-facilitated 401(k)
• ▹Exposure to a variety of ML startups, offering unparalleled learning and networking opportunities

About the company

Baseten powers mission-critical inference for some of the world's most dynamic AI companies, including Cursor, Notion, OpenEvidence, Abridge, Clay, Gamma and Writer. By uniting applied AI research, flexible infrastructure, and seamless developer tooling, Baseten helps companies operating at the frontier of AI bring cutting-edge models into production. The company is growing quickly and recently closed a $1.5B Series F round led by Altimeter Capital, Conviction Partners, and Spark Capital.