Software Engineer — GPU Networking & Distributed Systems

Stack

Responsibilities

• ▹Integrate RDMA/RoCE/InfiniBand capabilities directly into the inference stack, moving beyond TCP/IP
• ▹Optimize distributed inference: tune networking for disaggregated KV cache offload and WideEP across NVLink and InfiniBand for MoE models
• ▹Enable serverless-grade startup speeds for LLMs: checkpointing and storage mechanisms for sub-10-second startup of trillion-parameter models
• ▹Deep-dive into hardware: characterize and validate networking performance on bleeding-edge clusters (H100/H200, B200/B300, GB200/300 NVL72)
• ▹Build observability tools to visualize packet flow, congestion and effective bandwidth across GPU interconnects
• ▹Optimize kernels using communication libraries (NCCL, NVSHMEM), potentially writing custom communication kernels

Requirements

• ▹Deep experience with high-performance networking protocols (InfiniBand, RoCE v2) and understanding of the physics of data movement
• ▹Fluency in C++ or Python, bridging high-level logic and hardware
• ▹Deep understanding of the memory hierarchy in modern NVIDIA architectures (H100/Blackwell)
• ▹You like going deep: comfortable diving into TensorRT-LLM source code, writing custom C++/Python bindings, debugging NVLink topology issues
• ▹Judgment on when to use off-the-shelf solutions vs. building custom ones

Nice to have

• ▹Deep knowledge of NCCL, NVSHMEM and UCX
• ▹Experience with GPUDirect Storage (GDS) or high-performance filesystems like Weka or 3FS
• ▹Familiarity with TensorRT-LLM, vLLM or SGLang
• ▹Experience running low-level benchmarks to qualify new hardware clusters

Soft skills

What we offer

• ▹Competitive compensation, including meaningful equity
• ▹100% coverage of medical, dental and vision insurance for employee and dependents
• ▹Flexible PTO policy including company-wide Winter Break (offices closed from Christmas Eve to New Year's Day)
• ▹Paid parental leave
• ▹Fertility and family-building stipend through Carrot
• ▹Company-facilitated 401(k)
• ▹Exposure to a variety of ML startups, offering unparalleled learning and networking opportunities

About the company

Baseten powers mission-critical inference for the world's most dynamic AI companies, including Cursor, Notion, OpenEvidence, Abridge, Clay, Gamma and Writer. By combining applied AI research, flexible infrastructure and seamless developer tooling, Baseten helps companies at the frontier of AI bring cutting-edge models into production. The company is growing quickly and recently closed a $1.5B Series F round led by Altimeter Capital, Conviction Partners and Spark Capital.