Staff ML Performance Engineer (Training Efficiency)

Stack

Responsibilities

• ▹Profile ML workloads to identify their bottlenecks, e.g. using NVIDIA Nsight Systems
• ▹Design and implement efficiency improvements to maximize MFU and throughput, e.g. parallelism, model compilation, mixed precision
• ▹Design and implement observability tools to identify bottlenecks and drive performance improvements
• ▹Design and implement benchmarking tools to track efficiency gains or regressions
• ▹Collaborate closely with Research teams to integrate training efficiency improvements and create a culture of performance optimization

Requirements

• ▹10+ years of industry experience driving performance engineering across ML systems, GPU compute infrastructure, distributed platforms, or similar
• ▹Experience optimizing large-scale jobs on GPU compute clusters
• ▹Experience working in platform teams and with research teams
• ▹Experience writing, reporting, and tracking performance benchmarks in an open and accessible way
• ▹Ability to write high-quality, well-structured and tested Python code
• ▹BS or MS in Machine Learning, Computer Science, Engineering, or a related technical discipline, or equivalent experience

Nice to have

• ▹Experience working with concurrent, parallel and distributed computing
• ▹Experience using NVIDIA Nsight Systems or other system profilers
• ▹Experience implementing GPU kernels (CUDA, Triton, etc.)
• ▹Knowledge of computing fundamentals — what makes code fast, secure and reliable

Soft skills

What we offer

• ▹Competitive equity package
• ▹Hybrid working policy based out of the Sunnyvale office

About the company

Wayve's Training Tech team is responsible for optimizing large-scale ML training jobs to enable faster model scaling.