Senior ML Systems Engineer, Frameworks & Tooling

Stack

Responsibilities

• ▹Build and own the training framework responsible for large-scale LLM training
• ▹Design distributed training abstractions (data/tensor/pipeline parallelism, FSDP/ZeRO strategies, memory management, checkpointing)
• ▹Improve training throughput and stability on multi-node clusters (e.g. GB200/300, AMD, H200/100)
• ▹Develop and maintain tooling for monitoring, logging, debugging and developer ergonomics
• ▹Collaborate closely with infra teams to ensure cluster, container environments and hardware configurations support high-performance training
• ▹Investigate and resolve performance bottlenecks across the ML systems stack
• ▹Build robust systems that ensure reproducible, debuggable, large-scale runs

Requirements

• ▹Strong engineering experience in large-scale distributed training or HPC systems
• ▹Deep familiarity with JAX internals, distributed training libraries or custom kernels/fused ops
• ▹Experience with multi-node cluster orchestration (Slurm, Ray, Kubernetes or similar)
• ▹Comfort debugging performance issues across CUDA/NCCL, networking, IO and data pipelines
• ▹Experience working with containerized environments (Docker, Singularity/Apptainer)
• ▹A track record of building tools that increase developer velocity for ML teams
• ▹Excellent judgment around trade-offs: performance vs complexity, research velocity vs maintainability

Nice to have

• ▹Experience with training LLMs or other large transformer architectures
• ▹Contributions to ML frameworks (PyTorch, JAX, DeepSpeed, Megatron, xFormers, etc.)
• ▹Familiarity with evaluation and serving frameworks (vLLM, TensorRT-LLM, custom KV caches)
• ▹Experience with data pipeline optimization, sharded datasets or caching strategies
• ▹Background in performance engineering or profiling
• ▹Paper at a top-tier venue (such as NeurIPS, ICML, ICLR, AIStats, MLSys, JMLR, AAAI, Nature, COLING, ACL, EMNLP)

Soft skills

What we offer

• ▹Weekly lunch stipend ($75/£75 or equivalent in local currency)
• ▹Full health and dental benefits, including a separate mental health budget
• ▹RRSP matching, 401K, pension scheme
• ▹100% parental leave top-up for up to 6 months, for either parent
• ▹Annual enrichment benefits (arts & culture, fitness/wellness, workspace improvement)
• ▹Education and learning stipend for conferences and courses
• ▹6 weeks of paid vacation (30 working days)
• ▹Travel budget to other offices for remote employees, plus an annual company offsite
• ▹$500 home office stipend

About the company

Cohere is a leading security-first enterprise AI company building cutting-edge foundation AI models and end-to-end products that solve real-world business problems for enterprises building AI systems. The team spans researchers, engineers and designers, co-headquartered in Toronto and San Francisco, with offices in London, New York City, Montreal, Seoul, Germany and Paris.