Staff Software Engineer, Inference Infrastructure

Stack

Responsibilities

• ▹Deploy optimized NLP models to production in low-latency, high-throughput, high-availability environments
• ▹Work closely with many teams to operate the AI platform delivering Cohere's large language models through API endpoints
• ▹Interface with customers and create customized deployments to meet their specific needs

Requirements

• ▹5+ years of engineering experience running production infrastructure at scale
• ▹Experience designing large, highly available distributed systems with Kubernetes and GPU workloads
• ▹Experience with Kubernetes development, production coding and support
• ▹Experience with GCP, Azure, AWS, OCI or multi-cloud/on-prem/hybrid serving
• ▹Experience designing, deploying, supporting and troubleshooting complex Linux-based computing environments
• ▹Experience in compute/storage/network resource and cost management
• ▹Excellent collaboration and troubleshooting skills to build mission-critical systems

Nice to have

• ▹Familiarity with the computational characteristics of accelerators (GPUs, TPUs, custom accelerators) and how they affect inference latency and throughput
• ▹Strong understanding or working experience with distributed systems
• ▹Experience in Golang, C++ or other languages designed for high-performance, scalable servers

Soft skills

What we offer

• ▹Weekly lunch stipend ($75/£75 or equivalent in local currency)
• ▹Full health and dental benefits, including a separate mental health budget
• ▹RRSP matching, 401K, pension scheme
• ▹100% parental leave top-up for up to 6 months, for either parent
• ▹Annual enrichment benefits (arts & culture, fitness/wellness, workspace improvement)
• ▹Education and learning stipend for conferences and courses
• ▹6 weeks of paid vacation (30 working days)
• ▹Travel budget to other offices for remote employees, plus an annual company offsite
• ▹$500 home office stipend

About the company

Cohere is a leading security-first enterprise AI company building cutting-edge foundation AI models and end-to-end products that solve real-world business problems for enterprises building AI systems. The team spans researchers, engineers and designers, co-headquartered in Toronto and San Francisco, with offices in London, New York City, Montreal, Seoul, Germany and Paris.