Staff Software Engineer - Distributed Data Systems

Stack

Responsibilities

• ▹Develop Apache Spark, the de facto open source standard framework for big data
• ▹Data Plane Storage: deliver reliable, high-performance services and client libraries for storing and accessing huge amounts of data on cloud storage backends (AWS S3, Azure Blob Store)
• ▹Delta Lake: build a storage management system combining data-warehouse performance and reliability, data-lake scale and cost-efficiency, and streaming-level low latency, with ACID transactions and time travel
• ▹Delta Pipelines: build higher-level abstractions to orchestrate and operate tens of thousands of data pipelines
• ▹Performance Engineering: build the next-generation query optimizer and execution engine that's fast, tuning-free, scalable, and robust

Requirements

• ▹BS in Computer Science, related technical field, or equivalent practical experience (optionally MS or PhD in databases or distributed systems)
• ▹Comfortable working towards a multi-year vision with incremental deliverables
• ▹8+ years of production-level experience in Java, Scala, or C++
• ▹Strong foundation in algorithms and data structures with real-world use cases
• ▹Experience with distributed systems, databases, and big data systems (Apache Spark, Hadoop)

Soft skills

What we offer

• ▹Eligibility for annual performance bonus
• ▹Equity

About the company

Databricks is the data and AI company, founded in 2013 by the original creators of Apache Spark; more than 10,000 organizations, including over 50% of the Fortune 500, rely on the Databricks Data Intelligence Platform.