Technical Lead, Spark

Stack

Responsibilities

• ▹Design new features for Cloudera's data engineering experience, taking them from prototype to leading a team delivering the feature in production at scale
• ▹Contribute to Apache Spark and Livy
• ▹Develop new features in Scala/Java/Python on modern platforms
• ▹Gain expertise in distributed data processing: SQL planners and optimizers, data layout, table formats like Apache Parquet and Iceberg, fault tolerance in distributed systems
• ▹Gain deep technical knowledge across the Cloudera Data Engineering Experience stack, with a focus on Iceberg and Spark
• ▹Work on large-scale distributed systems, from hundreds to thousands of nodes, in production clusters
• ▹Debug system-level deployment issues, perform root cause analysis, system test analysis and resolve failures
• ▹Improve internal infrastructure
• ▹Collaborate with team members and stakeholders

Requirements

• ▹BSc/MSc in a related field or equivalent experience
• ▹6+ years of professional software development experience
• ▹Experience leading and delivering complex product enhancements
• ▹Strong understanding of at least one of Java, Scala or Python, with interest in learning the others
• ▹Experience with systems design and development
• ▹Passion for programming, clean coding habits, attention to detail and focus on quality
• ▹Strong oral and written communication skills
• ▹Strong ability to research and solve problems independently
• ▹Open-minded with a desire to learn new things and build great products
• ▹Experience with distributed systems
• ▹Not eligible for immigration sponsorship or relocation

Nice to have

• ▹Experience with SQL planners
• ▹Experience using or developing Apache Spark, Livy or related technologies
• ▹Experience with large-scale, distributed systems design and development, including scaling, performance and scheduling
• ▹Solid experience with at least one cloud platform

Soft skills

About the company

Cloudera empowers companies to transform complex data into clear, actionable insights, managing as much data as the hyperscalers. Powered by relentless open source community innovation, Cloudera advances digital transformation for the world's largest enterprises. The team is split between the United States and Hungary, including multiple Apache Spark committers.