Data Engineer

Stack

Responsibilities

• ▹Design and build a scalable data mesh platform on AWS
• ▹Build stream processing backend using Spark (Scala/Python) and Kafka
• ▹Ingest new data sources via REST APIs and file sharing
• ▹Handle ever-changing schemas and ensure resilient pipelines
• ▹Act as a consultant to clients and the team

Requirements

• ▹Solid Scala and/or Python skills, deep language internals knowledge
• ▹2+ years hands-on experience with AWS (Lambda, EMR, Glue, Lake Formation)
• ▹Experience with pub-sub/async systems (Kafka, SQS, Kinesis, RabbitMQ)
• ▹Understanding of algorithm complexity and data structures
• ▹3+ years using Linux/macOS, Docker and git in collaborative workflows
• ▹Excellent communication skills for a consultant role

Nice to have

• ▹Experience with ZIO
• ▹5+ years in live, high-availability production environments
• ▹Familiarity with Great Expectations or Deequ
• ▹Writing clean, testable, maintainable code

Soft skills

About the company

DataChef is a data engineering consultancy that also contributes to open-source projects such as ZIO and Great Expectations.