Data Engineer

Stack

Responsibilities

• ▹Design, build, and maintain batch or real-time data pipelines in production
• ▹Maintain and optimize data infrastructure for accurate ETL
• ▹Develop ETL processes to extract and manipulate data from multiple sources
• ▹Automate data workflows (ingestion, aggregation, ETL)
• ▹Prepare raw data in warehouses into consumable datasets for stakeholders
• ▹Partner with data scientists and functional leaders to deploy ML models in production
• ▹Build, maintain, and deploy data products on cloud platforms (AWS, Azure, GCP)
• ▹Ensure data accuracy, integrity, privacy, security, and compliance
• ▹Monitor data systems performance and implement optimizations

Requirements

• ▹3+ years of SQL and relational database design experience
• ▹Experience with cloud data warehouse solutions (Databricks, Apache Spark)
• ▹Experience with data ingestion tools (Fivetran, Stitch, Matillion)
• ▹Working knowledge of cloud solutions (AWS, Azure, GCP)
• ▹Experience building and deploying ML models in production
• ▹Strong proficiency in Python, Java, C++, or Scala
• ▹Strong proficiency in scripting languages like Bash
• ▹Strong proficiency in workflow management tools (e.g., Airflow)
• ▹Strong project management and organizational skills

Nice to have

• ▹Familiarity with NoSQL databases (CrateDB, Redis, Cassandra, MongoDB, Neo4j)
• ▹Experience with large datasets and distributed computing (Hive/Hadoop/Spark/Presto/MapReduce)

Soft skills