Data Engineer

Stack

Responsibilities

• ▹Maintain and evolve the existing data platform
• ▹Build processes to ingest data from Kafka, APIs, and databases using AWS MSK Connect
• ▹Develop and maintain real-time data processing applications using frameworks like Spark Structured Streaming and Kafka Streams
• ▹Apply transformations to data streams
• ▹Contribute to data modelling following standards such as Inmon, Kimball, and Data Vault
• ▹Ensure data quality by verifying data consistency and accuracy
• ▹Keep up to date on new technologies and techniques to enhance the data platform
• ▹Troubleshoot issues and support incident management with an investigative mindset
• ▹Take full ownership of assigned projects and tasks while working within a team
• ▹Document processes well and perform knowledge-sharing sessions

Requirements

• ▹Experience with modern cloud database technologies such as Snowflake
• ▹Experience orchestrating data pipelines using Airflow
• ▹Experience with AWS Glue
• ▹Experience with Apache Iceberg
• ▹Experience with SQL and data integration tools
• ▹Experience with a programming language such as Python or Scala
• ▹Knowledge of AWS services like S3/Lambda/API Gateway/DMS/RDS
• ▹Development experience in both Microsoft and Linux/Cloud environments

Nice to have

• ▹Familiarity with data warehousing concepts and modelling techniques such as Inmon, Kimball, Data Vault
• ▹Familiarity with AWS services such as MSK and EMR
• ▹Familiarity with containerization and orchestration technologies such as Docker and Kubernetes
• ▹Familiarity with time-series or analytics databases such as Elasticsearch
• ▹Familiarity with data streaming using technologies such as Kafka
• ▹Familiarity with GIT
• ▹Familiarity with infrastructure scripting like Terraform
• ▹Familiarity with data monitoring and visualization tools such as Prometheus, Grafana, CloudWatch
• ▹Scripting — PowerShell, Unix scripting
• ▹At least 2 years of experience in Data Engineering

Soft skills