Azure Senior Data Engineer

Stack

Responsibilities

• ▹Develop reusable, metadata-driven data pipelines using Databricks Lakehouse architecture and PySpark
• ▹Design and implement comprehensive data models
• ▹Build robust ETL/ELT solutions with advanced features such as Merge operations and SCD Type 2 implementations
• ▹Implement incremental data loads with idempotency patterns and overlap joins optimization
• ▹Automate and optimize data platform processes with focus on performance and reliability
• ▹Build integrations with data sources and consumers using event-driven patterns
• ▹Cooperate with the infrastructure engineering team to set up cloud resources
• ▹Initiate and implement improvements to the data platform architecture

Requirements

• ▹Databricks expertise: Lakehouse architecture and PySpark development
• ▹Data modeling mastery: fact and dimension tables, measures, grain analysis, SCD Type 2, surrogate keys, late-arriving records handling, overlap joins, incremental loads with idempotency
• ▹Advanced Python and PySpark skills for ETL/ELT development
• ▹Deep knowledge of Databricks optimization: optimize, zOrder, Liquid clustering, ACID transactions, performance tuning
• ▹Proven experience designing and implementing event-driven data solutions
• ▹Experience with Azure-based datasets and data pipelines
• ▹Strong SQL skills for complex data transformations
• ▹Experience handling large and complex datasets efficiently
• ▹Experience developing automated deployment pipelines (CI/CD)
• ▹Understanding of basic networking concepts
• ▹Familiarity with Scrum and agile development practices
• ▹Based in Poland

Nice to have

• ▹Understanding of stream processing challenges and Spark Structured Streaming
• ▹Experience with Infrastructure as Code (Terraform, Bicep)
• ▹Experience with containerized applications (Azure Container Apps, Kubernetes)
• ▹Knowledge of Azure cloud native solutions (Azure Data Factory, Azure Function App, Azure Container Instances)

Soft skills

What we offer

• ▹Flexible employment and remote work
• ▹International projects with leading global clients
• ▹International business trips
• ▹Non-corporate atmosphere
• ▹Language classes
• ▹Internal and external training
• ▹Private healthcare and insurance
• ▹Multisport card
• ▹Well-being initiatives

About the company

Software Mind develops software solutions for companies around the globe. The client provides risk management and talent screening solutions to over 33,000 clients, headquartered in Atlanta with about 5,500 employees.