Senior Data Engineer (Databricks Migration)

Stack

Responsibilities

• ▹Participate in migrating a large-scale analytical platform from BigQuery to Databricks
• ▹Design and implement scalable Lakehouse architectures using Databricks and Delta Lake
• ▹Analyze existing ETL/ELT workloads and define migration approaches
• ▹Develop and optimize data pipelines processing large volumes of retail and analytical data
• ▹Implement incremental processing strategies and scalable transformation frameworks
• ▹Build and maintain Spark-based (PySpark) data processing solutions
• ▹Design and maintain medallion architecture layers (Bronze, Silver, Gold)
• ▹Implement data governance and security best practices using Unity Catalog
• ▹Collaborate with Data Science, Analytics, Product and Customer Engineering teams
• ▹Participate in architecture discussions and technical solution design
• ▹Develop reusable data platform components and engineering standards
• ▹Conduct code reviews and contribute to platform reliability
• ▹Troubleshoot and optimize complex SQL and Spark workloads
• ▹Support production deployments and platform modernization

Requirements

• ▹5+ years of professional experience as a Data Engineer
• ▹Strong programming skills in Python and advanced SQL
• ▹Hands-on commercial experience with Databricks
• ▹Strong knowledge of Apache Spark, primarily PySpark
• ▹Experience designing and building modern cloud-based data platforms
• ▹Experience developing ETL/ELT pipelines and large-scale data processing
• ▹Hands-on experience with Delta Lake
• ▹Experience with Spark Declarative Pipelines
• ▹Experience with cluster monitoring, metrics analysis and performance optimization
• ▹Strong understanding of distributed data processing architectures
• ▹Solid understanding of data warehousing concepts and dimensional modeling
• ▹Experience with Airflow or similar orchestration tools
• ▹Experience optimizing complex analytical SQL workloads
• ▹Experience implementing CI/CD practices for data engineering platforms
• ▹Strong troubleshooting and performance optimization skills
• ▹Ability to work in cross-functional international teams
• ▹Upper-intermediate or higher English level

Nice to have

• ▹Experience with GCP cloud services
• ▹Experience with AWS or Azure cloud platforms
• ▹Experience in retail analytics or pricing optimization
• ▹Experience supporting machine learning or AI-related data workloads
• ▹Experience with platform modernization and cloud migration initiatives

Soft skills

About the company

Sigma Software's Data Engineering Center of Excellence is modernizing an enterprise-scale analytics ecosystem for the retail domain, working with international teams.