Data Platform Engineer (Azure to Databricks Migration)

Stack

Responsibilities

• ▹Convert Azure Data Factory pipelines into Databricks workflows on AWS, building reusable templates rather than migrating one at a time
• ▹Rehost Databricks workspaces onto AWS and migrate ADLS Gen2 storage to S3
• ▹Rewrite ADF Web Activities as Lambda functions or Step Functions tasks, replacing ADF-specific scaling with native Databricks mechanisms
• ▹Build and tune PySpark transformations for production data volumes
• ▹Replace Azure Synapse Serverless with Databricks SQL Warehouse
• ▹Reconcile migrated data against source systems as part of the definition of done

Requirements

• ▹Production experience with Databricks: workspaces, jobs and workflows — the central skill for this role
• ▹Strong Spark and PySpark experience for real data volumes, including tuning
• ▹Production-grade Python
• ▹Experience building or migrating Azure Data Factory pipelines, with solid understanding of the ADF activity model
• ▹AWS data services: S3, Glue, Athena, Lambda and Step Functions
• ▹Advanced SQL, including reading and reasoning about stored procedures
• ▹Professional written and spoken English

Nice to have

• ▹Delta Lake
• ▹Unity Catalog
• ▹Azure Synapse
• ▹Terraform
• ▹Airflow/MWAA
• ▹dbt
• ▹Kafka
• ▹Databricks certification
• ▹Data modeling
• ▹Great Expectations
• ▹SAS/analytics platform integration
• ▹CRM data

What we offer

• ▹Full-time, 100% remote
• ▹Open to candidates from all of LATAM