Senior Data Engineer (Databricks, Azure)

Stack

Responsibilities

• ▹Build and manage Databricks Workflows for large-scale data orchestration
• ▹Develop and deploy PyFunc models, integrate into production pipelines
• ▹Implement secrets and configuration management using Azure Key Vault
• ▹Automate data flows using Azure Logic Apps
• ▹Design and optimize ETL/ELT pipelines
• ▹Govern data assets using Unity Catalog and Data Lakehouse patterns
• ▹Implement messaging and event-driven workflows with Azure Service Bus
• ▹Collaborate with architects, analysts, and data scientists
• ▹Utilize Azure DevOps for CI/CD pipeline management

Requirements

• ▹Databricks Workflows, Unity Catalog, Delta Lake experience
• ▹ETL/ELT design and implementation
• ▹Data Lakehouse architecture knowledge
• ▹Strong Python skills for data engineering, automation, and integration
• ▹Experience writing modular, production-ready code
• ▹CI/CD pipelines for Databricks and Azure
• ▹Strong SQL knowledge (query optimization, data modeling)
• ▹Exposure to NoSQL databases (e.g. Cosmos DB)
• ▹Azure Key Vault, Logic Apps, Service Bus, APIM experience

Nice to have

• ▹Experience with Microsoft Cognitive Services (OCR, Layout, Form Recognizer)
• ▹Integration of cognitive services into data pipelines
• ▹Familiarity with event-driven and microservices architecture patterns
• ▹Experience with monitoring tools like Datadog, Application Insights
• ▹Agile team collaboration, DevOps practices

About the company

Ascent, recently acquired by Acuity Analytics, delivers deep insights and domain-led digital transformation to high-growth and heavily regulated organizations.