Senior Data Engineer - Databricks

Stack

Responsibilities

• ▹Own Databricks production support for the data platform, including monitoring, alerting and incident response across all production data flows
• ▹Maintain and report on SLA performance metrics for data pipeline delivery
• ▹Identify and implement pipeline optimizations that reduce Databricks compute costs, improve throughput and reduce processing windows, tracked through measurable KPIs
• ▹Migrate legacy ETL/ELT pipelines to Databricks and build automation tooling to reduce manual intervention
• ▹Support new customer onboarding by provisioning, validating and hardening tenant data pipelines
• ▹Design and build high-performance Databricks pipelines that ingest, transform and serve ERP and CRM data at scale across Azure and AWS
• ▹Own the Delta Lake architecture: schema design, partitioning strategies, data quality enforcement and incremental processing patterns
• ▹Enforce data security best practices across Databricks environments, including role-based access control, secrets management and compliance requirements
• ▹Implement data quality monitoring and observability across pipeline health and ML model inputs
• ▹Apply and enforce multi-tenant data isolation patterns
• ▹Partner with the Enterprise Architecture team to integrate data pipelines with the broader AI and analytics ecosystem
• ▹Support a globally distributed operation through on-call rotation and after-hours incident response
• ▹Maintain technical documentation, runbooks and architectural decision records
• ▹Apply CI/CD best practices to data pipeline development: version control, automated testing and deployment tooling

Requirements

• ▹4+ years of data engineering experience
• ▹At least 2 years of experience with Databricks or the Apache Spark ecosystem on Azure and/or AWS
• ▹Proficiency in PySpark, SQL and Python with a strong track record of building and operating production-grade pipelines under SLA constraints
• ▹Hands-on experience with Delta Lake: schema evolution, ACID transactions, optimize/vacuum lifecycle, incremental and streaming processing patterns
• ▹Hands-on experience with pipeline performance tuning and compute optimization in production Databricks environments
• ▹Solid working knowledge of PostgreSQL: query optimization, schema design, use as a source or sink in production pipelines
• ▹Experience supporting and maintaining legacy ETL tooling (SSIS, Informatica, custom Python/SQL pipelines or similar) in production
• ▹Experience supporting large-scale multi-tenant architectures with a focus on tenant isolation, per-tenant performance and data privacy
• ▹Proven ability to work across data science, product and infrastructure teams, owning end-to-end delivery
• ▹Strong understanding of data governance, security and compliance principles

Nice to have

• ▹Experience operating Databricks workspaces across Azure and AWS, including cost governance, cluster management and cross-cloud data access
• ▹Experience optimizing Databricks workloads in a Serverless environment
• ▹Experience with Microsoft SQL Server in a data engineering or ETL context
• ▹Exposure to ML feature engineering or feature stores (Databricks Feature Store, Feast or similar)
• ▹Experience with customer onboarding automation or IaC patterns for provisioning tenant data pipelines at scale
• ▹Databricks Certified Data Engineer Associate or Professional certification

Soft skills

About the company

Intetics is a global technology company specializing in custom software development, AI-powered solutions, cloud technologies and digital transformation, with over 30 years of experience.