Senior Databricks Data Engineer

Stack

Responsibilities

• ▹Design and implement complex ETL/ELT data flows using Apache Spark (PySpark, Spark SQL, Scala)
• ▹Manage the Data Lakehouse: optimize Delta Lake tables and implement partitioning/indexing logic to maximize read/write performance
• ▹Configure data governance and security, including access control via Unity Catalog, ensuring compliance with company policies
• ▹Collaborate with Data Scientists on integrating machine learning models (MLflow) and with Power BI developers on exposing optimized datasets (Gold Layer)
• ▹Automate workflows via Databricks Jobs, integrate with CI/CD tools, and monitor data quality
• ▹Coordinate workload optimization with Data Scientists and Data Architects
• ▹In complex scenarios, lead the design of the Medallion architecture (Bronze, Silver, Gold) and Delta Lake management, resolving technical bottlenecks

Requirements

• ▹Knowledge of Apache Spark
• ▹Knowledge of the Hadoop ecosystem
• ▹Scala and Python programming skills
• ▹At least 2 years of experience with Microsoft Azure cloud services and technologies, designing and managing cloud-native data architectures
• ▹Proven expertise in batch and streaming data ingestion, with a focus on performance and quality

What we offer

• ▹Dynamic work environment with national and international projects
• ▹Continuous learning and skills-development opportunities
• ▹Opportunity to work at a leading, fast-growing European IT services company
• ▹Smart-working option per company policy

About the company

Sopra Steria is one of Europe's leading technology companies, with 51,000 employees in about 30 countries, supporting clients' digital transformation through consulting, digital services and IT solutions. It posted revenue of €5.6 billion in 2025. In Italy, it employs around 1,200 people (2025 revenue: €107.35 million) across offices in Assago, Rome, Collecchio, Padua, Ariano Irpino and Pozzuoli. The company is listed on Euronext Paris.