Big Data Architect

Stack

Responsibilities

• ▹Design and define end-to-end architectures for large-scale data processing, aligning technology strategy with business goals and laying the groundwork for scalable, high-performance ETL/ELT pipelines
• ▹Define the technology stack and implementation standards, leading architectural decisions
• ▹Establish storage strategies, data formats (Parquet, Delta Lake, Apache Iceberg) and access patterns across multi-cloud environments (AWS, Azure, GCP)
• ▹Lead the design of advanced architectures (Data Lakehouse, Data Mesh, Modern Data Warehouse), defining the roadmap for the corporate data platform
• ▹Act as technical leader with multidisciplinary teams and executive stakeholders
• ▹Evaluate, select and drive adoption of emerging technologies (Delta Live Tables, dbt Cloud, Apache Iceberg, Snowflake)
• ▹Establish quality, observability and governance frameworks through DataOps, data quality frameworks and CI/CD

Requirements

• ▹Degree in Computer Engineering, Telecommunications, Mathematics, Physics or similar
• ▹At least 8 years of experience with distributed processing and ETL/ELT pipelines in complex environments, using Python or Scala, Apache Spark, and platforms like Airflow, Databricks or Snowflake
• ▹Knowledge of data architectures (Data Lakehouse, cloud) and formats/storage (Parquet, Delta Lake, Iceberg, Hudi)
• ▹Good practices and code management: Git, CI/CD, DataOps, pipeline monitoring, data quality control and data versioning
• ▹English proficiency to work in global environments and collaborate with international teams

Nice to have

• ▹Experience with real-time streaming projects

Soft skills

About the company

SDG Group has been a global leader in Big Data, Business Intelligence, Analytics and AI since 1994, supporting half of the IBEX-35 companies and other leading international firms through digital transformation, with strategic alliances with Microsoft, Google and AWS.