Senior Data Engineer – AWS Data Lake & Pipeline Architecture

Stack

Responsibilities

• ▹Participate in the architecture design and implementation of high-performance, scalable and optimized data solutions
• ▹Create data models from scratch using strong SQL fundamentals
• ▹Write and optimize in-application SQL statements
• ▹Ensure the performance, security and availability of databases
• ▹Prepare documentation and technical specifications
• ▹Handle database procedures such as upgrades, backups, recovery and migration
• ▹Profile server resource usage and optimize configurations
• ▹Design, build and automate the deployment of data pipelines and applications
• ▹Integrate data from on-premise databases and external sources using REST APIs and harvesting tools
• ▹Collaborate with business units and data science teams on data access, transformation, processing and reporting
• ▹Support implementation, technical issues and training related to the data lake ecosystem
• ▹Manage AWS resources, including EMR and ECS clusters
• ▹Evaluate and promote new cloud technologies that improve capabilities and lower operating costs
• ▹Support automation with Infrastructure as Code (Terraform) and CI/CD tools such as Jenkins
• ▹Implement data governance, access control and security risk reduction

Requirements

• ▹7-9 years of experience designing and developing cloud-based data models, ETL pipelines and infrastructure
• ▹Experience with structured and unstructured data
• ▹Strong SQL proficiency across popular databases, optimizing large, complex SQL statements
• ▹Knowledge of relational database best practices
• ▹Experience configuring database engines and orchestrating clusters
• ▹Ability to plan resource requirements from high-level specifications
• ▹Ability to troubleshoot common database issues
• ▹Experience with Spark, Glue, EMR and Apache Kafka or AWS Kinesis
• ▹Experience with Git or Subversion
• ▹Experience with automated build systems and CI/CD workflows
• ▹Programming in Java, Python and Scala
• ▹Knowledge of data structures and algorithms
• ▹Knowledge of relational, NoSQL, graph, document, key-value and time-series databases
• ▹Knowledge of scalable data model design and management
• ▹Knowledge of ML model deployment
• ▹Knowledge of AWS cloud platforms
• ▹Knowledge of TDD and BDD
• ▹Strong interest in improving software development skills, frameworks and technologies

Soft skills