Senior Data Engineer

Stack

Responsibilities

• ▹Design, build and maintain scalable data pipelines, ETL/ELT workflows and data models using Python, Apache Spark (PySpark), Databricks, dbt, SQL (PostgreSQL) and AWS Glue
• ▹Develop and optimise AWS-native data platforms (Glue, EMR, MWAA, Lambda, Step Functions, S3, Redshift, RDS, DMS, CloudWatch)
• ▹Build high-performance ingestion, transformation and orchestration workflows for structured and semi-structured data (Apache Iceberg, Parquet, ORC, Avro)
• ▹Design and optimise analytical data platforms (Athena, Trino, Hive, OpenSearch, data catalog technologies)
• ▹Integrate enterprise and external data sources across relational and NoSQL platforms (PostgreSQL, Oracle, Redshift, GraphDB)
• ▹Build AI-enabled data solutions using Amazon Bedrock, RAG pipelines and vector search (Amazon S3 Vector, OpenSearch vector indexes)
• ▹Develop cloud infrastructure with CloudFormation (Infrastructure as Code), GitHub, Harness and enterprise CI/CD pipelines; build event-driven architectures with SNS, SQS and EventBridge
• ▹Improve the reliability, scalability, performance and maintainability of enterprise data platforms through monitoring, troubleshooting and automation
• ▹Support mission-critical analytics and reporting solutions in large-scale federal AWS environments, complying with FedRAMP and NIST 800-53 controls
• ▹Lead modernisation initiatives migrating legacy platforms (IBM DataStage, Hadoop, RunDeck, shell-based workflows) to cloud-native AWS services
• ▹Mentor junior engineers through technical guidance, architecture discussions and code reviews
• ▹Collaborate with cross-functional teams in an Agile environment

Requirements

• ▹US citizenship and the ability to obtain a Public Trust clearance
• ▹Bachelor's degree in Computer Science, Engineering or a related technical field
• ▹9+ years of professional experience in data engineering or related domains
• ▹Strong hands-on experience with Databricks, Apache Spark (PySpark), Python, SQL (PostgreSQL) and dbt
• ▹Designing, building and maintaining AWS-native data platforms (Glue, EMR, MWAA, Lambda, Step Functions, S3, Redshift, RDS, DMS, CloudWatch)
• ▹Scalable data pipelines, workflow orchestration and data integration across enterprise environments
• ▹Modern data lake technologies (Apache Iceberg) and data formats (Parquet, ORC, Avro)
• ▹Relational and NoSQL databases (PostgreSQL, Redshift, Oracle, GraphDB)
• ▹Performance tuning and enterprise-scale ETL/ELT architectures
• ▹Java development and modern CI/CD practices using Harness
• ▹Strong analytical and problem-solving skills
• ▹Experience in agile, iterative software development environments
• ▹Ability to quickly learn new technologies and domain knowledge
• ▹Excellent written and verbal communication skills

Nice to have

• ▹Experience supporting analytics, data engineering or modernisation initiatives for financial regulators, capital markets or other highly regulated environments
• ▹Experience with Kafka (streaming/data pipelines)
• ▹Docker and Kubernetes for containerisation and orchestration
• ▹Splunk for log aggregation and system monitoring
• ▹Terraform for infrastructure automation
• ▹Strong SQL skills, including performance tuning and complex query design

Soft skills

What we offer

• ▹Remote work
• ▹Estimated annual salary range: $130k-$165k
• ▹Medical, dental and vision insurance
• ▹Life insurance, short/long-term disability
• ▹Employee assistance program
• ▹401(k) with 4% matching
• ▹Liberal PTO policy
• ▹Generous annual continuing education budget
• ▹Annual wellness budget
• ▹Bonus incentive programs (employee referrals and performance-based rewards)

About the company

Capital Technology Group provides consulting in software development, digital transformation, human-centered design, data analytics and cybersecurity for federal and commercial clients, using agile methods.