Lead Data Engineer

Role Overview

• Lead the design, development and deployment of secure, scalable data pipelines and data platform components.
• Design and implement batch and real-time streaming solutions using modern event-driven architectures.
• Own technical design decisions across projects, balancing scalability, security, performance and delivery requirements.
• Lead technical workshops, discovery sessions and architecture discussions with client stakeholders.
• Assess existing client data platforms and design practical approaches for modernisation, migration and integration.
• Build and maintain data ingestion, transformation and integration capabilities across a variety of data sources.
• Develop cloud-native data platforms using AWS services and open-source technologies.
• Ensure data quality, governance, lineage and security requirements are embedded throughout the data lifecycle.
• Drive engineering best practices including code reviews, automated testing, CI/CD and Infrastructure as Code.
• Support multiple client engagements, helping delivery teams maintain consistent engineering quality and standards.
• Mentor and support engineers across project teams, providing technical guidance and leadership.
• Perform data modelling, performance tuning and root cause analysis to resolve complex data and platform issues.

Requirements

• Strong commercial experience designing and implementing real-time streaming data platforms.
• Hands-on experience with technologies such as Apache Kafka, AWS Kinesis, Apache Flink, Spark Streaming, Apache Pulsar, or similar event-driven technologies.
• Experience building high-volume, low-latency, resilient data pipelines in production environments.
• Strong understanding of modern data engineering patterns, including batch, streaming and hybrid architectures.
• Extensive experience developing and operating enterprise-scale data platforms.
• Strong SQL skills and experience designing data models for analytical and operational workloads.
• Proficiency in Python or other modern programming languages used for data engineering.
• Experience configuring, tuning and optimising relational and NoSQL databases.
• Strong experience working with AWS cloud services, including EC2, RDS, Lambda, Athena, Glue, S3 and IAM.
• Experience working within Agile delivery environments.
• Strong understanding of Test-Driven Development (TDD) and automated testing practices.
• Experience implementing CI/CD pipelines and engineering automation.
• Strong understanding of Infrastructure as Code principles.
• Experience using version control systems such as Git.
• Experience delivering technology solutions within UK Public Sector, Utilities, Energy, Water, Financial Services or other highly regulated industries.
• Understanding of security, governance and compliance considerations within regulated environments.
• Experience with Terraform.
• Experience with Docker and Kubernetes.
• Knowledge of Ansible and Packer.
• Linux and Windows administration experience.
• Experience supporting AI, machine learning or advanced analytics platforms.
• Experience working on programmes requiring UK Government security clearance.
• Strong technical leadership and problem-solving skills.
• Ability to communicate effectively with both technical and non-technical stakeholders.
• Experience working in complex, regulated environments.
• A collaborative approach and willingness to support and mentor others.
• A focus on delivering practical solutions that create value for clients.
• A passion for building reliable, scalable and secure data platforms.

Tech Stack

• Ansible
• Apache
• AWS
• Cloud
• Docker
• EC2
• Kafka
• Kubernetes
• Linux
• NoSQL
• Packer
• Pulsar
• Python
• Spark
• SQL
• Terraform

Benefits

• 25 days holiday plus bank holidays
• company paid medical insurance
• life assurance
• pension scheme
• annual training allowance
• wellbeing allowance
• virtual GP
• Employee Assistance plan
• more.