Lead the design, development and deployment of secure, scalable data pipelines and data platform components.
Design and implement batch and real-time streaming solutions using modern event-driven architectures.
Own technical design decisions across projects, balancing scalability, security, performance and delivery requirements.
Lead technical workshops, discovery sessions and architecture discussions with client stakeholders.
Assess existing client data platforms and design practical approaches for modernisation, migration and integration.
Build and maintain data ingestion, transformation and integration capabilities across a variety of data sources.
Develop cloud-native data platforms using AWS services and open-source technologies.
Ensure data quality, governance, lineage and security requirements are embedded throughout the data lifecycle.
Drive engineering best practices including code reviews, automated testing, CI/CD and Infrastructure as Code.
Support multiple client engagements, helping delivery teams maintain consistent engineering quality and standards.
Mentor and support engineers across project teams, providing technical guidance and leadership.
Perform data modelling, performance tuning and root cause analysis to resolve complex data and platform issues.
Requirements
Strong commercial experience designing and implementing real-time streaming data platforms.
Hands-on experience with technologies such as Apache Kafka, AWS Kinesis, Apache Flink, Spark Streaming, Apache Pulsar, or similar event-driven technologies.
Experience building high-volume, low-latency, resilient data pipelines in production environments.
Strong understanding of modern data engineering patterns, including batch, streaming and hybrid architectures.
Extensive experience developing and operating enterprise-scale data platforms.
Strong SQL skills and experience designing data models for analytical and operational workloads.
Proficiency in Python or other modern programming languages used for data engineering.
Experience configuring, tuning and optimising relational and NoSQL databases.
Strong experience working with AWS cloud services, including EC2, RDS, Lambda, Athena, Glue, S3 and IAM.
Experience working within Agile delivery environments.
Strong understanding of Test-Driven Development (TDD) and automated testing practices.
Experience implementing CI/CD pipelines and engineering automation.
Strong understanding of Infrastructure as Code principles.
Experience using version control systems such as Git.
Experience delivering technology solutions within UK Public Sector, Utilities, Energy, Water, Financial Services or other highly regulated industries.
Understanding of security, governance and compliance considerations within regulated environments.
Experience with Terraform.
Experience with Docker and Kubernetes.
Knowledge of Ansible and Packer.
Linux and Windows administration experience.
Experience supporting AI, machine learning or advanced analytics platforms.
Experience working on programmes requiring UK Government security clearance.
Strong technical leadership and problem-solving skills.
Ability to communicate effectively with both technical and non-technical stakeholders.
Experience working in complex, regulated environments.
A collaborative approach and willingness to support and mentor others.
A focus on delivering practical solutions that create value for clients.
A passion for building reliable, scalable and secure data platforms.