Lead Data Engineer

Capital Technology Group provides consulting services in software development, digital transformation, human-centered design, data analytics and visualization, and cybersecurity. The Lead Data Engineer will design, build, and maintain scalable data pipelines, cloud-based data platforms, analytical systems, and AI-enabled data solutions using AWS and modern data engineering technologies. The role also includes improving platform reliability, supporting secure federal analytics environments, mentoring junior engineers, and collaborating with cross-functional Agile teams.

Responsibilities:

• Design, build, and maintain scalable data pipelines, ETL/ELT workflows, and data models using Python, Apache Spark (PySpark), SQL (PostgreSQL), and AWS Glue
• Develop and optimize AWS-native data platforms leveraging AWS Glue, Amazon EMR, Amazon MWAA (Apache Airflow), Amazon S3, RDS, and CloudWatch
• Build high-performance ingestion, transformation, and orchestration workflows for structured and semi-structured data using Apache Iceberg and Parquet
• Design and optimize analytical data platforms using Amazon Athena, Trino, Hive, OpenSearch, and enterprise data catalog technologies
• Build AI-enabled data solutions using Amazon Bedrock, RAG pipelines, and vector search technologies including Amazon S3 Vectors and OpenSearch vector indexes
• Develop cloud infrastructure using CloudFormation (Infrastructure as Code), GitHub and enterprise CI/CD pipelines
• Improve the reliability, scalability, performance, and maintainability of enterprise data platforms through monitoring, troubleshooting, automation, and continuous optimization
• Support mission-critical analytics and reporting solutions within large-scale AWS-based federal data environments, implementing solutions that comply with FedRAMP and NIST SP 800-53 security controls
• Mentor junior engineers through technical guidance, architecture discussions, and code reviews while promoting engineering best practices
• Collaborate with cross-functional teams in an Agile environment to define requirements, deliver high-quality data solutions, and communicate technical concepts effectively to technical and non-technical stakeholders

Requirements:

• applicants MUST BE US Citizens and be able to obtain Public Trust clearance
• • Bachelor's degree in Computer Science, Engineering, or a related technical field
• • 15+ years of professional experience in data engineering, data architecture, or related fields
• • Strong hands-on experience with:
• • Apache Spark (PySpark) required, Python, SQL (PostgreSQL), and dbt for large-scale data engineering, ETL/ELT development, data transformation, and data modeling
• • AWS Glue, Amazon EMR, Amazon MWAA (Apache Airflow), AWS Lambda, Amazon S3, Amazon RDS
• • Developing scalable data pipelines, workflow orchestration, and data integration solutions across enterprise environments
• • Working with modern data lake technologies and data formats such as Parquet and Iceberg
• • Designing and optimizing solutions using relational and NoSQL databases
• • Building reliable, high-performance data platforms through performance tuning, system optimization, and enterprise-scale ETL/ELT architectures
• • Strong analytical and problem-solving skills
• • Experience working in Agile, iterative software development environments
• • Ability to quickly learn and apply new technologies and domain knowledge
• • Excellent written and verbal communication skills, with the ability to explain complex topics to diverse audiences
• • Experience supporting analytics, data engineering, or modernization initiatives for financial regulators, capital markets, or other highly regulated environments is a plus
• • Experience with Apache Iceberg and modern data lakehouse architectures
• • Experience working with unstructured data processing, including document/text processing, embeddings, vector search, and LLM-based data solutions
• • Exposure to integrating LLMs and generative AI capabilities into enterprise data pipelines and platforms
• • Experience designing data architectures that support both structured and unstructured data at scale