Cybermedia Technologies, LLC (CTEC) is a leading technology firm that provides modernization, digital transformation, and application development services to the U.S. Federal Government. They are seeking a Data Engineer to design and develop scalable ETL pipelines, support data migration, and implement data governance initiatives using Azure Databricks and cloud-based data platforms.
Responsibilities:
- Design, develop, and maintain scalable ETL pipelines and data workflows to ingest, transform, and integrate data from legacy systems and external sources into modern cloud-based data platforms
- Build, optimize, and maintain data processing solutions using Azure Databricks and lakehouse architectures to support analytical, operational, and reporting use cases
- Support phased data migration from legacy databases and ETL tools to Azure Databricks environments, including transformation documentation and data mapping
- Implement layered lakehouse data architectures (e.g., bronze, silver, gold layers) in Databricks to support data quality, performance, and downstream reporting needs
- Develop data processing notebooks, workflows, and distributed data transformations using Python and PySpark within Databricks environments
- Develop data validation, reconciliation, and testing processes to ensure data accuracy, completeness, and consistency across data domains
- Integrate Databricks data platforms with analytics and reporting tools to enable business intelligence and operational dashboards
- Support data governance initiatives including metadata management, data catalog integration, encryption, access controls, and compliance with federal data protection requirements
- Maintain source control and CI/CD pipelines for Databricks and data engineering workflows, supporting automated promotion across environments
- Work closely with data architects, solution architects, business analysts, and reporting teams to implement approved data solutions
- Provide ongoing support for Databricks workflows, resolve pipeline failures, and troubleshoot complex data processing issues
- Provide guidance to junior data engineers and contribute to documentation and team enablement
- Works under minimal supervision with minor guidance from senior personnel
Requirements:
- 7+ years of experience in data engineering, ETL development, or large-scale data integration environments
- Strong experience designing and developing ETL pipelines and data transformations in Azure Databricks environments
- Strong proficiency in SQL and Python, with hands-on experience using PySpark for distributed data processing
- Experience working with cloud-based data platforms, data lakes, and lakehouse environments, preferably on Microsoft Azure
- Experience implementing layered lakehouse data architectures (bronze, silver, gold) for enterprise analytics
- Familiarity with Spark-based big data processing frameworks
- Experience supporting data migration from legacy databases and ETL tools to Databricks-based platforms
- Experience integrating Databricks platforms with business intelligence and reporting tools such as Power BI
- Familiarity with data governance, metadata management, and data security best practices
- Experience with source control and CI/CD pipelines for data engineering and Databricks workflows
- Working knowledge of SDLC and Agile delivery methodologies
- Excellent organizational, communication, and collaboration skills
- Bachelor's degree in Computer Science, Engineering, Information Systems, or a related technical discipline. Equivalent education or professional experience will be considered in lieu of a degree
- Must be a U.S. citizen and be able to obtain a Public Trust clearance
- Hands-on experience with Azure Databricks workflows, Delta tables, and Databricks job orchestration
- Experience with Azure Data Lake, Azure Data Factory, or similar Azure data services
- Experience supporting federal IT modernization or large-scale enterprise data transformation initiatives
- Familiarity with healthcare, insurance, or benefits administration data environments
- Experience implementing data governance or data catalog platforms