Senior Databricks Data Engineer / Technical Lead

Data Capital Incorporation is seeking a Senior Databricks Data Engineer / Technical Lead to support a U.S. Department of Transportation data modernization initiative. The role focuses on designing, developing, migrating, and supporting secure, scalable data pipelines in a Databricks Lakehouse environment, including legacy data migration, metadata-driven ingestion, data quality, production operations, and AWS/OneDOT cloud modernization.

Responsibilities:

• Designing, developing, migrating, and supporting secure and scalable data pipelines within a Databricks Lakehouse environment
• Supporting a U.S. Department of Transportation (DOT) data modernization initiative
• Legacy data migration
• Metadata-driven ingestion
• Medallion Architecture
• Data quality
• Reconciliation
• Production operations
• AWS/OneDOT cloud modernization
• Developing and supporting production-grade data pipelines
• Data quality, validation, reconciliation, monitoring, and troubleshooting
• Working effectively with technical and non-technical stakeholders

Requirements:

• 7+ years of professional data engineering experience
• Strong hands-on experience with Databricks and Apache Spark in production environments
• Advanced proficiency in: • Python • PySpark • SQL
• Strong experience with: • Databricks Lakehouse • Delta Lake • AWS / Amazon S3 • Batch and file-based ingestion • CSV/JSON processing • REST APIs • Git • CI/CD • Data pipeline orchestration
• Experience developing and supporting production-grade data pipelines
• Strong understanding of Medallion Architecture (Bronze/Silver/Gold)
• Experience with data quality, validation, reconciliation, monitoring, and troubleshooting
• Understanding of security practices for PII and sensitive data , including access controls and secrets management
• Ability to work effectively with technical and non-technical stakeholders