Databricks Data Engineer

Guidehouse is a consulting and professional services company supporting clients through its AI & Data team. The Databricks Data Engineer will design, build, and maintain scalable batch and streaming data pipelines, transform large-scale datasets, and deliver reliable data products using Databricks, Spark, SQL, Python, and cloud-native technologies. The role also involves troubleshooting, documentation, governance, and collaboration with technical teams and clients.

Responsibilities:

• Design, build, and maintain scalable data pipelines using Databricks, PySpark, SQL, Delta Lake, and related cloud-native data engineering tools
• Develop and support batch and streaming data workflows, including ingestion, transformation, validation, and publishing of curated data products
• Write, optimize, and maintain Python, PySpark, and SQL code for data processing, orchestration, and performance tuning
• Work with large-scale datasets using Databricks, Spark, Delta Lake, Unity Catalog, and cloud storage services
• Troubleshoot and resolve data pipeline failures, performance issues, data quality issues, and workflow bottlenecks
• Translate business and technical requirements into data engineering designs, processing scripts, and job orchestration workflows
• Perform data validation, quality checks, code reviews, and issue resolution to support accurate and reliable data products
• Collaborate with cross-functional teams including solution architects, data scientists, analysts, DevOps engineers, and client stakeholders
• Communicate technical concepts, delivery impacts, and data engineering considerations to technical and non-technical audiences
• Document pipelines, datasets, data models, workflows, and engineering decisions to support maintainability, transparency, and reuse
• Follow data governance, security, lineage, and compliance standards within the Databricks platform

Requirements:

• • Bachelor's degree in computer science, engineering, mathematics, statistics, or another relevant field
• • 3-8 years of relevant experience in data engineering, data architecture, or cloud data platform implementation
• • Strong experience with Python, PySpark, and SQL for data transformation, pipeline development, and data processing
• • Experience with Databricks, Spark, Delta Lake, Unity Catalog, or similar cloud-native data platforms
• • Experience developing batch or streaming data pipelines, ETL/ELT workflows, and reusable data assets
• • Experience with data modeling, data warehousing, data quality validation, and large-scale data processing concepts
• • Ability to troubleshoot technical issues, communicate engineering recommendations clearly, and work effectively in team-based delivery environments
• • Experience applying data governance, access control, lineage, and security practices within cloud data platforms
• • 2+ years of hands-on experience with the Databricks platform
• • Experience using Unity Catalog for data governance, schema design, data modeling standards, access controls, lineage, and secure management of enterprise data assets
• • Experience designing and building standard medallion data architectures for scalable ingestion, transformation, quality validation, and curated data delivery
• • Active Databricks Data Engineer Associate, Databricks Data Engineer Professional, or related certification
• • Experience with CI/CD tools and practices for notebooks, workflows, orchestration jobs, and infrastructure deployment
• • Experience with cloud platforms such as Azure, AWS, or GCP
• • Experience working in project-based or consulting delivery environments
• • Familiarity with streaming frameworks, orchestration tools, Terraform, or related data platform administration capabilities
• This position offers virtual work flexibility within the United States. While remote candidates will be considered, preference will be given to candidates located near a Guidehouse office in one of the following markets: Arlington, VA; Washington, DC; New York, NY; Chicago, IL; Austin, TX; Atlanta, GA; Boston, MA; and Boulder, CO.