Data Engineer - Databricks

Resultant is a modern consulting firm that partners with clients to solve complex challenges through data analytics and technology solutions. The Data Engineer will design, build, and optimize Lakehouse-based data platforms and collaborate with various stakeholders to deliver production-grade solutions.

Responsibilities:

• Design, build, and optimize scalable ETL/ELT pipelines on Databricks using PySpark, Spark SQL, and Delta Lake
• Implement Medallion (Bronze/Silver/Gold) architecture patterns, applying data quality checks, schema evolution, and enforcement along the way
• Build declarative pipelines with Delta Live Tables (DLT) and ingest streaming/incremental data using Auto Loader and Structured Streaming
• Orchestrate and monitor production workloads using Databricks Workflows, integrating with tools like Airflow or Azure Data Factory where needed
• Configure and maintain Unity Catalog for data governance — catalogs, schemas, access controls, lineage, and PII masking
• Partner with data scientists to prepare feature-engineered, ML-ready datasets and support model deployment workflows using MLflow
• Tune cluster configuration, job design, and Photon/serverless compute for performance and cost efficiency
• Build and maintain CI/CD pipelines for Databricks notebooks, jobs, and asset bundles (Git-based workflows, Azure DevOps, GitHub Actions, or similar)
• Query, profile, and assess the quality of large, complex datasets from a wide variety of source systems
• Collaborate with solution leads, architects, and project managers on solution design and technical architecture decisions
• Participate directly in client-facing work: requirements gathering, solution reviews, and translating technical tradeoffs into plain-language business impact
• Document solutions clearly — architecture diagrams, data flow documentation, code comments, and runbooks

Requirements:

• Bachelor's degree in Computer Science, Engineering, Data Science, or a related field (or equivalent practical experience)
• 2+ years of hands-on data engineering experience, including production work on the Databricks platform
• Strong hands-on experience with PySpark and Spark SQL
• Practical experience with Delta Lake fundamentals — ACID transactions, OPTIMIZE/Z-ORDER, partitioning, and schema evolution
• Solid SQL skills across relational platforms (SQL Server, Postgres, Oracle, Snowflake, etc.)
• Experience with at least one major cloud platform (Azure, AWS, or GCP) and its data services
• Working knowledge of data modeling (dimensional modeling, 3NF) and ETL/ELT design principles
• Strong communication skills and comfort working directly with clients and non-technical stakeholders
• A collaborative, detail-oriented mindset with a bias toward solution quality and follow-through
• Databricks Certified Data Engineer Associate or Professional
• Experience with Unity Catalog, Delta Live Tables, and Auto Loader in production environments
• Exposure to MLflow, Feature Store, or Databricks Vector Search for AI/ML-enabled use cases
• Experience with Databricks Asset Bundles and CI/CD tooling (GitHub Actions, Azure DevOps, GitLab)
• Familiarity with Terraform or other infrastructure-as-code tooling
• Experience with dbt, Kafka/Event Hubs, or BI tools (Power BI, Tableau) connected to Databricks
• Docker/Kubernetes experience for containerized workloads
• Prior consulting experience, or comfort moving across multiple client engagements and industries