Sr Data Engineer - Databricks

Adastra is a global leader in AI and data-driven transformation, helping organizations lead with artificial intelligence—responsibly, strategically, and at scale. They are looking for a hands-on Data Engineer to design and run Databricks pipelines that turn raw operational, marketing, and retail data into trusted, analytics-ready data products. The role involves owning pipelines end to end on the Databricks Lakehouse Platform and ensuring the delivery of high-quality data products.

Responsibilities:

• Design, build, and maintain scalable batch and streaming pipelines on Databricks using PySpark and Spark SQL within a medallion (Bronze/Silver/Gold) architecture
• Ingest and unify data from operational databases, cloud storage, APIs, and marketing platforms such as Braze
• Model and optimize Delta Lake tables for performance, reliability, and cost efficiency using techniques such as partitioning, liquid clustering, OPTIMIZE, and VACUUM
• Deliver analytics-ready data for reporting on sales, shipments, orders, and other key business metrics, supporting Power BI dashboards and downstream analytics
• Orchestrate jobs using Databricks Workflows, managing dependencies, retries, alerting, and SLAs
• Apply Unity Catalog for data governance, lineage, and access control
• Work to a high engineering standard using Git, CI/CD, testing, code reviews, and clear documentation
• Monitor pipeline health, troubleshoot issues, and tune Spark jobs and cluster configurations for speed and cost optimization
• Partner with business and technical stakeholders to deliver reliable, high-quality data products

Requirements:

• Bachelor's degree in Computer Science, Information Technology, Engineering, Mathematics, or a related field
• 5+ years of data engineering experience building and operating production data pipelines
• Hands-on expertise with the Databricks Lakehouse Platform, including Delta Lake, Databricks Workflows, and Unity Catalog
• Advanced experience with PySpark and Spark SQL
• Strong SQL development and performance tuning skills
• Solid Python development skills and experience with modern engineering practices
• Experience using Git, CI/CD pipelines, testing frameworks, and code review processes
• Experience working with AWS, Azure, or GCP and their core compute and storage services
• Strong understanding of data modeling concepts, including dimensional and medallion architectures
• Experience designing ELT/ETL solutions and integrating data from databases, APIs, files, and event streams
• Strong communication skills and the ability to work independently within a client-facing environment
• Databricks Certified Data Engineer Associate or Professional certification
• Experience with Structured Streaming, Auto Loader, Kafka, or Kinesis
• Experience working with marketing or customer data platforms such as Braze
• Retail, grocery, or consumer packaged goods (CPG) industry experience
• Experience supporting Power BI reporting and self-service analytics initiatives