Apolis is recruiting for a confidential client seeking a Databricks Data Engineer to design and implement a scalable sales data platform using Databricks on AWS. The role focuses on lakehouse architecture, data pipelines, platform integration, governance, operational excellence, and technical leadership.
Define end-to-end lakehouse architecture on Databricks running on AWS for sales data domains
Design Medallion Architecture (Bronze, Silver, and Gold layers) aligned with enterprise data standards
Establish data modeling standards for sales facts, dimensions, hierarchies, and aggregations
Define scalable ingestion patterns for batch and incremental data loads
Drive performance, scalability, and cost optimization best practices
Develop and guide the implementation of PySpark-based data pipelines using Databricks
Implement Delta Lake capabilities, including ACID transactions, schema evolution, schema enforcement, time travel, and versioning
Design and optimize large-scale joins, aggregations, and window functions
Implement Change Data Capture (CDC) and incremental processing using watermarking and change detection
Build idempotent, restartable, and fault-tolerant data pipelines
Develop and maintain data transformations using dbt, PySpark, and Spark SQL
Design and implement data solutions using AWS services, including:
Optimize Databricks cluster configurations, including job and all-purpose clusters
Implement secure connectivity, secrets management, and access-control patterns
Define and implement data quality checks, validations, and reconciliation processes
Ensure data lineage, metadata capture, and traceability
Implement error handling, auditing, and monitoring frameworks
Support enterprise data governance, security, and access-control requirements
Maintain data accuracy, consistency, and reliability across pipelines
Implement CI/CD pipelines for Databricks notebooks, jobs, and data engineering workflows
Establish code versioning, code review, and deployment standards
Design monitoring, alerting, and service-level agreement (SLA) tracking for data pipelines
Support production stabilization, troubleshooting, and performance tuning
Optimize pipeline execution, cluster utilization, and operational efficiency
Provide technical leadership and mentorship to Databricks Data Engineers
Collaborate with Sales, CRM, ERP, source system, cloud platform, and analytics teams
Work with downstream data consumers and application teams to understand data requirements
Translate business requirements into scalable technical designs
Conduct technical design reviews and promote reusable engineering standards