Senior Databricks Architect / Lead Data Engineer

Position Summary

We are seeking an experienced Senior Databricks Architect / Lead Data Engineer to lead the design, development, and implementation of modern data platforms leveraging Azure Databricks. The ideal candidate will possess deep expertise in building scalable data engineering solutions, strong hands-on coding skills, and proven experience delivering enterprise-grade Databricks implementations for clients.

This role requires a blend of technical leadership, architecture, solution delivery, and client-facing consulting capabilities.

Key Responsibilities

Solution Architecture & Design

• Design and architect scalable, secure, and high-performance data platforms using Azure Databricks.
• Lead the design and implementation of modern data lakehouse architectures utilizing Delta Lake, Unity Catalog, and Databricks best practices.
• Define data ingestion, transformation, orchestration, governance, and consumption patterns across enterprise data ecosystems.
• Establish architecture standards, development frameworks, and reusable components.

Data Engineering & Development

• Develop and optimize complex data pipelines using PySpark and Databricks.
• Build and maintain ETL/ELT frameworks leveraging Delta Live Tables (DLT) and Databricks Workflows.
• Ensure code quality, performance optimization, scalability, and reliability of data processing solutions.
• Implement data quality, monitoring, lineage, and governance controls.

Project Delivery

• Lead end-to-end Databricks implementations from strategy and design through deployment and stabilization.
• Collaborate with business stakeholders, data consumers, architects, and platform teams to translate business requirements into technical solutions.
• Guide development teams and perform design/code reviews.
• Troubleshoot complex technical challenges and provide solution recommendations.

Client Engagement & Consulting

• Serve as a trusted advisor to client stakeholders on data modernization and analytics transformation initiatives.
• Conduct architecture workshops, solution demonstrations, and technical assessments.
• Present technical solutions and recommendations to business and executive audiences.
• Support estimation, solutioning, proposal development, and pre-sales activities as needed.

Required Qualifications

• 10+ years of overall experience in Data Engineering, Data Architecture, or Analytics platform delivery.
• 5+ years of hands-on experience with Databricks.
• Demonstrated experience delivering at least two (2) end-to-end Databricks implementations in enterprise environments.
• Strong hands-on coding expertise in PySpark with the ability to develop solutions during technical interviews and coding assessments.
• Extensive experience with:
  • Unity Catalog
  • Delta Lake
  • Delta Live Tables (DLT)
  • Databricks Workflows
• Strong experience designing and implementing Azure Databricks architectures.
• Experience integrating Databricks with Azure services such as Azure Data Lake Storage (ADLS), Azure Data Factory (ADF), Azure Synapse, Azure Key Vault, and Microsoft Fabric (preferred).
• Strong understanding of data modeling, ETL/ELT design patterns, data governance, and performance tuning.
• Proven client-facing consulting experience with excellent communication and stakeholder management skills.

Preferred Qualifications

• Databricks Certified Professional or Associate certifications.
• Azure Data Engineering and/or Azure Solutions Architect certifications.
• Experience implementing CI/CD, Infrastructure as Code, and DevOps practices for Databricks environments.
• Experience with data governance, metadata management, security, and access controls in enterprise data platforms.
• Familiarity with streaming architectures, real-time analytics, and advanced analytics workloads.

Key Competencies

• Enterprise Data Architecture
• Data Lakehouse Design
• PySpark Development
• Azure Databricks
• Solution Leadership
• Client Relationship Management
• Technical Consulting
• Data Governance & Security
• Problem Solving & Analytical Thinking
• Team Leadership & Mentoring