Data Engineer / Architect – Databricks & dbt

Capgemini is seeking a Data Engineer/Architect to implement enterprise data pipelines on a modern cloud-based data platform using Databricks and dbt. The role defines scalable engineering patterns and standards, provides hands-on technical guidance, and collaborates with architecture, DevOps, governance, and delivery teams.

Responsibilities:

• Define the technical architecture and implementation patterns for dbt-based data transformations on Databricks
• Design and develop scalable curated and consumption-layer data pipelines using Databricks and dbt
• Establish dbt project structure, model organization, dependencies, naming conventions, coding standards, and reusable development patterns
• Define and implement appropriate usage of dbt models, sources, tests, macros, snapshots, incremental models, packages, and documentation
• Design efficient transformation patterns leveraging Databricks, Delta Lake, Spark SQL, and PySpark
• Establish reusable and standardized pipeline development patterns for adoption across the engineering team
• Work with architecture teams to ensure implementation aligns with enterprise architecture, governance, security, and data-platform standards
• Provide hands-on development support and technical guidance to Data Engineers
• Review existing data transformation and pipeline patterns and determine the appropriate implementation using Databricks and dbt
• Define data quality, reconciliation, testing, logging, monitoring, and error-handling approaches
• Design and implement appropriate incremental data-processing strategies
• Support performance optimization and troubleshooting of dbt models and Databricks workloads
• Work with DevOps teams to integrate dbt and Databricks development into the enterprise CI/CD framework
• Support orchestration and scheduling integration with Databricks Workflows and enterprise scheduling platforms
• Conduct technical design reviews and code reviews and ensure adherence to defined engineering standards
• Support testing, deployment, production readiness, and troubleshooting activities
• Develop reusable templates, frameworks, utilities, and implementation guidelines to accelerate development
• Prepare technical documentation and provide knowledge transfer to engineering and support teams
• Mentor Data Engineers and provide technical leadership throughout design, development, testing, and deployment

Requirements:

• 8+ years of experience in Data Engineering, Data Architecture, or related areas
• Strong hands-on experience with Databricks
• Strong hands-on experience implementing enterprise data solutions using dbt
• Strong understanding of dbt models, sources, macros, tests, snapshots, incremental models, packages, and documentation
• Strong programming and development skills in SQL, Spark SQL, and PySpark
• Strong experience with Delta Lake and Databricks data-engineering capabilities
• Experience designing and implementing Medallion Architecture / multi-layered data platforms
• Strong understanding of logical and physical data modeling concepts
• Experience building scalable, reusable, and metadata-driven data-engineering solutions
• Experience implementing data-quality, validation, reconciliation, lineage, monitoring, and observability capabilities
• Strong understanding of CI/CD and DevOps practices for dbt and Databricks
• Experience with Git-based source control and automated deployment processes
• Strong understanding of performance optimization and troubleshooting of large-scale data pipelines
• Ability to translate architecture standards and business requirements into practical engineering solutions
• Experience conducting technical design and code reviews
• Ability to mentor and provide technical direction to Data Engineers
• Strong analytical, problem-solving, communication, and stakeholder-management skills
• Databricks certification
• dbt certification or significant production implementation experience
• Experience with AWS-based Databricks environments
• Experience with enterprise data governance, metadata management, lineage, and data-product concepts
• Experience with Control-M or other enterprise workload scheduling/orchestration platforms
• Experience working on large-scale enterprise data-platform transformation programs