Principal Data Platform Engineer (Databricks)

Sphere is seeking a Principal Data Platform Engineer (Databricks) to join their modern data platform initiative for healthcare and life sciences clients. This role involves owning the architecture and delivery of scalable data solutions and collaborating closely with stakeholders to drive complex data decisions.

Responsibilities:

• Own end-to-end architecture and delivery of scalable data solutions, with strong emphasis on Databricks-based platforms and modern cloud ecosystems
• Lead the design and implementation of data pipelines, data models, and transformation frameworks supporting analytics, reporting, and advanced use cases
• Serve as the primary client-facing technical lead, building trusted relationships and guiding stakeholders through complex data decisions
• Translate ambiguous business requirements into clear technical architectures and delivery plans
• Establish and enforce best practices across data engineering — ingestion, pipeline orchestration, testing, optimization — and DevOps
• Drive platform strategy and architecture decisions, including lakehouse design, medallion architecture, and governance frameworks
• Lead and mentor delivery teams, providing technical guidance, code reviews, and hands-on support
• Collaborate with cross-functional teams — data architects, analysts, client stakeholders — to ensure alignment and value delivery
• Identify risks and proactively address challenges to ensure high-quality, on-time delivery
• Contribute to internal capability building: reusable frameworks, accelerators, and thought leadership
• Support business development by shaping technical solutions and contributing to proposals and client discussions

Requirements:

• 7+ years of data engineering experience, with clear progression into technical leadership and architecture ownership
• Deep expertise in Databricks and modern lakehouse architectures, including Delta Lake and Spark-based processing
• Advanced SQL and Python skills, with strong experience building and optimizing large-scale data pipelines
• Hands-on experience with cloud platforms (AWS, Azure, or GCP), including data services and infrastructure design
• Solid understanding of data modeling concepts, ETL/ELT patterns, and distributed data processing
• Experience with orchestration tools (e.g., Airflow) and transformation frameworks (e.g., dbt)
• Proven ability to lead technical delivery while staying hands-on
• Strong client-facing experience: requirements gathering, solution design, executive communication
• Ability to navigate ambiguity, prioritize effectively, and drive clarity in complex environments
• Healthcare data experience (e.g., Epic, HL7, FHIR, claims data)
• Experience with CI/CD, DevOps practices, and infrastructure-as-code tools