Data Engineer

Pyx Health is looking for a motivated and technically solid Healthcare Data Engineer to join our growing Data & Analytics team. In this role, you will contribute to building and maintaining our data infrastructure on Azure, work alongside senior engineers to develop reliable data pipelines, and collaborate with analytics teams to engineer data foundations that support our healthcare solutions.

Responsibilities:

• Build and maintain data pipelines for ingesting, transforming, and cleaning healthcare data in the Azure cloud using Databricks, PySpark, and Delta Lake
• Implement pipeline logic from defined specifications, with guidance from senior engineers on architectural decisions
• Build reusable testing frameworks to ensure pipeline reliability
• Monitor pipelines for failures and performance issues, escalating complex problems appropriately
• Develop and maintain Airflow DAGs with appropriate error handling and retry logic
• Support deployment and configuration of pipelines via Astronomer on Azure using the Astro CLI
• Contribute to improving pipeline reliability and reducing manual intervention
• Implement data models for efficient storage and retrieval using Delta Lake, including merge/upsert patterns
• Design and optimize Delta Lake tables for analytic workloads
• Leverage Unity Catalog for data governance, organization, and security
• Apply Change Data Capture (CDC) patterns under the direction of senior team members
• Write and optimize T-SQL and Databricks SQL scripts, stored procedures, and ETL support
• Work within the Azure ecosystem including ADLS Gen2, Key Vaults, Logic Apps, and Azure DevOps
• Follow established security and scalability standards when building data infrastructure
• Support infrastructure tasks with guidance from senior engineers on architecture
• Ensure scripts and datasets are well-documented to support enterprise data governance
• Design, implement and continuously improve automated data quality monitoring, reconciliation, and alerting to ensure reliable downstream reporting
• Troubleshoot and resolve pipeline failures, documenting root causes and resolutions
• Work closely with data scientists, analysts, and business stakeholders to understand reporting requirements, troubleshoot issues, and improve data assets
• Participate in code reviews, incorporating feedback to improve code quality
• Document pipelines, processes, and implementation decisions clearly and consistently
• Stay current with data engineering technologies, particularly in the healthcare space

Requirements:

• 2–4 years of experience as a Data Engineer or in a closely related data role
• Hands-on experience with Azure cloud services (ADLS, Databricks, or similar)
• Working knowledge of SQL for scripting and data modeling including T-SQL, Spark, and Databricks SQL
• Ability to contribute to technical projects with moderate oversight
• Strong communication skills — comfortable asking questions and providing updates to cross-functional partners
• Familiarity with CI/CD concepts, Git, and version control workflows
• Solid problem-solving skills with a systematic approach to debugging
• Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field (or equivalent work experience)
• Databricks SQL (Mid): Delta Lake fundamentals, basic merge/upsert patterns, familiarity with CDC concepts
• Python (Mid): Pipeline logic, data transformation, SQL scripting; some PySpark/Spark DataFrame experience
• Databricks Spark Notebooks (Mid): Notebook-based development, basic cluster usage, Delta table operations
• Airflow Python Development (Mid): Ability to write and maintain DAGs; understanding of error handling and retry patterns
• Airflow Astro Configuration (Foundational): Familiarity with Astronomer or willingness to learn; basic Astro CLI usage
• Azure Ecosystem (Foundational): Working knowledge of ADLS Gen2, Key Vaults, and Azure DevOps
• T-SQL (Foundational–Mid): Basic stored procedures, SQL Server querying, and data manipulation
• Familiarity with healthcare data standards and regulations (HIPAA, HL7, etc.) is a plus
• Azure Data Factory (Foundational): Basic familiarity with pipeline authoring and triggers
• Git + Azure DevOps CI/CD (Foundational): Experience with branching, pull requests, peer review processes, and version control best practices as well as deploying production data pipelines using CI/CD workflows