Senior Artificial Intelligence Data Engineer

Role Overview

• Build and support scalable data engineering solutions using Azure Databricks, PySpark, SQL, Delta Lake, Azure Data Factory, dbt, and Unity Catalog.
• Improve metadata-driven Azure Data Factory and Databricks patterns for orchestration, configuration, monitoring, restartability, and operational support.
• Develop reusable accelerators including CI/CD templates, Databricks Asset Bundle patterns, deployment automation, environment configuration, and data product onboarding templates.
• Design, develop, and support dbt models, macros, tests, documentation, and transformation standards for governed analytical data products.
• Provide guidance on appropriate technology selection and implementation patterns across dbt, Databricks notebooks and workflows, Delta Live Tables, Spark, and Starburst/Trino.
• Support cross-domain analytics initiatives by transforming source-refined data into trusted, reusable, business-aligned derivative data products.
• Leverage Unity Catalog to establish and support governed catalogs, schemas, tables, lineage, access controls, naming standards, and certification practices.
• Support Starburst/Trino as an analytical and federated query layer for governed enterprise data consumption.
• Apply Azure DevOps, Git, CI/CD, and Infrastructure as Code (IaC) practices to create repeatable, testable, and environment-aware platform delivery processes.
• Troubleshoot and resolve production issues related to orchestration, transformations, data quality, access management, query performance, deployments, and operational workflows.
• Collaborate with data engineering, analytics, platform, governance, and business teams to establish reusable, scalable, and supportable data engineering patterns.
• Contribute to the evolution of enterprise data engineering standards, governance practices, observability capabilities, and AI-ready data product frameworks.

Requirements

• Relevant degree preferred.
• 5 or more years of hands-on data engineering experience building production-grade data platforms, pipelines, or analytical data products required.
• Strong experience with Azure Databricks, PySpark, Spark SQL, Delta Lake, Azure Data Factory, SQL, and dbt required.
• Experience with Azure DevOps, Git, pull request workflows, CI/CD pipelines, and release management practices required.
• Working knowledge of lakehouse architecture, metadata management, data governance, lineage, access control, and operational support required.
• Demonstrated ability to function as a senior individual contributor with strong ownership, technical judgment, and cross-functional collaboration skills required.
• Experience supporting enterprise-scale analytical platforms and governed data product delivery preferred.
• Experience with Unity Catalog, Starburst/Trino, Pulumi or other Infrastructure as Code tools, Databricks Asset Bundles, Apache Iceberg concepts, and AKS/Kubernetes-based platform operations preferred.
• Experience building reusable frameworks, accelerators, templates, or platform capabilities for engineering teams preferred.
• Experience preparing governed structured data for AI/ML, GenAI, Retrieval-Augmented Generation (RAG), semantic search, copilots, or agentic workflows preferred.
• Experience within healthcare, analytics, supply chain, finance, or other regulated enterprise environments preferred.
• Strong problem-solving, communication, and collaboration skills with the ability to influence technical direction and establish best practices preferred.
• You must be authorized to work in the United States without sponsorship.

Tech Stack

• Apache
• Azure
• Kubernetes
• PySpark
• Spark
• SQL
• Unity

Benefits

• Vizient has a comprehensive benefits plan!