Principal Data Engineer

AvidXchange is a provider of accounts payable automation software and payment solutions for middle-market businesses and their suppliers. The Principal Data Engineer will architect, modernize, and scale cloud-native data platforms, lead Azure SQL Server migrations to Databricks, and build Kafka-based real-time pipelines. The role also enables AI and agentic data capabilities, establishes data engineering standards, and mentors engineers across the organization.

Responsibilities:

• Lead the design and implementation of scalable, cloud-native data architectures on Databricks (Delta Lake, Unity Catalog, Lakehouse patterns)
• Own and execute the migration strategy from legacy Azure SQL Server to Databricks, including schema translation, ETL/ELT re-platforming, data validation, and cutover planning
• Define data modeling standards (medallion architecture, star/snowflake schemas) and ensure consistency across all pipelines and domains
• Evaluate and recommend tools, frameworks, and platforms to support long-term data strategy and organizational goals
• Collaborate with Solution and Enterprise Architects to review and approve new data architecture designs
• Architect and implement Kafka-based streaming pipelines for real-time data ingestion, transformation, and delivery
• Design event-driven architectures and streaming topologies using Kafka Streams, ksqlDB, or Spark Structured Streaming on Databricks
• Establish patterns for schema management (Confluent Schema Registry), consumer group strategy, offset management, and dead-letter queuing
• Ensure streaming pipelines meet SLA requirements for latency, throughput, and fault tolerance
• Debug and optimize Spark jobs, Delta Lake tables, and SQL workloads for performance, cost efficiency, and maintainability
• Lead code reviews focused on senior engineers to enforce standards, best practices, and technical quality
• Manage pipeline quality, data models, and CI/CD delivery workflows; guide teams on continuous improvement
• Promote strong data management practices — data quality, lineage, observability, and governance
• Identify opportunities to improve service delivery methods, processes, and resource utilization
• Mentor data engineers at all levels, with particular emphasis on developing senior talent
• Establish and evolve data engineering standards, best practices, and management of technical debt
• Stay current on data platform trends (Databricks releases, Kafka ecosystem, open table formats) and contribute to long-term architectural vision
• Develop plans for data security, disaster recovery, backup, business continuity, and archiving across the Lakehouse
• Design and enable AI and agentic capabilities on the Databricks platform, including Databricks Genie for natural language data exploration and self-service analytics
• Architect data foundations — clean, governed, well-documented Delta tables — that power Genie spaces, AI/BI dashboards, and LLM-driven data agents
• Collaborate with ML and AI teams to build and maintain feature stores, vector stores, and retrieval-augmented generation (RAG) pipelines on Databricks
• Evaluate and integrate emerging agentic frameworks (LangChain, Mosaic AI Agent Framework) to automate data workflows and enable intelligent data products
• Define governance and observability standards for AI-driven data pipelines, ensuring reliability, auditability, and responsible AI practices
• Partner with project managers and business leaders on initiatives involving enterprise data
• Collaborate across teams to influence and strengthen data engineering practices organization-wide
• Work with Analytics, ML, and product engineers to design and deliver end-to-end data solutions that meet business needs

Requirements:

• Bachelor's degree in Computer Science, Engineering, or a related field with 10+ years of data engineering experience in a high-availability, business-critical environment
• Hands-on Databricks expertise: Delta Lake, Unity Catalog, Databricks Workflows, Databricks SQL, and cluster/job optimization
• Proven experience migrating from legacy Azure SQL Server (or other relational RDBMS) to a Databricks Lakehouse — schema translation, data validation, and cutover strategies
• Strong proficiency in Apache Kafka for streaming pipelines — producers/consumers, topic design, partitioning strategy, and Kafka Connect
• Expert-level PySpark and/or Scala Spark skills, including performance tuning, broadcasting, partitioning, and caching
• Deep understanding of data architecture patterns: medallion (bronze/silver/gold), Lambda/Kappa, event sourcing, and streaming-first designs
• Hands-on experience with Azure cloud services (ADLS Gen2, Azure Event Hubs, ADF, Azure Key Vault, Azure Monitor)
• Strong knowledge of infrastructure components — networking, cloud storage (Delta/Parquet), and cloud cost optimization
• Databricks Certified Data Engineer Associate or Professional certification
• Experience with ksqlDB, Kafka Streams, or Spark Structured Streaming for stateful stream processing
• Confluent Platform experience: Schema Registry, Kafka Connect connectors, RBAC, and cluster management
• Advanced Azure SQL Server expertise (2018+/Azure SQL MI) — stored procedures, indexing strategies, query plan analysis — valuable for migration contexts
• Experience with dbt (data build tool) for SQL-based transformation layers on Databricks
• Proficiency with Git and CI/CD pipelines for data engineering (Azure DevOps, GitHub Actions)
• Experience working in Agile environments (Scrum/Kanban)
• Familiarity with data governance frameworks, data cataloging (Unity Catalog, Microsoft Purview), and data quality tooling (Great Expectations, Monte Carlo)
• Familiarity with secure coding practices, including OWASP Top 10 and secrets management
• Experience implementing DataOps practices: automated testing, data observability, pipeline CI/CD, data contracts, and SLA monitoring across the Lakehouse
• Hands-on MLOps experience: model versioning (MLflow), experiment tracking, model serving, and integrating ML pipelines with production data workflows on Databricks
• Familiarity with Databricks Mosaic AI (formerly MLflow + Model Serving) for end-to-end MLOps lifecycle management
• Experience with real-time ML feature stores or Lakehouse-based ML pipelines is a plus