Job Description: Data Quality Engineering & FrameworksDesign and implement enterprise-wide data quality frameworks aligned to Lakehouse architecture (bronze, silver, gold layers)
Define and enforce data quality rules including completeness, accuracy, consistency, timeliness, and validity
Develop reusable data validation, reconciliation, and monitoring patterns within Databricks pipelines
Establish automated data quality checks embedded within ELT/ETL workflows
Databricks & Pipeline IntegrationIntegrate data quality controls directly into Databricks (Spark/Delta Lake) pipelines and workflows
Develop scalable validation processes for batch and event-driven ingestion pipelines
Partner with Data Engineers to ensure quality gates are enforced across ingestion, transformation, and consumption layers
Optimize data quality processes for performance and scalability within large distributed datasets
Monitoring, Observability & Issue ManagementImplement and manage data observability frameworks, including metrics, alerts, and dashboards
Monitor data pipelines and proactively identify anomalies, failures, and quality degradation
Lead root cause analysis (RCA) efforts for data quality issues and drive remediation
Develop and maintain quality scorecards and reporting for stakeholders
Data Governance & ComplianceEnsure adherence to enterprise data governance standards, including metadata, lineage, and auditability
Partner with Data Governance teams (e.g., Collibra) to align data definitions, ownership, and controls
Support regulatory requirements (e.g., SOX, GLBA, data integrity standards) through auditable data quality controls
Define and enforce data quality SLAs and data contracts across domains
Automation & DevOpsImplement CI/CD practices for data quality rules, validations, and monitoring
Automate testing frameworks for validating data transformations and pipelines
Develop reusable libraries and frameworks for enterprise-scale data quality enforcement
Collaboration & LeadershipPartner with Data Engineers, Data Architects, BI teams, and business stakeholders to embed quality-by-design principles
Provide technical leadership and mentorship on data quality best practices
Act as a subject matter expert (SME) for data quality across the organization
Drive continuous improvement and innovation in data quality tooling and methodologies
Required Qualifications5+ years of experience in data engineering, data quality engineering, or related roles
Strong hands-on experience with Databricks, Spark (PySpark), and Delta Lake
Proven experience implementing data quality frameworks and controls in modern data platforms
Advanced SQL and data profiling/validation skills
Experience working with large-scale datasets in cloud environments (AWS or Azure)
Experience integrating data quality into ELT/ETL pipelines and orchestration tools
Strong understanding of data governance and data lifecycle management
Preferred QualificationsExperience in financial services or regulated environments
Familiarity with data governance tools (e.g., Collibra)
Experience with data observability or quality tooling (e.g., Monte Carlo, Great Expectations, Deequ, or similar)
Experience with real-time data quality validation (streaming pipelines)
Knowledge of regulatory reporting and data controls frameworks
Cloud or Databricks certifications
Technical SkillsDatabricks (Lakehouse, Unity Catalog, workflows)
Spark / PySpark
SQL (advanced)
Delta Lake
Data quality frameworks (rule engines, validation patterns)
Data observability and monitoring
Cloud platforms (AWS or Azure)
Orchestration tools (Airflow, Control-M)
APIs and data integration
CI/CD and DevOps
Data modeling and lineage concepts
Professional CompetenciesStrong analytical and problem-solving skills with a focus on data integrity
High attention to detail and commitment to data accuracy
Strong communication skills across technical and non-technical stakeholders
Ability to influence standards and drive enterprise adoption
Collaborative mindset with a focus on continuous improvement
