Data Engineer

The hiring client is confidential. The Data Engineer will develop and maintain scalable data platforms and pipelines for insurance analytics, risk management, and business intelligence, with a focus on data governance and multi-region architecture.

Responsibilities:

• Design and develop scalable ETL/ELT pipelines using Databricks, Delta Lake, and Auto Loader to ingest, transform, and load data from diverse sources
• Architect and implement data lake/lakehouse solutions that support medallion architecture patterns (Bronze/Silver/Gold layers) with multi-region deployment capabilities
• Implement and maintain secure data governance frameworks leveraging Unity Catalog, ABAC policies, and Row-Level Security (RLS) for fine-grained row and column-level access control across multi-region environments, ensuring compliance with insurance regulatory requirements
• Build and optimize Delta Live Tables (DLT) pipelines with expectations and quality checks, supporting distributed deployments across multiple cloud regions
• Collaborate with data product owners, analysts, and business stakeholders to define data requirements, SLAs, and performance optimization strategies
• Implement data governance, quality controls, and audit trails to ensure data accuracy, integrity, and regulatory compliance across geographically distributed systems
• Develop and maintain documentation, data dictionaries, and technical specifications for data pipelines and domain models
• Troubleshoot data pipeline failures, performance bottlenecks, and data quality issues in production environments spanning multiple regions

Requirements:

• Hands-on experience with Databricks (Delta Lake, Spark, Notebooks, Jobs, DLT)
• Deep understanding of data lake and lakehouse architecture patterns, including medallion architecture and query optimization
• Strong expertise in ETL/ELT design patterns and distributed data processing
• Experience implementing ABAC (Attribute-Based Access Control) and Row-Level Security (RLS) frameworks
• Proficiency in Python and SQL for data pipeline development
• Familiarity with Unity Catalog for data governance and metadata management
• Experience with multi-region data architecture, federation, and data residency requirements
• AWS cloud platform expertise
• Proficiency with version control (Git) and CI/CD pipelines
• Understanding of data quality frameworks, validation patterns, and test-driven data development
• Working knowledge of insurance industry terminology, business processes, and data models
• Familiarity with common insurance data entities: Party, Claims, Fiduciary, etc
• Understanding of regulatory requirements relevant to insurance
• Strong problem-solving and analytical thinking abilities
• Excellent communication skills to explain technical concepts to non-technical stakeholders
• Collaborative mindset with experience working in cross-functional teams
• Ability to manage multiple priorities and projects in a fast-paced environment