Senior Data Engineer

Ellis Porter is an immigration law firm focused on helping employers and individuals navigate complex immigration matters. The Senior Data Engineer will be the firm’s first in-house data engineer, owning the Databricks platform, building reliable data pipelines and products, managing platform operations, and partnering with stakeholders to deliver practical data solutions.

Responsibilities:

• Build and maintain ingestion pipelines that bring data from EP’s business systems into Databricks, with automated quality checks and source reconciliation to identify missing, inaccurate, or duplicate data
• Design and develop silver and gold data layers, delivering governed data products that support reporting, cross-system matching, and identity resolution
• Own day-to-day platform operations, including job scheduling, monitoring, alerting, data freshness, access control, and cost optimization; maintain runbooks, test recovery procedures, and document knowledge so the platform can be supported by others
• Manage CI/CD pipelines and deployments across development, testing, and production environments using Azure DevOps
• Coordinate with EP’s IT vendor on Azure infrastructure needs outside the data platform; preserve systems of record, enforce user-appropriate access, and validate licensing and platform constraints before deployment
• Partner directly with business stakeholders to understand their needs and translate them into practical, reliable data solutions
• Design and implement approved confidentiality, retention, least-privilege access, and audit controls for sensitive client and matter data
• Support bounded AI and partner pilots with appropriate masking, evaluation data, and human approval before expanding access or enabling changes to operational records
• Perform other duties as assigned by management

Requirements:

• Bachelor's or Master's degree in Computer Science, Engineering, or related field, or equivalent practical experience
• 4+ years of experience in data engineering
• 2+ years of hands-on experience building and operating Databricks solutions in production, including multi-task Lakeflow Jobs over Delta tables governed by Unity Catalog
• Ability to explain a production pipeline you've built, the reasoning behind your design decisions, and how you addressed failures or operational challenges
• Communicate clearly with nontechnical partners and connect engineering decisions to business impact
• Proficiency in Python, including testing (pytest) and packaging
• Strong SQL and dimensional modeling expertise, including serving governed datasets to BI tools
• Experience building REST API and SaaS ingestion pipelines, including handling pagination, rate limits, schema drift, and credential rotation
• Experience with Git-based CI/CD workflows
• Experience using AI-assisted development tools, with sound judgment in reviewing, testing, and validating generated code before deployment
• Experience deploying through Declarative Automation Bundles (formerly Databricks Asset Bundles) or equivalent infrastructure-as-code is strongly preferred
• PySpark experience preferred
• Azure DevOps is a plus
• Experience building Lakeflow pipelines, managed connectors, entity resolution, or record linkage is a plus
• Experience managing the Azure resources that support a data platform (Databricks workspace administration, storage, Key Vault and secrets, service principals, networking), ideally as infrastructure-as-code, is a plus