Lead Data Engineer

Attain Finance is looking for a Lead Data Engineer to ensure the reliability and operational excellence of their data platform. The role involves hands-on work with data pipelines, optimizing architecture, and leading incident responses while also defining standards and driving cost optimization.

Responsibilities:

• Build, maintain, and operate data pipelines and curated data products across Snowflake, Airflow (MWAA), AWS, Python, and SQL
• Optimize Snowflake architecture at scale. Design warehouse strategies, implement clustering and materialized views, enforce resource monitors, control costs, and tune performance across large datasets
• Implement observability and data quality controls. Build monitoring for freshness, volume, schema, distribution, and lineage. Define data quality SLOs and ensure teams can see issues before users do
• Support and operate production data systems. Troubleshoot failures, respond to alerts, debug pipeline issues, and improve system behavior based on real incidents
• Own SLAs, uptime, and service commitments for data platforms and critical services. Monitor performance, track reliability metrics, and make sure availability and freshness targets are met
• Lead major incident responses and postmortems. Coordinate resolution during critical outages, drive root cause analysis, and implement corrective actions that prevent recurrence
• Architect data platforms across dev, test, and production. Design for disaster recovery, upgrades without downtime, automation that recovers from failure, and availability that supports business continuity
• Define and enforce data platform standards. Establish orchestration patterns, DAG anti-patterns, deployment practices, observability standards, data quality patterns, and operational runbooks used across the organization
• Lead cost optimization programs. Identify savings across Snowflake (compute/storage) and AWS resources, forecast capacity needs, and balance cost against reliability and performance goals
• Lead AI and LLM adoption and governance. Build and validate AI-assisted pipelines, define approved use cases and guardrails, and ensure AI-generated output meets reliability and quality expectations
• Make architectural decisions with an organization-wide impact. Evaluate build versus buy, set migration strategies for legacy systems, define data contracts and API standards, and balance innovation with operational stability
• Lead technical planning and remove blockers. Coordinate with Cloud, Security, Analytics, Data Science, and Product teams to resolve dependencies and keep delivery moving
• Prevent knowledge silos. Write documentation, maintain runbooks, record architectural decisions, and create shared practices that let engineers operate independently and onboard quickly
• Design and enforce data governance controls for regulated environments. Implement PII handling, access controls, audit logging, data retention policies, and compliance validation for SOC 2, GDPR, HIPAA, or similar frameworks

Requirements:

• Deep hands-on experience with Snowflake, Airflow (MWAA), AWS, Python, and SQL in production environments
• Strong ability to design, build, debug, and operate data pipelines at scale
• Experience owning production systems with uptime and reliability expectations
• Proven ability to lead incident response and drive postmortems that result in real system improvements
• Solid data architecture and system design skills, including disaster recovery, high availability, and upgrades without downtime
• Experience implementing observability and data quality controls across complex data systems
• Demonstrated success defining and enforcing engineering standards adopted by other teams
• Strong cost management experience, including running optimization efforts and forecasting capacity
• Experience leading technical initiatives across teams without direct authority
• Experience implementing governance and compliance controls in regulated or high-risk environments
• Ability to mentor through standards, examples, and shared practices rather than formal people management
• 8+ years of data engineering experience, with several years owning critical platforms or services
• Experience leading AI and LLM adoption in data platforms, including validation of AI-generated pipelines and guardrail design
• Advanced Airflow experience, including reusable DAG patterns, scheduler tuning, and failure prevention
• Experience with dbt at scale, including testing standards, macros, and deployment practices
• Experience with OpenLineage, schemachange, or similar tools for lineage and schema management
• Experience leading platform migrations or modernization efforts with minimal disruption
• Experience streaming or near-real-time data systems such as Kafka, Kinesis, or Flink
• Experience running resilience testing such as failure injection or disaster recovery drills
• Cloud certifications such as AWS Certified Solutions Architect, AWS Data Analytics Specialty, or Snowflake SnowPro
• Bachelor's or master's degree in computer science, Data Engineering, or equivalent practical experience