Senior Data Pipeline Engineer

Candid is a nonprofit providing data and insights about the social sector. The Senior Data Pipeline Engineer develops and delivers data pipelines, moving data reliably from source systems into Candid’s data warehouse and lakehouse using SQL, ETL engineering, and cloud data platform technologies.

Responsibilities:

• Pipeline development: Architect, develop, and maintain ETL/ELT and C#/.NET pipeline services (SSIS, Airflow, AWS Lambda, enhancer and message-processing services) that move data reliably from source systems into Candid's warehouse and lakehouse
• Cloud platform & performance: Support Candid's evolving cloud data platform, including Iceberg ingestion and transformation pipelines, DynamoDB, and query performance tuning via Trino/Starburst
• Data quality & incident resolution: Monitor data quality end-to-end and investigate production pipeline issues at the root cause, documenting findings and driving fixes through to reprocessing
• Source onboarding & schema coordination: Scope and onboard new external and regulatory data sources end-to-end, and contribute to schema and architecture decisions, coordinating changes across ingestion and transformation layers
• Documentation, mentorship & process: Maintain SOP and technical documentation, mentor and review the work of Associate-level engineers, and participate in the team's release and deployment process
• Partnership & support: Partner with the Data Operations Manager on architecture and roadmap decisions, and provide escalation-level support for mission-critical pipeline issues, including after-hours as needed

Requirements:

• Applicants for this position must be legally authorized to work in the United States without employer sponsorship, now or in the future
• 5+ years of experience in database engineering, data engineering, or ETL development, with demonstrated ownership of production data pipelines
• Expert-level Transact-SQL skills (stored procedures, views, indexing) and hands-on C#/.NET development experience
• Proven experience architecting ETL processes with SSIS
• Experience with AWS and open table formats (Apache Iceberg, Parquet)
• Strong data quality and root-cause troubleshooting skills, with experience resolving production data incidents
• Familiarity with source control, CI/CD, and release/deployment processes across Dev/QA/Prod environments
• Ability to mentor and review the work of other engineers
• excellent written and verbal communication
• Good business judgment, independent prioritization, and the ability to work well in a distributed team
• Sensitivity and respect for racial, gender, sexual orientation, and cultural differences
• Commitment to Candid's values: driven, direct, accessible, curious, and inclusive
• experience with orchestration platforms such as Apache Airflow
• AWS Lambda, DynamoDB, and Trino/Starburst a plus