Senior Data Engineer

APN Consulting, Inc. is a progressive IT staffing and services company offering innovative business solutions to improve client business outcomes. The Senior Data Engineer will design, build, and operationalize scalable data pipelines and data products using a Python-based approach, ensuring alignment with enterprise architecture and governance goals.

Responsibilities:

• Develop pipelines and transformations using Python (PySpark / notebooks / scripts)
• Work within VS Code + GitHub development workflows
• Build ingestion, transformation, and curated data layers aligned to medallion architecture
• Integrate data from ERP, MES, OT, historian, and operational systems
• Convert notebooks into production-grade, reusable components
• Implement unit testing, logging, monitoring, and observability frameworks
• Follow branching strategies, pull request processes, and CI/CD pipelines
• Package reusable logic into shared modules and libraries
• Enable creation of certified, semantic-ready data products
• Optimize performance, troubleshoot failures, and ensure production reliability
• Maintain technical documentation and operational runbooks

Requirements:

• Strong proficiency in Python and PySpark-based data engineering
• Experience with VS Code, GitHub, and code-based pipeline development
• Strong experience with Azure Data Factory, Synapse, Microsoft Fabric, SQL
• Understanding of: Notebook vs. production pipeline design
• Understanding of: Code modularization and reuse
• Strong debugging, optimization, and problem-solving capabilities
• Manufacturing data experience
• Exposure to AI-ready datasets, feature engineering, and data observability