Platform Data Engineering Lead

Boston Insights is an innovative startup creating competitive advantage for pharmaceutical companies by unlocking clinical supply chain data and enabling end-to-end visibility. The Platform Data Engineering Lead will lead the design, development, governance, automation, and maintenance of scalable multi-cloud data platforms and pipelines, including Microsoft Fabric AI agents, to support pharmaceutical supply chain analytics and optimization.

Responsibilities:

• Lead the integration strategy for ingesting and harmonizing data from external sources, including AWS-based data lakes/warehouses (such as S3, Redshift) and SAP systems
• Design, build, and maintain scalable data pipelines using Azure Data Factory, Azure Synapse Analytics, and Azure Data Lake / MS Fabric OneLake
• Design reusable metadata-driven ingestion framework for rapid on-boarding of new datasets
• Hands-on experience in metadata driven pipelines with event-based ingestion
• Automate ETL processes for data extraction, transformation, and loading across hybrid and multi-cloud environments
• Build and maintain real-time and batch data integration workflows between Azure, AWS, and on-premises sources
• Design and implement data lake and data warehouse solutions on Azure platform
• Establish data governance frameworks and ensure data quality across all pipelines
• Implement security best practices for handling sensitive pharmaceutical data
• Create and maintain data documentation and lineage tracking
• Define and enforce data governance frameworks: data cataloging, lineage, quality, privacy, and compliance
• Implement robust data validation, cleansing, and monitoring systems to ensure accuracy and reliability
• Support security standards through effective data management practices
• Design and develop Microsoft Fabric-based AI agents leveraging OneLake, Fabric Data Agents, Semantic Models, Lakehouses, and enterprise data governance capabilities to support metadata discovery, lineage interpretation, policy validation, anomaly triage, and data stewardship workflows
• Develop data automation tools and reusable components using Python, PySpark (and other relevant frameworks/languages)
• Enable end-to-end process automation for data ingestion, processing, and reporting
• Implement CI/CD processes for data solutions, including testing, monitoring, and alerting
• Collaborate with data scientists and analysts to support advanced analytics
• Build data models that enable risk assessment and supply chain optimization
• Develop APIs and data services to support front-end applications
• Create monitoring and alerting systems for data pipeline health
• Build agents that support data cataloging, metadata enrichment, lineage interpretation, and data stewardship workflows
• Partner with supply chain, analytics, and business stakeholders to understand business requirements and translate them into scalable technical solutions
• Collaborate with SAP functional and technical teams to optimize data extraction and synchronization

Requirements:

• • 5+ years of professional data engineering experience
• • Data Integration: Proven track record integrating data from AWS services (S3, Redshift, Glue, etc.) into Azure or other cloud environments
• • Azure Data Services: Expert-level knowledge of Azure Data Factory, Azure Synapse Analytics, Data Bricks, Azure Data Lake Storage, Azure SQL Database, and Apache Spark
• • Database Technologies: Strong knowledge of both relational (SQL Server, PostgreSQL) and NoSQL (Cosmos DB) databases
• • Programming Languages: Proficiency in Python, SQL, PySpark, and PowerShell for data automation and wrangling
• Hands-on experience developing AI agents using Microsoft Fabric infrastructure and services, including Fabric Data Agents, Lakehouses, Semantic Models, OneLake, Notebooks, and Data Factory
• • Building agents that support data cataloging, metadata enrichment, lineage interpretation, and data stewardship workflows
• • Designing agent workflows, retrieval patterns, governance controls, and security guardrails for enterprise pharmaceutical and supply chain data
• • Integrating Fabric-based agents with data platforms, APIs, notebooks, pipelines, and automation frameworks to streamline data quality reviews, compliance checks, and anomaly detection
• • Evaluating agent responses for accuracy, traceability, governance compliance, security, and business relevance
• • Version Control: Proficiency with Git and Azure DevOps
• • Hands-on experience with SAP data models and integrating SAP data with Azure data lake
• • Strong problem-solving and analytical thinking abilities
• • Excellent communication skills with ability to explain technical concepts to non-technical stakeholders
• • Experience with Agile development methodologies
• • Attention to detail and commitment to data quality
• • Experience with API-based data integration for cloud and enterprise applications
• • Experience with Infrastructure as Code (ARM templates, Terraform)
• • Familiarity with data quality tools, metadata management, and automated data lineage tracking
• • Knowledge of containerization (Docker, Kubernetes) for data automation workflows
• • Knowledge of machine learning pipelines and MLOps practices
• • Experience with data visualization tool, Power BI