Job Title: Lead Data Engineer
Location: Remote
Visa: USC
Duration: Longterm
Job Summary:
The Lead Data Engineer is responsible for designing, developing, and leading the implementation of scalable data engineering solutions using Microsoft Fabric. The ideal candidate should have strong expertise in modern data engineering practices, ETL/ELT architecture, cloud-based data platforms, data warehousing, and data governance. The role also involves providing technical leadership, ensuring adherence to best practices, and driving the successful delivery of enterprise data solutions.
Key Responsibilities:
Lead the design and implementation of enterprise-scale data engineering solutions using Microsoft Fabric.
Design, develop, and optimize ETL/ELT pipelines for processing.
Architect and implement metadata-driven data ingestion and transformation frameworks.
Develop and maintain scalable PySpark notebooks for data processing.
Design and implement robust data warehousing solutions using Microsoft Fabric.
Provision and manage Microsoft Fabric workspaces, ensuring appropriate security, governance, and environment configuration.
Implement and manage data governance, cataloging, metadata management, and data lineage using Microsoft Purview.
Ensure data quality, security, governance, and compliance, including handling Personally Identifiable Information (PII).
Collaborate with business stakeholders, solution architects, and reporting teams to understand business requirements and translate them into technical solutions.
Optimize data pipelines for performance, scalability, reliability, and maintainability.
Support Power BI reporting by developing optimized data models and curated datasets.
Work closely with DevOps teams to establish CI/CD pipelines, deployment strategies, and release management processes.
Required Qualifications:
Bachelor’s degree in computer science, Information Technology, Engineering, or a related discipline.
8+ years of experience in Data Engineering with at least 3–4 years of hands-on experience in Microsoft Fabric or similar cloud-based data platforms.
Proven experience in designing and implementing enterprise ETL/ELT solutions.
Strong understanding of modern data architecture and cloud-based analytics platforms.
Hands-on experience with Microsoft Purview for data governance, data cataloging, metadata management, and data lineage.
Experience provisioning and administering Microsoft Fabric workspaces.
Good understanding of metadata management concepts and their implementation in relational database systems.
Familiarity with DevOps methodologies, CI/CD processes, Git-based source control, and deployment automation.
Strong analytical, problem-solving, leadership, and communication skills.
Technical Skills:
Microsoft Fabric
Microsoft Purview
Fabric Workspace Provisioning and Administration
Fabric Data Pipelines
PySpark Notebooks
Metadata-Driven Design
Metadata Management in Relational Databases
OneLake
Fabric Data Warehouse
Data Engineering and ETL/ELT Architecture
Data Modeling
SQL
Python
Power BI
PII Handling and Data Governance
Azure DevOps
Git
CI/CD Pipeline Implementation
Performance Optimization and Data Pipeline Monitoring
