Data Engineer - Remote Charleston, SC - W2/Own corporation only

Job title: Data Engineer

Expert in SQL and Python for developing and debugging scalable data pipelines.

Deep hands-on experience with Azure and Databricks, including Delta Live Tables and Unity
Catalog

JOB DESCRIPTION

The Senior Data Engineer, leads the design and implementation of robust data solutions across multiple
domains, driving technical excellence and scalability. This role mentor others, shape best practices, and
influence data architecture. This role is expected to proactively identify opportunities to improve systems,
drive reliability, and collaborate with product and business stakeholder to again data strategy and company
goals.

Profile Description:

Design, build and scale robust, high-performing batch and real-time data pipelines. Drive
architectural decisions for transformation logic, storage formats, and schema design. Lead
complex data ingestion efforts and mentor peers on performance optimization and
scalability.
Lead the design and optimization of complex data models and storage architecture,
balancing performance, scalability, and usability. Partner with stakeholders to translate
business requirements into robust data structures.
Contribute significantly to delivery planning and execution, mentor junior engineers on agile
approaches, and ensure timely completion of tasks by managing dependencies and
escalating delivery challenges.
Design and standardize advanced data validation frameworks and testing strategies across
platforms. Lead root cause analysis and data quality issues and mentor others on quality
best practices. Partner with stakeholders to define SLAs and quality metrics.
Leads efforts to automate, monitor, and scale deployment of production-grade data
pipelines. Design resilient workflows with retry logic, failure handling, and resource
optimization. Proactively address performance and reliability issues and contribute to
runbooks and on-call documentation.
Lead the creation and maintenance of detailed technical documentation for complex
pipelines, data models, and system integrations. establish and enforce documentation and
development standards across projects. Mentor junior engineers on clear, consistent coding
and documentation habits.
Act as a key technical partner to product, analytics and data science teams. Lead design,
discussions, communicate complex data trade-offs with clarity, and proactively surface risks
and blockers. Support collaborative planning and mentor junior team members in effective
communication and partnership.

Knowledge & Experience:
5-9 years in data engineering, data modeling, and pipeline development
Expert in SQL and Python for developing and debugging scalable data pipelines.
Deep hands-on experience with Azure and Databricks, including Delta Live Tables and Unity
Catalog
Skilled with data integration/orchestration tools (SnapLogic, Azure Data Factory, Jenkins)
Strong use of infrastructure-as-code tools like Terraform to manage deployment pipelines.
Design and optimize API integration in pipelines.
Familiar with data quality observability tools such as Soda or similar.
Proficient in version control and CI/CD workflows using GitHub.
Advanced understanding of dimensional modeling and data warehousing concepts.
Comfortable leading efforts in agile environments and strong ownership and collaboration.