Data Engineer

Data Engineer

Atlanta, GA

Long Term

Skills: DataFrames, Spark SQL,ETL, Databricks, ADF, Azure Data Factory,

Job Requirement:

Apache Spark Expertise

• Deep understanding of Spark concepts, including RDDs, DataFrames, Spark SQL, and performance tuning, to efficiently process large-scale data on the Databricks platform.

Programming Proficiency

• Strong coding skills in Python and SQL for developing robust ETL pipelines and data transformations within Databricks notebooks and jobs.

Delta Lake & Lakehouse Architecture

• Experience working with Delta Lake for reliable, ACID-compliant data storage and familiarity with Lakehouse principles for unified analytics. Strong Working knowledge of Medallion Architecture. Time Travel.

ETL & Data Pipeline Development

• Proven ability to design, build, and orchestrate scalable ETL pipelines for both batch and streaming data, ensuring data quality and reliability.

Databricks Platform Usage

• Hands-on experience with Databricks notebooks, jobs, clusters, and workspace management for collaborative data engineering workflows.

Data Modeling & Advanced SQL

• Ability to design efficient data models and write complex SQL queries for data analysis and transformation within Databricks

Cloud Platform Knowledge

• . Understanding of cloud infrastructure (Azure or Google Cloud Platform) as it relates to Databricks deployments, including storage, networking, and security best practices.

Workflow Orchestration & Automation

• Experience with orchestration tools (e.g., Databricks Lake flow, Airflow, or Azure Data Factory) to automate and schedule data workflows. Asset bundles

Collaboration & Version Control

• Strong collaboration skills and experience using version control systems (such as Git) for code management and teamwork. Experience working with Git hub actions and using Git to store the code required for moving the data within Databricks environment

Skillset Requirements

• Performance Optimization & Troubleshooting Ability to monitor, troubleshoot, and optimize Spark jobs and Databricks workflows for efficiency and cost-effectiveness. Al Proficiency Al Proficiency in Databricks /Data science