Sr. Data Engineer

Celebal Technologies is seeking a Senior Data Engineer to deliver data engineering solutions focused on Databricks, Spark, and Big Data. The role involves building and optimizing data pipelines, orchestrating workflows, and ensuring data security, quality, and governance.

Responsibilities:

• Design, develop, and optimize data pipelines and ETL processes using Databricks and Apache Spark
• Implement and optimize performance on Databricks Spark, ensuring efficient data processing and management
• Develop and validate data formulation and data delivery for Big Data projects
• Collaborate with cross-functional teams to define, design, and implement data solutions that meet business requirements
• Conduct performance tuning and optimization of complex queries and data models
• Manage and orchestrate data workflows using tools such as Databricks Workflow, Azure Data Factory (ADF), Apache Airflow, and/or AWS Glue
• Maintain and ensure data security, quality, and governance throughout the data lifecycle

Requirements:

• Data engineering and analytics project delivery experience – Min. 4+ years
• Minimum one project done in past of Databricks Migration (Ex. Hadoop to Databricks, Teradata to Databricks, Oracle to Databricks, Talend to Databricks etc)
• Hands on with Advanced SQL and Pyspark and/or Scala Spark
• Extensive experience with PySpark and Scala-Spark
• Advanced SQL skills for complex data manipulation and querying
• Proven experience in performance optimization on Databricks Spark across at least three projects
• Hands-on experience with data formulation and data delivery validation in Big Data projects
• Experience in data orchestration using at least two of the following: Databricks Workflow, Azure Data Factory (ADF), Apache Airflow, AWS Glue
• Experience with cloud platforms such as AWS
• Familiarity with data governance and data security best practices
• Experience with other Big Data technologies and frameworks is a plus