Data Engineer

Job Summary

We are seeking an experienced Senior Data Engineer with strong expertise in Python, Java, PySpark, SQL, and cloud technologies such as Google Cloud Platform/AWS/ Microsoft Azure.

The ideal candidate will design, develop, and maintain scalable data engineering pipelines, build data models and data marts, and support data integration, analytics, and machine learning initiatives across large-scale cloud environments.

Key Responsibilities

• Design and maintain scalable data pipelines using Python, Java, PySpark, and Apache Spark.
• Develop and optimize ETL processes for integrating data from multiple sources.
• Build data marts, data models, and large-scale data structures for analytics and reporting.
• Develop cloud-based data solutions using Google Cloud Platform or Azure.
• Work with BigQuery, Cloud Storage, Dataproc, Airflow/Cloud Composer, and Apache Kafka.
• Develop batch and real-time data ingestion and processing pipelines.
• Optimize SQL queries and work with relational and NoSQL databases.
• Ensure data quality, consistency, reliability, and accessibility.
• Implement CI/CD-enabled data pipelines, automated testing, and deployment processes.
• Collaborate with Data Science and cross-functional teams to integrate machine learning models into automated workflows.
• Troubleshoot complex data engineering challenges and improve performance, scalability, and cloud architecture.

Required Skills

• Strong programming experience in Python and/or Java.
• Hands-on experience with PySpark and Apache Spark.
• Experience with Google Cloud Platform or Azure cloud environments.
• Strong knowledge of ETL, data pipeline development, and data integration.
• Proficiency in SQL, MySQL, and NoSQL databases.
• Experience with data modeling, large-scale data processing, and data quality.
• Understanding of batch processing and real-time data ingestion.

Preferred Skills

• Apache Airflow, Cloud Composer, and Apache Kafka.
• CI/CD pipelines, Bash scripting, and UNIX utilities.
• REST APIs, microservices, and machine learning integration.
• Tableau and cloud architecture optimization.

Education

Bachelor’s degree in Computer Science, Engineering, Machine Learning, or a related STEM discipline, or equivalent work experience. A Master’s degree or PhD is preferred.

Interested candidates: Please share your updated resume along with your current location, work authorization, contact details, and availability.