Responsibilities
• Design and Implement scalable solutions for ever-increasing data volumes, using big data/cloud technologies like Pyspark, Kafka, etc.Collaborate with cross-functional teams to understand data requirements and provide effective solutions.
• Implement real-time data ingestion and processing solutions
• Develop and maintain ETL/ELT processes to support data analytics and reporting.
• Implement best practices for data security, integrity, and quality.
• Developing data integration and migration strategies to move data from legacy systems to Cloud
• Monitoring and troubleshooting the performance and availability of the data systems and implementing strategies to improve their reliability and scalability
• Managing security and access controls for the data systems, including managing roles and permissions, implementing encryption, and complying with regulatory requirements.
Requirements
· Bachelor’s in Engineering / Master’s degree in Computer Science, Information Systems or related field
· Minimum of 5 - 9 years of experience in data engineering.
· Experience with databases and data warehouse implementations (Preferred – Snowflake, PostGreSQL)
· Strong SQL skills with experience in writing and optimising complex queries
· Working knowledge of Data warehousing, Data Governance, Data Quality and Data Architecture
· Experience in Data Modelling
· Ability to handle large scale structured and unstructured data from internal and third-party sources
· Hands On Experience in Python, Pyspark, Kafka
· Experience with data engineering tools/technologies in Google Cloud Platform Cloud environment
· Proficiency in designing and maintaining scalable data architectures
· Experience with CICD Tools like GitHub
