Data Engineer

Data Engineer

Connecticut (Remote)

Contract - 1 Year

Job Description:

• Review and translate requirements to end-end technical design with a goal of building data products
• Develop pipelines in Databricks using PySpark and DBT to onboard various sources
• Working knowledge in Kafka to stream the data into databricks platform
• Identify Common approaches and best practices to data integration such as metadata drive ingestion, data validations, complex data type handling
• Drive standards for reusability, while developing pipelines
• Create or update documentation associated to key decisions on requirements, design and development
• Liaise with architects, data owners and users for reviews on day-day basis to ensure the quality of deliverables are high.

Skill Proficiency Requirements:

• 8+ years of overall IT experience with 6 years as a Data Engineer
• Experience in providing data engineering solutions
• Working knowledge of below technology toolsets:
  • Cloud (Azure/AWS)
  • Databricks, Data Mesh, Data as a Product
  • Python
  • Data streaming (Kafka, EventHub, Kinesis, SNS)
  • Cloud-based API management good to have
  • Immuta good to have
• Multi-cloud and on-prem application integration good to have