Elfonze TechnologiesRemote
Data Engineer, Databricks, DataStage
India
Full Time
3 weeks ago
No Sponsorship
Key skills
ETLPySparkDatabricksUnit TestingIntegration TestingGitCI/CD
About this role
Role Overview
- Analyze existing DataStage server and parallel jobs
- Convert ETL logic to SparkSQL/PySpark on Databricks
- Implement metadata-driven ingestion patterns
- Support framework-based development and reusable templates
- Perform unit testing, integration testing, and reconciliation
- Optimize jobs using Delta Lake, partitioning, Z-ordering
- Support CI/CD pipelines (Git-based)
- Collaborate with architects and DataStage SMEs
Requirements
- Analyze existing DataStage server and parallel jobs
- Convert ETL logic to SparkSQL/PySpark on Databricks
- Implement metadata-driven ingestion patterns
- Support framework-based development and reusable templates
- Perform unit testing, integration testing, and reconciliation
- Optimize jobs using Delta Lake, partitioning, Z-ordering
- Support CI/CD pipelines (Git-based)
- Collaborate with architects and DataStage SMEs
Tech Stack
- ETL
- PySpark
Benefits
- Professional development opportunities