Key skills
Data EngineeringAnalyticsBigQuerySparkAirflowdbtBigTableSpannerPub/SubPostgresAgileCommunication
About this role
Egen is a fast-growing company with a data-first mindset, seeking a seasoned Data Engineer to join their team. The role involves designing and building large-scale data processing pipelines and ensuring data accessibility, security, and accuracy while collaborating with various stakeholders.
Responsibilities:
- Design, develop, and deploy large scale data processing pipelines, both batch and streaming, using technologies such as Dataflow, Apache Beam, Spark, Akka, Pub/Sub
- Expertise with multiple data storage technologies such as Bigtable/HBase, BigQuery, Spanner, CloudSQL/Postgres
- Work with stakeholders to understand business problems, develop use-cases, and translate them into pragmatic and effective technical solutions
- Design and develop appropriate schema for data based on understanding of the domain problem
- Manage data lineage and ensure data security with appropriate tools and methodologies
- Collaborate with data scientists, architects, and other stakeholders to ensure alignment between technical and business strategy
- Continuously monitor, refine and report on the performance of data management systems
- Mentor junior data engineers, reviewing their outputs and directing their professional development
Requirements:
- 2 - 5 years of experience in data engineering
- Deep understanding of large-scale data processing systems
- Experience in designing and building both batch and streaming data pipelines
- Design, develop, and deploy large scale data processing pipelines using technologies such as Dataflow, Apache Beam, Spark, Akka, Pub/Sub
- Expertise with multiple data storage technologies such as Bigtable/HBase, BigQuery, Spanner, CloudSQL/Postgres
- Work with stakeholders to understand business problems, develop use-cases, and translate them into pragmatic and effective technical solutions
- Design and develop appropriate schema for data based on understanding of the domain problem
- Manage data lineage and ensure data security with appropriate tools and methodologies
- Collaborate with data scientists, architects, and other stakeholders to ensure alignment between technical and business strategy
- Continuously monitor, refine and report on the performance of data management systems
- Mentor junior data engineers, reviewing their outputs and directing their professional development
- Proven expertise with technologies such as Dataflow, Apache Beam, Spark, Akka, Pub/Sub
- Ability to design data schemas based on an understanding of the domain problem
- Exceptional communication skills, able to explain complex technical concepts in clear, plain English
- BSc degree in Computer Science, Engineering or a related field, or equivalent work experience
- Experience with data security and data lineage methodologies and tools
- Familiarity with agile development methodologies
- Experience with data migration projects
- Knowledge of dbt, Airflow, or similar orchestration tools
- Experience in multi-cloud environments
- Familiarity with data modeling and analytics use cases