Senior Data Engineer

Brooksource is seeking a highly skilled Senior Data Engineer to join their dynamic data team. In this role, you will be responsible for designing, developing, and maintaining scalable data pipelines and architectures that support advanced analytics and business intelligence initiatives.

Responsibilities:

• Design, develop, and optimize large-scale ETL pipelines utilizing tools such as Informatica, Talend, Apache Spark, and Apache Hive
• Build and maintain robust data architectures on cloud platforms including AWS, Azure Data Lake, and Public Cloud environments
• Collaborate with cross-functional teams to gather requirements and translate them into scalable data solutions
• Develop and implement data models using dimensional modeling techniques for data warehousing and analytics purposes
• Manage data integration processes involving linked data, RESTful APIs, and other external sources
• Ensure data quality, security, and compliance standards are met throughout the data lifecycle
• Support the deployment of machine learning models through efficient data pipeline integration
• Utilize SQL, Python, Java, Bash scripting, and Shell Scripting for automation and process enhancement
• Contribute to agile development cycles by participating in sprint planning, reviews, and continuous improvement initiatives
• Maintain documentation of data workflows and architecture diagrams for transparency and knowledge sharing

Requirements:

• Proven experience as a Data Engineer or similar role with a focus on big data systems and cloud-based architectures
• Extensive knowledge of AWS services such as S3, Redshift, Glue; Azure Data Lake; Hadoop ecosystem components; and other public cloud offerings
• Strong proficiency in SQL databases including Microsoft SQL Server, Oracle, and cloud databases
• Hands-on experience with ETL/ELT development using tools like Informatica or Talend
• Demonstrated expertise in data modeling techniques including dimensional modeling for enterprise data warehouses
• Skilled in programming languages such as Python, Java, VBA, Bash scripting for automation tasks
• Design, develop, and optimize large-scale ETL pipelines utilizing tools such as Informatica, Talend, Apache Spark, and Apache Hive
• Build and maintain robust data architectures on cloud platforms including AWS, Azure Data Lake, and Public Cloud environments
• Collaborate with cross-functional teams to gather requirements and translate them into scalable data solutions
• Develop and implement data models using dimensional modeling techniques for data warehousing and analytics purposes
• Manage data integration processes involving linked data, RESTful APIs, and other external sources
• Ensure data quality, security, and compliance standards are met throughout the data lifecycle
• Support the deployment of machine learning models through efficient data pipeline integration
• Utilize SQL, Python, Java, Bash scripting, and Shell Scripting for automation and process enhancement
• Contribute to agile development cycles by participating in sprint planning, reviews, and continuous improvement initiatives
• Maintain documentation of data workflows and architecture diagrams for transparency and knowledge sharing
• Familiarity with business intelligence tools like Looker or similar platforms for reporting and analysis
• Understanding of RESTful API integration for linked data access
• Knowledge of analysis skills related to database design, query management, model training, and analytics methodologies
• Experience working within Agile methodologies to deliver iterative value in fast-paced environments