Senior Data Engineer – Streaming, Analytics & Visualization

enGen is seeking a Senior Data Engineer to design and operate real-time and analytical data pipelines that support enterprise reporting and decision-making. The role focuses on streaming and batch data engineering in Azure Data Explorer, data governance and security, and developing curated datasets, analytics, and Power BI or Tableau dashboards.

Responsibilities:

• Design, develop, and maintain robust data processes and solutions to ensure the efficient movement and transformation of data across multiple systems
• Develop and maintain data models, databases, and data warehouses to support business intelligence and analytics needs
• Collaborate with stakeholders across IT, product, analytics, and business teams to gather requirements and provide data solutions that meet organizational needs
• Monitor work against production schedule, provide progress updates, and report any issues or technical difficulties to lead developers regularly
• Implement and manage data governance practices, ensuring data quality, integrity, and compliance with relevant regulations
• Collaborate on the design and implementation of data security measures, including access controls, encryption, and data masking
• Mentor other associate and intermediate data engineers as needed
• Perform data analysis and provide insights to support decision-making across various departments
• Stay current with industry trends and emerging technologies in data engineering, recommending new tools and best practices as needed
• Other duties as assigned or requested

Requirements:

• CANDIDATE MUST BE US Citizen (due to contractual/access requirements)
• 5 years of experience in design and analysis of algorithms, data structures, and design patterns in the building and deploying of scalable, highly available systems
• 5 years of experience in a data engineering, ETL development, or data management role
• 5 years of experience in SQL and experience with database technologies (e.g., MySQL, PostgreSQL, MongoDB)
• 5 years of experience in data warehousing concepts and experience with data warehouse solutions (e.g., Snowflake, Redshift, BigQuery)
• Demonstrated ability to achieve stretch goals in a highly innovative and fast-paced environment
• Adaptability: Strong ability to take on diverse tasks and projects, adapting to the evolving needs of the organization
• Analytical Thinking: Strong analytical skills with a focus on detail and accuracy
• Interest and ability to learn other data development technologies/languages as needed
• Technical Proficiency: Comfortable with a range of data tools and technologies, with a willingness to learn new skills as needed
• Strong track record in designing and implementing large-scale data sources
• Strong sense of ownership, urgency, and drive
• Demonstrated passion for user experience and improving usability
• Team Collaboration: A team player who can work effectively in cross-functional environments
• Experience and willingness to mentor junior data engineers and help develop their skills and leadership
• Bachelor's degree in Computer Science, Information Systems, Data Science, Computer Engineering or related field
• 7 years of experience with data pipeline and workflow management tools (e.g., Apache Airflow, AWS Glue, Azure Data Factory)
• 7 years of experience defining system architectures and exploring technical feasibility trade-offs for optimizing short term execution while planning for long term technical capabilities
• 7 years of experience working with a variety of technology systems, designing solutions or developing data solutions in healthcare
• 7 years of experience with cloud platforms (AWS, Azure, GCP) and their respective data services
• 7 years of experience in data governance, data quality, and data security best practices
• 7 years of experience translating requirements, design mockups, prototypes or user stories into technical designs
• 7 years of experience in producing data-related code that is fault-tolerant, efficient, and maintainable
• 7 years of experience structuring and transforming data from multiple source systems
• 7 years of experience creating data visualizations and dashboards using Power BI or Tableau for internal and external audiences
• 7 years of experience using Cribl as a steaming and transformation solution
• 7 years of experience using Azure Data Factory as a transformation solution
• 5 years of experience using Azure Data Explorer
• Master's degree in Computer Science, Information Systems, Data Science, Computer Engineering or related field