Senior Data Engineer

Insomniac Design is a global digital agency headquartered in Washington D.C., specializing in human-centered design and digital transformation. The Senior Data Engineer will architect and optimize scalable data platforms, leading the design of data pipelines and ensuring best practices in data governance and security while mentoring junior engineers.

Responsibilities:

• Architect and implement enterprise-grade data platforms, including data lakes, warehouses, and real-time pipelines
• Define and enforce data engineering standards, including ETL/ELT frameworks, data modeling, and quality controls
• Lead performance optimization of databases, queries, and pipelines (e.g., partitioning, indexing, cost-efficient cloud solutions)
• Stay updated with emerging trends, evaluating and integrating new technologies
• Design and implement scalable data architecture that ensures high availability, security, and performance
• Define best practices for data modeling, storage, retrieval, and governance
• Optimize database structures, schemas, indexing, and partitioning for efficient query performance
• Implement CI/CD for data pipelines and automate infrastructure using Infrastructure-as-Code (IaC) tools
• Architect and develop end-to-end ETL/ELT pipelines for structured and unstructured data ingestion, processing, and transformation
• Automate and optimize data workflows using tools like Apache Spark, Kafka, Airflow, and dbt
• Monitor, troubleshoot, and optimize data pipeline performance
• Implement data governance policies, including metadata management, lineage tracking, and role-based access control (RBAC)
• Ensure compliance with data privacy regulations (e.g., GDPR, CCPA) and enforce security best practices
• Develop frameworks for data quality validation, anomaly detection, and integrity checks
• Collaborate with stakeholders (clients, PMs) to translate business needs into technical specifications and scalable architectures
• Develop interactive dashboards and reporting solutions to provide actionable insights
• Lead technical discussions, workshops, and knowledge-sharing sessions within the team
• Mentor junior engineers, conduct code reviews, and set best practices
• Drive automation of data workflows, including CI/CD for pipelines and Infrastructure-as-Code (IaC) solutions

Requirements:

• 7+ years of experience in data engineering with a proven track record of building scalable data solutions
• Expertise in SQL and relational database management systems (MySQL, PostgreSQL, Oracle)
• Proficiency in Python (or similar languages) for data processing and automation
• Deep understanding of cloud platforms (AWS, GCP, Azure) and cloud-native data services: AWS (Redshift, Glue, S3), Azure (Synapse, Data Factory), GCP (BigQuery, Dataflow)
• Hands-on experience with big data tools: Apache Spark, Kafka, Airflow, dbt, Snowflake, Databricks
• Expertise in data modeling
• Strong knowledge of data security, governance, and compliance frameworks
• Proven track record of leading data projects, including migrations, warehousing, and real-time analytics
• Ability to lead cross-functional teams and manage client expectations
• Strong problem-solving skills for debugging distributed systems and resolving data anomalies
• Experience with Agile/Scrum methodologies, including tools like Jira, Git
• Excellent communication and stakeholder management skills
• Certifications in cloud/data technologies (e.g., AWS Certified Data Analytics, Google Cloud Data Engineer)
• Experience with ML pipelines (e.g., feature stores, model deployment)
• Contributions to open-source data projects