Director of Data Engineering | Allconnect

Red Ventures is a global portfolio of high-growth companies, and they are seeking a Director of Data Engineering to lead the team responsible for building and scaling their data infrastructure. The role involves setting technical direction, managing engineering teams, and collaborating with business leaders to ensure effective data usage across the organization.

Responsibilities:

• Lead a team of data engineers and engineering managers building large-scale, real-time data pipelines in Spark
• Act as the primary technical liaison to business leadership, translating trade-offs between technology and business priorities in both directions
• Set the multi-year technical direction for Data Engineering and communicate that strategy across both technical and business audiences
• Partner with data scientists, analysts, and business stakeholders to turn business requirements into data products the team builds against
• Represent Data Engineering in cross-functional and executive planning, building alignment across engineering, product, and business leaders
• Set the architectural direction and standards for how data moves across platforms, from streaming ingestion through ETL to aggregation and analytics, hold your managers and teams accountable to that direction, and establish the engineering and agile practices that support delivery speed and quality
• Stay close to the technical details to guide architecture design decisions, performance tuning, and tradeoff calls, and to give your team the context they need to move quickly without waiting on you
• Work with your teams and the RV Data Platform team to design scalable solutions across distributed systems using AWS, Databricks, and modern big data technologies
• Champion a governance-first mindset across your team, making sure the way we collect, move, and use data protects the privacy of the people behind it
• Manage and develop the data engineering managers on your team, setting the bar for how they coach, hire, and run their teams
• Own career development across your org, building growth paths for individual contributors and managers alike, and coaching through regular, direct feedback
• Hold the team accountable for delivering high-quality, scalable solutions across the full data lifecycle

Requirements:

• 8-10 years of data or software engineering experience, including 5+ years managing engineering teams and at least 1 year managing managers
• Track record of communicating technical strategy and tradeoffs clearly to non-technical executives, and building trust quickly with cross-functional peers
• Deep curiosity about the systems and processes that produce our data, with enough understanding of the source to help your team strengthen and troubleshoot pipelines rather than treating data as a black box
• Track record of leading engineering teams in an agile environment, coaching both engineers and managers, and giving teams enough context to operate independently
• Deep hands-on experience building large-scale Spark applications for batch and streaming workloads
• Experience with modern lakehouse platforms (Databricks or similar) in addition to traditional Spark-based pipelines
• Expertise with the Spark RDD and DataFrame/Dataset APIs, with emphasis on DataFrames, for querying and data manipulation
• Experience with Spark Structured Streaming and SparkSQL, including broadcast joins
• Strong programming background in Python and/or Scala, with SQL fluency across large-scale datasets
• Hands-on experience with AWS data services (Kinesis, S3, Lambda, DynamoDB)
• Experience with ANSI SQL relational databases (Oracle, SQL Server, Postgres, MySQL)
• Strong grounding in distributed systems and distributed data processing frameworks