Lead Data Engineer

Reuben Cooley Inc. is seeking a Lead Data Engineer to design and deliver enterprise-scale data platforms and production-ready data engineering solutions. The role focuses on lakehouse architecture, Azure data technologies, LLM-based systems, data governance, CI/CD, infrastructure as code, and communicating technical designs to technical and non-technical stakeholders.

Responsibilities:

• Expert - level proficiency in Python, Scala, and PySpark, with a strong track record of designing and delivering production-ready, modular, and well-tested solutions; developing and troubleshooting Spark workloads; and optimizing large-scale batch and streaming data pipelines using Delta Lake and Spark technologies
• Strong SQL and data modelling dimensional and normalised; schema design and data contract definition
• Databricks expertise Delta Lake, Unity Catalog, Jobs & Workflows, cluster and pool management, performance tuning, Model Serving
• Azure data stack — ADLS Gen2 (zone design, ACLs, lifecycle), Azure Data Factory (parameterized / metadata-driven frameworks, error handling), Azure Event Hubs
• 3+ years designing and shipping LLM-based systems in production: RAG pipelines, agentic / tool-calling workflows, structured output, chunking and embedding strategy, vector and hybrid retrieval, and prompt engineering
• Evaluation discipline — golden datasets, regression suites, accuracy and hallucination tracking, human-in-the-loop feedback loops; you measure AI quality, not assert it
• Hands-on experience with LangChain, LlamaIndex, or LangGraph, plus at least one provider stack (Azure OpenAI, OpenAI, or Databricks Model Serving)
• Metadata-driven thinking — schema inference, data profiling, lineage, catalogs, and configuration-driven frameworks that onboard the next source without new code
• 12–18 years of total experience in data engineering, data platform delivery, or related disciplines
• Proven delivery of a medallion/lakehouse architecture at enterprise scale — not just familiarity with the concept
• Azure security and governance: Entra ID, managed identities, RBAC, POSIX ACLs on ADLS Gen2, Key Vault, private endpoints, and PII handling
• CI/CD and infrastructure as code: Azure DevOps, Terraform, Databricks Asset Bundles, and automated testing of data pipelines
• Clear technical writing and the ability to present and defend a design to both engineers and non-technical stakeholders

Requirements:

• Expert - level proficiency in Python, Scala, and PySpark, with a strong track record of designing and delivering production-ready, modular, and well-tested solutions; developing and troubleshooting Spark workloads; and optimizing large-scale batch and streaming data pipelines using Delta Lake and Spark technologies
• Strong SQL and data modelling dimensional and normalised; schema design and data contract definition
• Databricks expertise Delta Lake, Unity Catalog, Jobs & Workflows, cluster and pool management, performance tuning, Model Serving
• Azure data stack — ADLS Gen2 (zone design, ACLs, lifecycle), Azure Data Factory (parameterized / metadata-driven frameworks, error handling), Azure Event Hubs
• 3+ years designing and shipping LLM-based systems in production: RAG pipelines, agentic / tool-calling workflows, structured output, chunking and embedding strategy, vector and hybrid retrieval, and prompt engineering
• Evaluation discipline — golden datasets, regression suites, accuracy and hallucination tracking, human-in-the-loop feedback loops; you measure AI quality, not assert it
• Hands-on experience with LangChain, LlamaIndex, or LangGraph, plus at least one provider stack (Azure OpenAI, OpenAI, or Databricks Model Serving)
• Metadata-driven thinking — schema inference, data profiling, lineage, catalogs, and configuration-driven frameworks that onboard the next source without new code
• 12–18 years of total experience in data engineering, data platform delivery, or related disciplines
• Proven delivery of a medallion/lakehouse architecture at enterprise scale — not just familiarity with the concept
• Azure security and governance: Entra ID, managed identities, RBAC, POSIX ACLs on ADLS Gen2, Key Vault, private endpoints, and PII handling
• CI/CD and infrastructure as code: Azure DevOps, Terraform, Databricks Asset Bundles, and automated testing of data pipelines
• Clear technical writing and the ability to present and defend a design to both engineers and non-technical stakeholders