Senior Lead Data Engineer - Part Time - 24 hours per week

Lighthouse is hiring a Senior Lead Data Engineer for its Prism team to set technical direction and build core data engineering infrastructure for eDiscovery AI products. The role owns scalable data pipelines and orchestration for document processing, machine learning workflows, and generative AI applications, while partnering with other teams and mentoring data engineers.

Responsibilities:

• Set the technical direction and own the end-to-end architecture for data engineering across Prism eDiscovery AI products, ensuring designs are scalable, reliable, secure, and cost-effective
• Lead the design, build, and management of data preprocessing, validation, and transformation pipelines that extract metadata and full text from heterogeneous ESI corpora at multi-million document scale
• Lead the build and operation of Data Orchestration platforms that coordinate training data preparation, model scoring, embedding generation, and result delivery across eDiscovery AI workflows
• Architect and implement scalable, performant data pipelines that cleanse, structure, and integrate disparate ESI datasets into optimal formats for encoder model training (BERT fine-tuning) and LLM context preparation
• Direct the design and evolution of infrastructure for LLM/Generative AI applications: document chunking, embedding generation pipelines, vector database population, and retrieval layer optimization for RAG architectures
• Define and drive data engineering standards, best practices, and design patterns, including code quality, testing, documentation, and design reviews
• Establish the approach and tooling to monitor, debug, and analyze data pipeline health, data quality, and model serving data flows in production, and lead resolution of complex production issues
• Design and govern portable, scalable data schemas for document metadata, model inputs/outputs, scoring results, and embeddings
• Lead experiments to validate efficiency, throughput, and cost of modern data engineering stacks for legal AI workloads and translate findings into roadmap recommendations
• Own technical recommendations regarding buy vs. build decisions for data infrastructure components and present them to engineering and product leadership
• Partner with data science, product, and engineering stakeholders to shape requirements, plan, and sequence technical work, and align data engineering priorities with business goals
• Mentor and provide technical guidance to data engineers, fostering skill growth and consistent engineering practices across the team
• Perform other related duties as assigned

Requirements:

• Degree in computer science or computer engineering with 8+ years of experience in a data engineering related field, including experience leading the technical design of large-scale data platforms
• Demonstrated technical leadership: setting architecture direction, leading design reviews, and guiding engineers on complex, cross-team initiatives
• Expertise building data pipelines for unstructured document data: text extraction, OCR processing, metadata extraction, and legal document format handling
• Experience implementing big data workflows in cloud-native tech stacks (Azure preferred): Spark, Databricks, Airflow, Azure Data Factory, Kafka, and Kubernetes
• Deep knowledge of data science and ML pipeline patterns: training data preparation, feature engineering, embedding generation, and MLOps operationalization
• Experience with vector databases and semantic search infrastructure (Azure AI Search, Pinecone, or similar) for RAG pipeline support
• Experience with relational and NoSQL databases, cloud-native storage technologies, and data serialization formats (Parquet, Delta Lake, JSON)
• Experience working with data scientists to operationalize encoder-based classification models and LLM applications in production
• Strong communication skills, with the ability to explain complex technical trade-offs to technical and non-technical stakeholders
• Proficiency with agile development methodologies