Staff Machine Learning Engineer

People In AI is a research-driven organization operating in a high-stakes decision-making environment, seeking a Staff Machine Learning Evaluation Engineer. This role focuses on designing and owning the evaluation layer for large language models and AI systems, ensuring trust and quality in consequential decision-making.

Responsibilities:

• Building and maintaining golden evaluation datasets grounded in expert judgment
• Designing offline benchmarks to compare models, prompts, and retrieval strategies
• Defining quality metrics that go beyond surface-level accuracy
• Partnering closely with researchers and domain experts to translate intuition into measurable criteria
• Connecting offline evaluation results with online behavior once systems are live
• Detecting regressions, drift, and subtle failures over time
• Iterating on evaluation frameworks as models and use cases evolve

Requirements:

• Experience designing offline evaluation or experimentation frameworks
• Deep understanding of online vs offline mismatch and how to manage it
• Ownership of model monitoring, drift detection, or regression analysis
• Comfort working with ambiguous, qualitative outputs (e.g. LLMs)
• Experience partnering with domain experts or stakeholders to define 'what good looks like'
• Strong Python and data skills; comfort building lightweight pipelines and analysis tooling
• Hands-on experience with LLMs is highly relevant, especially around LLM evaluation and benchmarking
• RAG evaluation
• Hallucination or grounding checks
• Prompt or system regression testing
• Human-in-the-loop review workflows
• Applied ML engineers
• ML tech leads
• Research engineers with production exposure
• Engineers from decisioning, risk, pricing, trust & safety, or ranking systems