Senior Applied AI Researcher — Post-Training (RL) & Evaluation

Lumos is a health and life-science AI evaluation company based in San Francisco. They are seeking a Senior Applied AI Researcher to build systems that evaluate medical models and agents for safety and accuracy in healthcare settings. The role involves designing reinforcement learning environments, conducting post-training evaluations, and collaborating with clinical experts.

Responsibilities:

• Design and ship RL environments for medical agents — tasks, tool interfaces, reward and verification logic
• Run post-training — RLHF / RLAIF / RLVR, reward modeling, preference-data pipelines, SFT, and RL fine-tuning
• Build evaluations that feed back into training — verifiers, graders, benchmarks, hard gates, and failure-mode analysis that directly shape the next model
• Design and build RL environments for medical agents — defining tasks, tool interfaces, and the reward and verification logic that decides what 'correct' means
• Own post-training of LLMs and agents end to end: RLHF / RLAIF / RLVR, reward modeling, preference-data pipelines, SFT, and RL fine-tuning
• Design case-specific verifiers, graders, and rubrics that capture medical correctness — in a domain that actually has ground truth, where reward design is a rich, hard, and largely unsolved problem
• Build evaluation frameworks — benchmarks, scoring dimensions, hard gates, red-teaming, and failure-mode analysis — and close them back into the training loop
• Train and evaluate multi-step agentic behavior: tool use, trajectories, and long-horizon decisions, not single-turn outputs
• Partner with clinical subject-matter experts to encode medical correctness into automated signal