Research Scientist, Open Benchmarks Red Team

Snorkel AI is hiring a researcher for its Open Benchmarks Red Team program. The role identifies and reproduces benchmark weaknesses, develops and validates repairs and reusable detection methods, and collaborates with benchmark authors to improve and release reliable evaluations.

Responsibilities:

• Red-team frontier benchmarks: Identify and reproduce issues such as reward hacking, grader failures, task ambiguity, shortcuts, and environment vulnerabilities
• Develop benchmark repairs: Propose next steps for validated issues, implement selected fixes with benchmark authors, and verify fixes without breaking valid solutions
• Develop reusable methods: Reuse existing tools where possible and develop new checks to detect recurring benchmark weaknesses
• Collaborate with benchmark authors: Review findings with benchmark authors before publication, collaborate on repairs, and support official improved releases
• Drive research impact: Contribute to technical reports, open-source tools, publications, and new approaches to benchmark evaluation and improvement