AI/ML Engineer

Lemma provides production monitoring for AI agents, identifying silent failures that traditional observability tools and evaluations miss. The AI/ML Engineer will own detection quality, build evaluations, turn implicit user signals into evidence, make patch generation trustworthy, control LLM-related costs, and analyze real customer traces.

Responsibilities:

• Own detection quality. Find the failures that don't look like failures: compliant but wrong, omissions, and patterns that only show up across thousands of traces
• Turn implicit signals into evidence. Rephrasing, abandonment, retries, and the other ways users tell you something broke without saying so
• Build the evals for our own system. If we can't measure precision on a problem with no labels, we can't improve it
• Make patch generation trustworthy. Reproduce the failure, verify the fix, and know when to stay quiet instead of opening a bad PR
• Keep it affordable. LLM-as-judge on every event is easy. Doing it at a cost per event that doesn't eat the business is the actual job
• Read real customer traces every week. The best ideas here come from staring at production, not papers