AI Diagnostics & Observability Engineer

Sage Care is a fast-growing Series A healthcare technology startup focused on transforming healthcare through AI. The Senior AI Diagnostics & Observability Engineer will build systems that enhance the learning and reliability of AI agents by analyzing production conversations and improving agent performance through automated feedback and quality systems.

Responsibilities:

• Design systems that continuously analyze production conversations and identify quality issues
• Develop automated workflows that transform human feedback into actionable improvements
• Build evaluation pipelines that measure agent performance across key dimensions
• Create systems that detect recurring failure patterns and cluster similar issues
• Design mechanisms for routing issues to the appropriate AI, engineering, or operational workflows
• Investigate production failures and identify root causes across transcription, reasoning, retrieval, and orchestration systems
• Build tooling that helps engineers quickly understand why an agent behaved a certain way
• Establish quality metrics and reliability standards for production agents
• Partner with AI engineers to measure the impact of improvements and validate fixes
• Build systems that reduce manual effort required to improve agents
• Create workflows that propose fixes, recommendations, or improvements based on production behavior
• Explore approaches for automatically incorporating feedback into agent development workflows
• Help define the long-term architecture for self-improving AI systems
• Work closely with AI engineers developing production agents
• Collaborate with operations teams who review calls and provide quality feedback
• Help define processes for identifying, prioritizing, and resolving agent issues
• Mentor junior engineers working in the AI quality and evaluation space