Design and execute test plans for AI-powered applications across both output quality and functional behavior
Evaluate AI responses for accuracy, consistency, hallucination, and edge case handling
Test application workflows, UI behavior, and integrations across internal and client-facing tools
Identify, document, and track defects using standard issue tracking tools (e.g., Jira or equivalent)
Partner with AI Builders and Product Managers to define acceptance criteria and quality standards for AI features
Develop and maintain repeatable test cases and QA documentation as AI applications evolve
Conduct regression testing as applications are updated or new features are released
Collaborate with the Principal Engineer and product team to surface quality risks early in the build process
Contribute to defining what “good” looks like for AI output quality at OneDigital, helping shape internal QA standards for LLM-based applications
Support testing across multiple business lines including Employee Benefits, Retirement, Wealth, and P&C

Experience in a QA, testing, or quality-focused role, with some exposure to AI or LLM based products preferred
Strong ability to evaluate AI-generated outputs for quality, accuracy, and appropriateness
Familiarity with prompt-driven systems and an intuition for how language models can fail or produce unexpected results
Experience writing and executing test cases and documenting defects clearly
Comfort working in no-code and low-code environments
Strong written communication skills, with the ability to describe issues precisely and concisely
Curiosity and adaptability in a fast-moving, emerging technology space
Ability to work independently and manage multiple testing workstreams at once

AI Quality Assurance Analyst

Key skills