AI Business Analyst

Allied Benefit Systems is seeking an AI Business Analyst to evaluate its internal AI solutions and determine whether they deliver intended business outcomes. The role investigates system behavior, translates findings into product requirements, develops evaluation frameworks, and collaborates with product, engineering, QA, and subject matter experts to improve AI capabilities.

Responsibilities:

• Evaluate AI capabilities integrated into core systems (e.g., Dynamics CRM, claims adjudication platform, document repositories) to determine whether they are functioning as intended and delivering expected business outcomes
• Investigate edge cases, anomalies, and failure patterns by reviewing raw data, logs, and system outputs
• Collaborate with technical product owners and SMEs to document what is and isn’t working, and help translate findings into user stories and product requirements
• Participate in model and capability reviews, providing business-level validation of AI behavior and identifying gaps in functionality or alignment with intended use
• Contribute to the development of test scenarios and evaluation frameworks that reflect real-world usage and business expectations
• Recommend improvements or refinements to AI capabilities based on observed behavior, user feedback, and business context
• Support the AI Product Owner and QA team in defining acceptance criteria and success metrics for new AI features
• Help ensure that AI solutions are understandable, trustworthy, and aligned with Allied’s operational goals
• Other duties as assigned

Requirements:

• Bachelor's degree in related field or equivalent work experience required
• 8–12 years of experience in business analysis, quality assurance, or systems evaluation roles
• Deep subject matter expertise in claims, health call centers, provider operations, or health plan administration
• Strong analytical mindset with the ability to identify subtle patterns, inconsistencies, or edge cases in complex systems
• Comfortable working with structured and unstructured data, including reviewing logs, outputs, and raw files to trace issues or validate behavior
• Experience writing clear, structured documentation such as user stories, test cases, and issue reports
• Excellent communication skills and the ability to collaborate across technical and non-technical teams
• Familiarity with tools for data inspection and analysis (e.g., Excel, SQL, Power BI)
• Experience with log analysis and debugging tools (e.g., Azure Monitor, Splunk)
• Agile collaboration tools (e.g., Freshservice, Azure DevOps, or similar) for tracking issues and documenting requirements
• ideally within the insurance or administrative healthcare industry
• Familiarity with AI/ML systems and their evaluation is a plus, but not required
• Exposure to API testing tools (e.g., Postman) and basic scripting or query languages is helpful