Senior AI QA Engineer with Python (Manual & Automation)

EPAM Systems is seeking a Senior AI QA Engineer to validate the performance of AI-driven video analysis systems focused on sports content. The role combines manual and automated testing to ensure accuracy in detecting key moments in live sports games and involves collaboration with AWS engineers and stakeholders.

Responsibilities:

• Audit live sports games (NBA, MLB, NFL, NHL) to ensure the AI/Inference Service correctly tracks and labels major sports moments such as touchdowns, home runs, and buzzer-beaters as they happen
• Work with timecodes, video frames, transcriptions, and captions, applying sports-related contexts, lingo, and metrics
• Act as the human expert who catches when the AI hallucinates, misinterprets a sports rule, or produces errors
• Collaborate directly with AWS engineers to detail bugs and validate resolutions
• Review AI-generated labels and metadata tags to ensure they make sense for the sport and meet advertising industry standards (IAB rules), keeping content brand-safe for ad placement
• Compare the inference system's outputs against customer-provided labels to determine accuracy and identify missed or incorrectly detected events
• Execute both manual test cases (for nuanced or edge cases) and automated test cases (for validating outputs at scale and ensuring consistency)
• Log discrepancies, defects, and gaps between expected and actual results, and collaborate with the inference/development team to triage and resolve issues
• Maintain clear QC documentation, track accuracy metrics, and help build a smooth testing process that bridges traditional sports broadcasting with new AI technology
• Serve as a communication bridge between the customer and technical teams, clarifying results and expectations
• Repeat validation iteratively as new labeled assets or model updates are provided to ensure ongoing accuracy and reliability

Requirements:

• 3+ years of experience in both manual and automation QA, preferably in video-focused or AI-driven environments
• Knowledge of testing LLMs and understanding of their workflow
• Proficiency in automation scripting with Python, Selenium, or similar tools for comparing JSON outputs to ground truth at scale
• Understanding of software development and QA cycles, including defect logging, triage, and reporting
• Ability to interpret labeled data and validate model outputs
• Familiarity with concepts like computer vision and transcript analysis (no need to understand internal model workings)
• Strong communication skills for stakeholder interaction and reporting
• Detail-oriented and iterative approach to testing
• Proficiency in English at an Upper-Intermediate level (B2) or higher
• Experience with video testing tools or frameworks
• Prior work in sports analytics or media technology