Software Development Test Engineer (SDET), AI & Automation

Keeper Security is a cybersecurity software company that protects organizations and individuals from cyber attacks. The Senior SDET, AI & Automation will partner with the AI team to define test strategies, build automated frameworks, validate deterministic and non-deterministic AI behavior, and integrate quality gates into CI/CD workflows.

Responsibilities:

• Partner directly with AI engineers, Software Engineers and Product Managers during design and development to define test strategy, acceptance criteria and automation requirements
• Design and maintain automated testing frameworks for AI-powered applications, backend services, APIs and supporting software systems
• Develop testing strategies for both deterministic software behavior and non-deterministic AI/ML outputs
• Define measurable evaluation criteria, tolerances and quality thresholds for AI model and LLM-driven functionality
• Build automated regression, integration, API and end-to-end coverage alongside feature development
• Develop test harnesses and evaluation workflows for prompts, models, agents, tool-calling workflows and AI-integrated applications
• Validate AI behavior across different models, prompts, inputs and system configurations
• Develop automated test utilities using languages such as Python, Java, JavaScript/TypeScript, Rust or similar technologies
• Integrate automated validation and quality gates into CI/CD pipelines using GitHub Actions, Jenkins, GitLab CI or similar platforms
• Build and maintain test datasets, mocks, fixtures and synthetic data required for repeatable AI and software validation
• Analyze model and application failures to distinguish defects, expected model variability, data issues and test instability
• Evaluate AI outputs for reliability, consistency, safety and alignment with expected product behavior
• Work with engineering teams to improve application testability, observability and diagnostic capabilities
• Track test cases, execution results and coverage using TestRail or similar test management platforms
• Analyze logs, telemetry and system behavior to identify root causes and produce clear, reproducible defect reports
• Reduce flaky tests and improve the speed, stability and maintainability of automated test suites
• Contribute to quality engineering standards, AI testing methodologies and automation best practices across Keeper’s AI organization
• Use AI-assisted tools such as Claude, ChatGPT, GitHub Copilot or similar platforms to improve automation development, test generation, debugging, analysis and documentation

Requirements:

• 5+ years of experience in SDET, software development, test automation or another code-intensive quality engineering role
• Hands-on experience testing AI/ML, LLM-powered or model-integrated software systems
• Strong understanding of the differences between deterministic software testing and non-deterministic AI/ML validation
• Experience defining evaluation criteria, expected behaviors, tolerances or scoring approaches for AI-driven functionality
• Strong programming skills in Python, Java, JavaScript/TypeScript, Rust or another general-purpose programming language
• Significant experience designing, developing and maintaining automated test frameworks
• Experience building automated functional, integration, API and end-to-end test coverage
• Experience testing REST APIs and distributed application workflows using tools such as Postman, REST Assured or similar platforms
• Experience integrating automated tests into CI/CD pipelines using GitHub Actions, Jenkins, GitLab CI or similar technologies
• Experience working within an Agile software development team with close collaboration between QA, Engineering and Product
• Strong understanding of shift-left testing, continuous quality and testability principles
• Experience with TestRail or similar test management platforms
• Strong debugging and troubleshooting skills using logs, APIs, application telemetry and development tools
• Experience with Git-based development workflows, code review and modern software engineering practices
• Familiarity with AI-assisted development and testing tools such as Claude, ChatGPT, GitHub Copilot or similar technologies
• Strong analytical and problem-solving skills with the ability to evaluate variable AI behavior without relying solely on exact-match outcomes
• Strong written and verbal communication skills and ability to collaborate closely with AI engineers, Software Engineers, Product Managers and QA professionals
• Bachelor's degree in Computer Science, Engineering or a related technical field, or equivalent practical experience
• qualified candidates must be a U.S. Person, defined as a U.S. citizen or lawful permanent resident
• Experience testing LLMs, agentic systems, retrieval-augmented generation, tool calling or prompt-driven workflows
• Experience building automated model evaluation, benchmarking or regression frameworks
• Familiarity with AI evaluation metrics and techniques for measuring quality across non-deterministic outputs
• Experience with Rust in production or test automation environments
• Experience with performance, reliability, security or adversarial testing of AI-enabled systems
• Experience with AI-assisted generation of automated tests or synthetic test data
• Familiarity with Docker, Kubernetes or cloud-based testing environments
• Experience testing cybersecurity, identity, PAM or other security-sensitive software
• Experience building shared automation frameworks or internal testing tools used across engineering teams