Key skills
AWSAzureCloudDockerGoogle Cloud PlatformJenkinsPythonSQLAILLMAgenticData EngineeringPytestGCPGoogle CloudGitHub ActionsGitGitHubCommunication
About this role
Role Overview
- Design and build evaluation harnesses for agentic systems in Python
- Author automated test suites for prompts and workflows
- Validate guardrails around tool execution
- Wire evaluations into CI using Dataiku Evaluations, GitHub Actions or Jenkins
- Build observability into testing
- Own quality end-to-end
- Partner with data engineers on retrieval testing patterns
- Help shape internal QA standards for AI & Data engineering
- Participate in a collaborative DevOps environment
Requirements
- 6+ years of experience in QA / SDET
- 3+ years of professional experience automating tests for backend services or data pipelines
- 1+ years of hands-on experience testing LLM or AI features in production
- Working knowledge of evaluation frameworks such as RAGAS, DeepEval, LangSmith or comparable LLM-as-judge tooling
- Strong Python and PyTest skills
- Solid SQL skills
- Familiarity with at least one cloud platform (AWS, Azure or GCP)
- Fluency with Git, Docker, REST APIs and at least one CI tool
- Solid understanding of data security and responsible AI practices
- Proven ability to work independently and within a team
- Strong written and verbal communication skills
- A bachelor’s degree is not required — equivalent practical experience counts
Tech Stack
- AWS
- Azure
- Cloud
- Docker
- Google Cloud Platform
- Jenkins
- Python
- SQL
Benefits
- Comprehensive health coverage
- Well-being perks supporting teammates and their dependents
- Flexible time off
- Recognition of public holidays
- Opportunities for charitable partnerships and volunteer work