Own and extend our offline eval suite across products — datasets (capability + regression), judges, metrics
Build and maintain online quality dashboards: resolution rate, CSAT, thumbs up/down, LLM-as-judge signals, error rate, latency
Close the production feedback loop: mine failure patterns from real traffic → turn them into regression cases → propose fixes to Product and domain experts
Harden methodology: judge stability, non-determinism handling
Translate numbers into decisions – weekly syncs, clear trade-offs, no dashboards for their own sake

Python and SQL – you can build an analysis end-to-end
Solid foundation in statistics – sampling, hypothesis testing, variance, understanding what a noisy metric is
Analytical mindset – you start from the business question, not from the tool
3+ years in analyst / data scientist roles, at least one in a product context
Hands-on experience evaluating LLM applications (RAG, agents, tool use, judges)
Experience building LLM agents – side projects, toy builds, personal experiments all count
AI-assisted coding is our default authoring environment
Curious and fluent with AI coding, or genuinely excited to become fluent fast
Care about what you ship and how clearly you think

Make a genuine impact on the product
Join our upward trajectory and grow with us
Work in the EU
Enjoy the flexibility of traveling and working remotely or in a hybrid model across Europe
Become a stock options holder
Unlock your inner entrepreneur through our Stock Options Program
Receive unwavering support and care
Constant support and care to ensure your experience is successful and fulfilling
Immerse yourself in our exclusive Work & Swim Program
Spend one month in a comfortable corporate apartment in enchanting Cyprus
Ideal opportunity for work-life balance while enjoying Mediterranean views
Equal Opportunity Employer valuing diversity

Product Data Scientist – AI Evaluation, Quality

Key skills