Principal Quality Engineer

Peach Pilot is an innovative company transforming how businesses operate through AI technology. They are seeking a Principal Quality Engineer to build and own the QA function, ensuring the quality of AI-generated insights and recommendations before they reach clients.

Responsibilities:

• Build the QA Foundation (First 90 Days)
• Establish the testing framework from zero: unit, integration, end-to-end, and AI-specific evaluation pipelines using Playwright and Vitest
• Define quality standards, test coverage requirements, and documentation practices in partnership with the Lead Engineer
• Audit the existing platform and identify the highest-risk surfaces before the next client deployment
• Define the team structure you will need — onshore vs. offshore mix, roles, and a hiring roadmap — and begin executing against it
• AI Agent & Knowledge Graph Testing
• Design evaluation frameworks for non-deterministic LLM outputs — including prompt regression testing, model drift detection, and output quality scoring
• Build automated test suites for the agent orchestration layer, including governance agent audit trail integrity and human-override behavior
• Validate the Company Brain (Memgraph + Qdrant) for data accuracy, retrieval quality, and failure modes under real enterprise data conditions — including entity resolution across systems and temporal data patterns
• Test the Analysis Engine pipeline that surfaces Company X-Ray findings — ensuring that insights are not just technically accurate but reliable enough to present to a client
• Platform & Integration Testing
• Own end-to-end testing of the data ingestion pipelines that connect to client systems — CRM, email, calls, calendars, documents, financial systems — through Nango's 700+ connector integration layer
• Test multi-model routing logic to confirm cost-optimized task allocation behaves correctly across LLM providers via LiteLLM
• Validate streaming response handling, latency thresholds, and graceful degradation when a model is unavailable or slow
• Own file ingestion pipeline testing (Word, Excel, PowerPoint, PDF) including encryption, formatting edge cases, and audit trail continuity
• Build and Lead the QA Team
• Recruit, hire, and onboard QA engineers as the team grows — setting clear expectations, working standards, and a bar for technical excellence from day one
• Mentor junior and mid-level QA engineers, building their ability to own test domains independently
• Act as the quality culture carrier across the full engineering team — QA is not a department, it is everyone's responsibility
• Report directly to the Lead Engineer and participate in product planning to ensure quality is designed in, not bolted on

Requirements:

• 7+ years of QA engineering experience, with at least 3 years in a senior or lead capacity where you shaped process and standards — not just executed them
• You have tested AI/LLM-powered applications. You understand prompt sensitivity, output variance, and how to build eval pipelines that catch regressions across model updates
• You write test code. Python is your primary tool. You have built and maintained CI/CD-integrated test suites and you don't wait for someone to file a bug to find one
• Hands-on experience with Playwright and Vitest in a production environment — and you've built automation frameworks from scratch, not just inherited them
• Comfortable testing complex API chains, async/streaming responses, and multi-service workflows. Data pipelines and knowledge graph outputs don't intimidate you
• You have built a QA function from the ground up in an early-stage environment. You know when to move fast and when to go deep
• You test for confusion and trust failure — not just broken functionality. Your end users are non-technical executives, and you advocate for them
• You have experience with LLM evaluation frameworks (e.g., LangSmith, PromptFlow, or custom eval pipelines)
• You have tested agent frameworks or orchestration layers in a production environment
• You have a background in a regulated industry (insurance, finance, healthcare) where audit trail integrity is non-negotiable
• You have worked alongside Forward Deployed or solutions engineering teams and understand field deployment risk