Principal Engineer – Data Intelligence

Yahoo is a global technology company connecting its brands and partners with hundreds of millions of people through large-scale digital platforms. Yahoo is seeking a Principal Engineer to lead the architecture and development of AI-powered data intelligence platforms, including distributed data systems, retrieval-augmented generation, and agentic AI capabilities. The role will establish technical direction, build reusable platform capabilities, and advance reliable AI-assisted software development across the organization.

Responsibilities:

• Define technical vision and architecture for Yahoo's AI-powered data intelligence platform
• Architect petabyte-scale batch and streaming systems with Beam, Dataflow, Spark, and Kafka
• Build production RAG and multi-agent systems: semantic metadata integration, agent orchestration
• Establish evaluation frameworks — retrieval quality, agent accuracy, hallucination rate, operational safety — with automated benchmarking, so we ship AI we can trust
• Use AI development tools to accelerate design, implement the bar for how the org validates AI-generated code and agent plans
• Drive adoption of RAG and agentic patterns across engineers
• Partner with product, governance, infrastructure, and exit-form priorities

Requirements:

• 10+ years building distributed systems and large-scale data platforms, or equivalent experience
• Degree in CS, AI/ML, Engineering, Math, or related field
• Deep Python expertise and strong software engineering fun
• Production experience with batch and streaming processing (Kafka, or equivalent)
• Shipped and operated RAG applications in production; search, and reranking
• Track record evaluating and validating AI-generated code correctness and operational safety
• Strong communication and technical leadership across cross-functional stakeholders
• Agentic AI frameworks (LangGraph, CrewAI, AutoGen, Semant)
• GCP: Vertex AI, Gemini, BigQuery, Vector Search, Dataflow
• Enterprise metadata, lineage, governance, or observability
• Knowledge graphs and semantic reasoning systems
• Daily use of AI pair-programming tools (Claude, Copilot, g workflows)
• Led large technical initiatives influencing strategy across multiple teams