Software Engineer L5/L6 — Model Evaluations & Data Curation (MEDC)

Netflix is an entertainment company developing technology to support global storytelling and member experiences. The Software Engineer will build reusable infrastructure for data curation, including scalable LLM-driven transformation pipelines, sampling strategies, filtering, quality control, and dataset management. The role partners with researchers to measure how curation decisions affect model performance and drive adoption of shared practices.

Responsibilities:

• Design and build shared data curation infrastructure (reusable components, libraries, and workflows) that replaces ad-hoc notebooks and makes new curation projects fast to launch and easy to productionize
• Build scalable LLM-driven data transformation pipelines that turn raw sources such as the Netflix catalog and metadata into training and evaluation data (e.g., question-answer pairs, synthetic scenarios), using large-scale batch inference with attention to quality and token cost
• Develop sampling strategies (coverage, diversity, difficulty, and balance across content and member segments) for constructing training and evaluation sets
• Develop filtering and quality-control methods, including LLM-as-judge and evaluation-model-based scoring, deduplication, and validation, to maximize data quality
• Partner with researchers to measure how curation choices affect model performance, closing the loop between data quality signals and model outcomes
• Make curated datasets first-class, discoverable artifacts with versioning, explicit lineage, and reproducibility, so teams can find, reuse, and build on each other's work
• Drive adoption of shared curation practices across MEDC and partner modeling teams

Requirements:

• Strong software engineering in Python, with experience building reusable infrastructure, libraries, or frameworks used by other engineers and researchers
• Experience building LLM-driven data generation or transformation pipelines (e.g., synthetic data, structured outputs, batch inference at scale)
• Hands-on experience with data quality methods: sampling strategies, filtering, deduplication, and model-based quality scoring such as LLM-as-judge
• Modeling intuition: an understanding of how data choices affect model behavior, and the ability to design experiments that measure it
• Experience with distributed data processing (e.g., Spark, Ray, or similar)
• Excellent collaboration skills, particularly with researchers, data scientists, and platform teams
• Experience with LLM evaluation systems (must-have for L6)
• Technical leadership across data and evaluation infrastructure; experience setting technical direction for a multi-engineer effort (must-have for L6)
• Experience with dataset versioning, lineage, and artifact management (e.g., versioned datasets, experiment tracking, model registries)
• Experience optimizing cost and throughput for large-scale LLM inference
• Experience with human annotation workflows and calibrating LLM judges against human raters
• Experience with pipeline orchestration frameworks (e.g., Metaflow, Airflow, or similar)
• Background in recommendation systems, personalization, search, or working with content catalog and metadata