Technical Project Manager

LILT is an AI company focused on making information accessible across languages through machine translation, human-in-the-loop expertise, and language technology. The Technical Project Manager will lead multilingual data collection, human-in-the-loop workflows, and LLM evaluation initiatives, translating technical AI requirements into scalable workflows, evaluation pipelines, and measurable delivery plans.

Responsibilities:

• Manage Applied AI data collection and evaluation projects from technical scoping through validation, delivery, and retrospective review. through implementation, validation, delivery, and retrospective review
• Translate technical AI requirements into structured data specifications, domain-specific annotation requirements, evaluation plans, and acceptance criteria
• Manage timelines and throughput velocity, dependencies, delivery risks, and technical blockers across Applied AI, engineering, and operations stakeholders
• Coordinate data collection and annotation for supervised fine-tuning (SFT), preference data used in reinforcement learning from human feedback (RLHF), and model evaluation
• Partner with engineering teams to define and validate data workflows, including ingestion, annotation tooling, systems integrations, quality checks, data generation pipelines, and delivery
• Operationalize hybrid evaluation methodologies, combining human review with automated evaluation tooling, and LLM-as-a-judge frameworks
• Coordinate requirements for data formats, schemas, metadata, annotation tools, and system integrations
• Coordinate requirements for data formats, schemas, metadata, annotation tooling, and automated quality checks
• Work with Applied AI specialists to operationalize evaluation methods, including human review, response ranking, and safety testing
• Investigate data, pipeline, or tooling issues and coordinate prompt resolution with technical owners
• Use SQL and business intelligence tools to analyze throughput velocity, quality, unit economics, and supplier/delivery performance
• Define and monitor quality standards, including inter-annotator (IAA) agreement, gold-set performance, and dataset completeness
• Lead root-cause analysis on quality discrepancies, workflow bottlenecks, benchmark discrepancies, and quality edge cases
• Establish delivery validation checks and coordinate stakeholder acceptance of completed, accurate datasets, delivery targets, and evaluation results
• Translate complex technical requirements into actionable plans and clear instructions for global contributor teams, external vendors, and specialized domain SMEs and teams
• Communicate project status, technical risks and tradeoffs, unit economics, and operational risks to technical and non-technical stakeholders
• Ensure contributor feedback and evaluation findings inform improvements to internal tooling, guidelines, and dataset pipelines

Requirements:

• 3–5+ years of technical project management experience in AI/ML, data platforms, or technical data operations
• Strong understanding of LLM training and evaluation concepts, including SFT, RLHF, human evaluation, automated evaluations, LLM-as-a-judge, and red teaming
• Proficiency in SQL for analyzing delivery velocity, data quality metrics, and cost structures
• Working knowledge of data pipelines, structured data formats, APIs, and validation processes
• Experience partnering with engineers to manage technical requirements, dependencies, and issue resolution
• Proven ability to track and optimize project unit economics (cost per token/task) and delivery velocity
• Strong communication skills, including the ability to translate technical requirements for multilingual audiences and diverse global contributor networks
• Experience managing complex workflows using Agile, Scrum, or Kanban
• Experience using Python or scripting tools for data analysis, workflow automation, or evaluation scripting
• Experience managing high-skill Subject Matter Experts (SMEs) or specialized contributor networks across technical domains
• Experience with annotation platforms, automated evaluation tools, business intelligence tools, and Jira
• Experience delivering complex multilingual or multimodal data projects
• Background in computer science, data science, engineering, or equivalent practical experience
• Fluency in an additional language