24-MAG is offering a specialized full-time consulting opportunity for machine learning engineers and research practitioners. The role focuses on developing advanced evaluation benchmarks for AI systems, requiring hands-on experience with ML models, task design, experimentation, and collaboration with researchers.

Responsibilities:

Turn practical ML research ideas into well-defined, multi-step evaluation tasks
Develop assignments involving model training, experimental modifications, and performance analysis
Define clear technical requirements, expected outputs, and success criteria
Ensure tasks assess genuine implementation and experimental reasoning rather than superficial library usage
Implement reference solutions using Python, scripts, and notebook environments
Configure and run model-training experiments from setup through final evaluation
Modify model components, training procedures, reward functions, or experimental parameters
Validate code, dependencies, datasets, intermediate outputs, and final results
Document complete workflows so experiments can be reproduced independently
Review how frontier AI models approach complex machine learning tasks
Assess implementation quality, experimental methodology, and technical conclusions
Identify coding errors, unsupported assumptions, weak experimental controls, and misleading interpretations
Determine whether reported improvements are supported by the observed results
Explain clearly where and why a model-generated solution fails
Develop selected tasks involving reinforcement learning fundamentals
Evaluate reward-function changes, policy-training behaviour, and experimental outcomes
Assess whether proposed modifications produce the intended training effect
Identify instability, unintended incentives, or incorrect interpretations of RL results
Work closely with researchers, task authors, and fellow machine learning specialists
Compare evaluation decisions to maintain consistent and rigorous benchmark standards
Refine task instructions, reference solutions, and grading criteria based on testing outcomes
Share recurring model failure patterns and opportunities for stronger benchmark coverage

Requirements:

At least 1 year of experience in machine learning research, research engineering, or a comparable technical role
Hands-on experience training and evaluating ML models through complete experimental workflows
Strong understanding of experiment setup, execution, analysis, and reproducibility
Familiarity with large language model capabilities, limitations, and evaluation techniques
Working proficiency in Python and Git
Comfort using both scripting and notebook-based environments
Strong technical writing, analytical reasoning, and attention to detail
Ability to work independently through ambiguous, open-ended research problems
Reliable availability for approximately 35 hours per week
A master's degree or PhD in machine learning, computer science, artificial intelligence, engineering, mathematics, or another relevant STEM discipline is highly relevant
Equivalent practical experience in a research-intensive machine learning role may also be considered
Academic or professional work involving model training, experimentation, or ML systems may strengthen an application
Publications, open-source contributions, technical reports, or substantial research projects may also be valuable
Understanding of reinforcement learning concepts, including reward functions and policy training
Experience in AI training, model evaluation, or benchmark development
Background authoring technical tasks, reference solutions, or grading rubrics
Familiarity with agentic AI systems and multi-step model evaluations
Experience diagnosing model-training failures or unexpected experimental behaviour
Knowledge of experimental design, ablation studies, and performance comparison
Experience reviewing code, notebooks, or research analyses prepared by other practitioners
Familiarity with reproducible ML environments and collaborative Git workflows

Cloud Engineer with AWS

About this role