Machine Learning Engineer — Pre-training (LLM)

MBZUAI’s Institute of Foundation Models is a research lab focused on building, understanding, using, and risk-managing foundation models. The Machine Learning Engineer will extend and scale distributed ML training systems for large-scale LLM pre-training, implementing distributed optimizers, building multi-node and multi-GPU launch and monitoring systems, and improving infrastructure reliability and performance.

Responsibilities:

• Extend distributed training frameworks (e.g., DeepSpeed, FSDP, FairScale, Horovod)
• Implement distributed optimizers from mathematical specs
• Build robust config + launch systems across multi-node, multi-GPU clusters
• Own experiment tracking, metrics logging, and job monitoring for external visibility
• Improve training system reliability, maintainability, and performance
• Distributed Framework Ownership – Extend or modify training frameworks (e.g., DeepSpeed, FSDP) to support new use cases and architectures
• Optimizer Implementation – Translate mathematical optimizer specs into distributed implementations
• Launch Config & Debugging – Create and debug multi-node launch scripts with flexible batch sizes, parallelism strategies, and hardware targets. Select and validate data, tensor, pipeline, expert, and context parallelism strategies as appropriate for the model and cluster
• Metrics & Monitoring – Build systems for experiment tracking, job monitoring, and logging usable by collaborators and researchers
• Infra Engineering – Write production-quality code and tests for ML infra in PyTorch or JAX; ensure reliability and maintainability at scale
• Data Loading & Checkpointing – Build and maintain data-loading and checkpoint/restart workflows, restoring model, optimizer, RNG, and data progress after interruptions