Software Engineer - ML Infrastructure

Watney Robotics Inc is focused on expanding human ambition in the physical world by building autonomous robotic systems. The role of ML Infrastructure Software Engineer involves architecting high-performance computing foundations for perception and intelligence models, managing large-scale multimodal training infrastructure, and ensuring efficient resource allocation.

Responsibilities:

• Own Training & Inference Infrastructure: Design and maintain multi-tenant scheduling systems that automatically place training and inference jobs based on hardware topology, cost, and priority, while enforcing fair resource sharing and preemption policies
• Scale Distributed Training: Partner with researchers to scale JAX and PyTorch-based training loops across heterogeneous GPU/TPU clusters with minimal friction, ensuring rock-solid checkpointing and metrics collection
• Optimize Performance & Hardware Bounds: Profile and improve memory usage, device utilization, throughput, and distributed synchronization, specifically navigating edge hardware bottlenecks like on-chip video decoders and memory bandwidth
• Enable Rapid Iteration: Build clean abstractions for launching, monitoring, debugging, and reproducing experiments so researchers can submit massive jobs without needing to manage underlying cluster state
• Contribute to Core Training Code: Evolve our core JAX model code and training pipelines to natively support new architectures, multimodal video/telemetry data streams, and robust evaluation metrics
• Manage Compute Resources: Ensure highly efficient allocation and utilization of massive cloud-based compute clusters while aggressively monitoring and controlling resource costs