Staff/Principal DevOps Engineer, AI Inference

Lila Sciences is building Scientific Superintelligence™ to solve humankind's greatest challenges. The Staff/Principal DevOps Engineer - AI Inference will drive the design and optimization of infrastructure for serving machine learning models, collaborating with engineers and scientists to ensure efficient model serving and high performance.

Responsibilities:

• Drive the design, implementation, and optimization of infrastructure purpose-built for serving machine learning models at scale
• Collaborate with ML engineers, research scientists, and software engineers to build inference platforms that serve models reliably to production users while maximizing compute efficiency
• Build GPU/accelerator infrastructure on Kubernetes: scheduling, resource isolation, multi-tenant GPU sharing, device plugins, and topology-aware placement for inference workloads
• Create model serving platforms using frameworks such as vLLM, Triton Inference Server, TGI, or custom serving stacks with optimized batching, caching, and request routing
• Implement intelligent request routing and load balancing across heterogeneous accelerator fleets (NVIDIA GPUs, AWS Inferentia/Trainium) to maximize utilization and minimize latency
• Develop autoscaling systems that dynamically match inference compute supply with demand across production, research, and experimental workloads
• Establish production-grade deployment pipelines for ML models: canary rollouts, A/B testing, model versioning, and safe rollback across multi-region deployments
• Utilize infrastructure-as-code with Terraform and Helm for GPU-accelerated EKS clusters, including node pools, spot/on-demand strategies, and accelerator-specific networking
• Monitor observability and performance optimization: GPU utilization monitoring, inference latency profiling, token throughput dashboards, and SLO/SLI tracking for model endpoints
• Create CI/CD pipelines for model artifacts: container image builds with CUDA/driver dependencies, model registry integration, and automated inference benchmarking in CI
• Manage AWS cloud infrastructure for ML: EKS with GPU node groups, EC2 accelerated instances (P4/P5, Inf2, Trn1), S3 model storage, EFA/high-bandwidth networking, and IAM least privilege
• Conduct cost optimization and capacity planning: right-sizing accelerator instances, spot instance strategies for inference, and fleet-wide efficiency reporting

Requirements:

• Expertise in DevOps, SRE, or Platform Engineering with significant experience operating GPU/accelerator infrastructure at scale
• Deep experience with Kubernetes for ML workloads: GPU scheduling, resource quotas, node affinity, and accelerator device management
• Strong proficiency deploying to AWS using infrastructure-as-code (Terraform, Helm) with hands-on experience managing GPU-based compute (EKS, EC2 P-series/Inf/Trn instances)
• Experience with model serving infrastructure: inference servers, request batching, KV-cache optimization, or LLM serving frameworks
• Strong understanding of networking for distributed inference: high-bandwidth interconnects, NCCL, VPC/PrivateLink, and load balancing at L4/L7
• Strong proficiency in Python for automation, tooling, and integration with ML frameworks
• Experience with LLM inference optimization: continuous batching, speculative decoding, quantization (GPTQ, AWQ, FP8), tensor parallelism, and pipeline parallelism
• Hands-on experience with multiple accelerator families (NVIDIA A100/H100, AWS Inferentia2, Trainium, AMD MI300X) and maintaining hardware-agnostic serving infrastructure
• Multi-region deployment experience with geographic routing and failover for latency-sensitive inference endpoints
• Proficiency in Rust or Go for performance-critical infrastructure components
• SRE practices for ML systems: chaos engineering on GPU workloads, incident management, capacity modeling for bursty inference traffic
• Experience with model registries, artifact versioning, and ML supply chain security
• Observability platform expertise: building custom metrics for token-level throughput, time-to-first-token, and per-request GPU memory profiling
• Prior startup/high-growth experience balancing velocity with reliability in rapidly scaling AI systems