D
DigitalOceanRemote
Senior Software Engineer I - AI Inference Data Plane
United States of America
Full Time
1 day ago
$139,000 - $174,000 USD
H1B Sponsor LikelyKey skills
AI/ML Domain KnowledgeLarge Language ModelsMultimodal ModelsInference EnginesvLLMSGLangTensorRTDistributed Inference Serving Frameworksllm-dNVIDIA DynamoRay ServeInference Engine InternalsContinuous BatchingPaged AttentionPrefix CachingDistributed InferenceKV-cache LocalityGoLangPythongRPCCloud OperationsOpen Source Software IntegrationGolangAIGenerative AILLMKubernetesDigitalOceanCachingLoad Balancing
About this role
DigitalOcean is expanding its AI Infrastructure layer to support the next generation of AI-driven applications. We are seeking a Senior Engineer 2 to join our AI Inference Data Plane team, responsible for designing, developing, and delivering high-scale, resilient data plane services that power our 'Inference as a Service' offering.
Responsibilities:
- Act as a technical leader on the team, driving the end-to-end design, development, and delivery of critical data plane components hosting large generative AI models
- Architect and refine system design proposals for our high-scale, multi-tenant AI inference cloud ecosystem, ensuring they meet rigorous availability and resiliency standards
- Implement and optimize distributed inference hosting using techniques like tensor/data parallelism, KV cache optimizations, and smart routing
- Work cross-functionally with Product Managers, customer-facing teams, and other engineering teams to align technical roadmaps with customer needs
- Build on Kubernetes-native distributed inference frameworks like llm-d (or alternatives such as NVIDIA Dynamo, Ray Serve) to deliver prefill/decode disaggregation, KV-cache-aware routing, tiered prefix caching, and wide expert parallelism for MoE models
- Solve the distributed-systems problems unique to LLM serving — inference-aware load balancing on queue depth, cache locality, and predicted latency; flow control and fairness across tenants; autoscaling inference pools; and moving gigabytes of KV-cache between prefill and decode instances with negligible overhead
- Contribute upstream to llm-d, vLLM, and the inference gateway ecosystem, and represent DigitalOcean in these communities
- Coach and mentor junior engineers, fostering a culture of technical excellence and continuous improvement
- Maintain and operate critical, high-scale services, utilizing observability tools and defining SLOs to ensure superior platform health
Requirements:
- Hands-on experience hosting large language or multimodal models using inference engines like vLLM, SGLang, or TensorRT
- Familiarity with distributed inference serving frameworks such as llm-d, NVIDIA Dynamo, or Ray Serve
- Hands-on experience with vLLM or alternatives (SGLang, TensorRT-LLM, TGI, Modular MAX), including internals like continuous batching, paged attention, and prefix caching
- Understanding of why cluster-scale serving is hard: KV-cache locality is partitioned across workers, naive round-robin routing destroys cache hit rates and tail latency, and disaggregated prefill/decode requires fast cross-pod KV transfer (e.g., NIXL)
- Knowledge of common LLM architectures and optimization techniques (e.g., continuous batching, quantization)
- Expert-level proficiency in GoLang or Python and familiarity with gRPC
- Proven experience shipping customer-facing software products and running critical services in a high-scale environment similar to DigitalOcean
- Experience integrating and building with open-source software
- Merged contributions to vLLM, llm-d, SGLang, or similar projects strongly preferred