Key skills

LLM Inference Engines vLLMLLM Inference Engines SGLangLLM Inference Engines TensorRT-LLMLLM Inference Engines TGILLM Inference Engines NVIDIA NIMLLM Inference InternalsTensor and Pipeline ParallelismNCCLGPU Memory and Interconnect ArchitectureKubernetesGoPythonC++/CUDANsightRAG and Agent WorkloadsRDMAGPUDirect StorageInfiniBandRoCEMIG and Fractional GPU AllocationC++CAILLMRAGCachingMentoring

About this role

Hewlett Packard Enterprise is a global edge-to-cloud company advancing how people live and work through connected, protected, and analyzed data and applications. The Senior Software Engineer, Inference will build and evolve the LLM model runtime for HPE AI Essentials, focusing on inference efficiency, distributed execution, GPU utilization, and Kubernetes orchestration. The role also includes evaluating emerging inference technologies, resolving customer issues, conducting design reviews, and mentoring team members.

Responsibilities:

Design, implement, and own major components of the LLM serving deployment, including engine integration, continuous batching, KV cache management and reuse, and quantized execution
Partner with inference engineering teams and contribute to improving time-to-first-token, inter-token latency, throughput per GPU, and P95/P99 tail latency
Build and operate distributed execution capabilities, including disaggregated prefill/decode, tensor and pipeline parallelism, and KV cache offload across GPU memory, host memory, and RDMA-attached storage
Evaluate emerging runtimes, quantization schemes, speculative decoding, and mixture-of-experts serving, and make well-supported recommendations on adoption
Contribute to the orchestration layer supporting the runtime, including model admission, GPU scheduling and partitioning, cache-aware request routing, and autoscaling
Triage and resolve customer issues end-to-end, identifying root causes and improving systems and processes to prevent recurrence
Provide insightful code and design reviews, mentor team members, and lead by example on engineering practices within the team

Requirements:

Familiar with LLM inference engines such as vLLM, SGLang, TensorRT-LLM, TGI, or NVIDIA NIM, including modification of engine internals
Strong understanding of inference internals, including continuous batching, paged attention, KV cache reuse and prefix caching, chunked prefill, quantization, and speculative decoding
Working knowledge of tensor and pipeline parallelism, NCCL collective operations, and the GPU memory hierarchy and interconnect characteristics that govern them
Advanced proficiency in Kubernetes platform architectures, including operators, custom resources, controllers, and scheduling
Strong programming proficiency in Go and Python, with the ability to read, debug, and profile C++/CUDA using tools such as Nsight
Familiar with debugging/profiling multi-tier application workloads such as RAG, Agents
Excellent analytical, debugging, and problem-solving abilities
Minimum of 8 years of experience in Software Engineering, including 1-2+ years working directly on LLM inference runtimes or production model serving
Degree in Computer Science or related field
Upstream contribution to vLLM, SGLang, TensorRT-LLM, llm-d, LMCache, or KServe
Disaggregated prefill/decode serving, or KV cache offload and reuse at scale
RDMA, GPUDirect Storage, InfiniBand, or RoCE
MIG, fractional GPU allocation, and multi-tenant GPU isolation
On-premises, air-gapped, or regulated enterprise software delivery