Senior Software Engineer, GPU Inference & Kernel Systems

Runara is building the inference serving engine for a world where AI compute is scarce, fragmented, and unevenly priced. The Senior Software Engineer will focus on optimizing GPU inference and kernel systems, developing innovative solutions to enhance model serving performance across heterogeneous compute environments.

Responsibilities:

• Tune acceptance rates, batching strategy, and draft/verify scheduling for maximum throughput
• Design, build, and optimize speculative decoding pipelines. Extend speculative workloads beyond a single machine, routing prefill and decode across heterogeneous hardware classes based on real-time cost and availability
• Design the scheduling and orchestration logic that spans different xPU hardware, treating a mixed fleet as one fungible compute pool
• Develop a working model of token economics, cost per token, throughput per dollar, and use it to drive design decisions
• Write and optimize CUDA kernels that make inference viable on every GPU SKU — including quantization, memory layout, and kernel strategies tuned to each xPU target
• Optimize for memory bandwidth and compute across inference stacks such as TensorRT-LLM, vLLM, and SGLang, adapting each to the constraints of the underlying hardware
• Work on tensor parallelism and LLM sharding, and on the compiler/runtime layer that takes a model graph and produces hardware-aware execution plans across a wide, messy diversity of GPU architectures and driver/runtime versions
• Integrate NGC compute availability while continuously probing dozens of cloud providers for capacity, so routing decisions are always based on live pricing and supply