Software Engineer - Model Performance

Baseten powers mission-critical inference for dynamic AI companies by combining applied AI research, flexible infrastructure, and developer tooling. The company is seeking a Software Engineer focused on ML performance to optimize and productionize techniques for large language model inference, investigate performance issues in ML libraries and infrastructure, and own projects from idea to production.

Responsibilities:

• Implement, refine, and productionize cutting-edge techniques (quantization, speculative decoding, KV cache reuse, chunked prefill and LoRA) for ML model inference and infrastructure
• Deep dive into underlying codebases of TensorRT, PyTorch, TensorRT-LLM, vLLM, sglang, CUDA, and other libraries to debug ML performance issues
• Apply and scale optimization techniques across a wide range of ML models, particularly large language models
• Collaborate with a diverse team to design and implement innovative solutions
• Own projects from idea to production