Software Engineer, Production Inference (Distributed Inference)

Thinking Machines Lab builds AI systems designed to extend human will and judgment, including frontier models and human-AI interfaces. The company is seeking a Software Engineer to build and scale distributed production inference systems, optimizing model serving for latency, throughput, reliability, and cost while collaborating with research and infrastructure teams.

Responsibilities:

• Design, build, and operate distributed infrastructure for large-scale model serving, including request routing, load balancing, batching, and multi-node coordination
• Optimize inference latency and throughput in production, including work on KV cache management, continuous batching, speculative decoding, and quantization
• Build and maintain high-concurrency serving systems with strong uptime, low tail latency, and deep observability
• Benchmark, tune, and extend inference engines to support new model architectures as they move from research into production
• Partner with research and infrastructure teams to translate emerging model designs into production-ready serving systems
• Build tooling for tracing, debugging, and resolving issues across the serving stack, from orchestration down to GPU kernels
• Participate in on-call rotation to support production inference systems