Required:
5+ years of software engineering experience, including 2+ years on ML/LLM systems in production
Strong Python, plus working knowledge of Go, C++, or Rust for performance-critical paths
Hands-on experience with LLM serving frameworks: vLLM, TensorRT-LLM, TGI, SGLang, or Triton Inference Server
Inference optimization techniques:
Quantization (INT8/INT4, FP8, AWQ, GPTQ)
KV-cache management, paged attention, prefix caching
Continuodynamic batching and speculative decoding
Tensor and pipeline parallelism
Experience with PyTorch and the Hugging Face ecosystem
Kubernetes, Docker, and GPU scheduling (NVIDIA GPU operator, node pools, autoscaling)
Integration with hosted LLM APIs (OpenAI, Gemini, Anthropic), including streaming, retries, and rate limiting
Observability tooling: PrometheGrafana, Splunk, Datadog, OpenTelemetry
Solid grounding in distributed systems, API design (REST/gRPC), and performance profiling
Preferred:
GenAI orchestration frameworks: Genkit, LangChain, LlamaIndex
CUDA, Triton kernels, FlashAttention, or other low-level GPU optimization
Model compilation and runtimes: ONNX Runtime, TorchScript/torch.compile, Core ML
RAG pipelines and vector databases (FAISS, Milvus, pgvector)
LLM evaluation frameworks and building automated quality and regression benchmarks
Cost modeling and FinOps for GPU workloads
Infrastructure as code (Terraform, Helm) and CI/CD (GitHub Actions, Jenkins)
Contributions to open-source inference projects, or published work in ML systems
Soft skills:
Can turn ambiguous requirements into concrete technical plans and trade-off proposals
Communicates clearly in writing (design docs, benchmark reports)
Works well with cross-functional teams of engineers and PMs.
