Software Development Engineer 5 – Agentic ML Infrastructure

Sunrise Systems, Inc. is building agentic software to enhance the development, deployment, and operation of machine learning workloads on GPUs. The Software Development Engineer 5 will focus on designing and delivering production-grade intelligent agents across the software stack, leading implementation of Python systems that automate high-value engineering work.

Responsibilities:

• Architect and implement Python agent frameworks, including:
• Tool adapters
• Orchestration loops
• Structured outputs
• Session handling
• CLI or service interfaces suitable for production use
• Build LLM-powered workflows using LiteLLM, DSPy, or equivalent
• Implement bounded prompts, allowlisted tools, budget caps, structured JSON outputs, and evidence-backed reporting
• Integrate agents with the AMD GPU and ML software ecosystem, including:
• ROCm tooling
• PyTorch
• Distributed training and inference stacks
• Profilers
• Cluster metrics
• Log pipelines
• Debug utilities
• Solve ML operations problems on training and inference clusters, including:
• Multi-node failure analysis
• Performance regression investigation
• Configuration and launch issues
• Operational automation
• Own agent quality and evaluation infrastructure
• Develop fixture-driven tests and regression suites
• Implement JSON Schema validation and accuracy metrics
• Integrate testing and evaluation into CI before rollout
• Deliver deployment-ready artifacts, including:
• Docker packaging
• Kubernetes packaging
• Runbooks
• Technical documentation
• Apply security-conscious defaults for customer-controlled environments
• Partner with framework, performance, and field teams
• Incorporate code review feedback and iterate based on production and pilot learnings
• Own significant technical components independently and deliver production-quality solutions with minimal supervision

Requirements:

• Bachelor's degree or higher in: Computer Science, Computer Engineering, Electrical Engineering, Related field. Equivalent practical experience accepted
• 10+ years of professional software engineering experience
• 5+ years of experience developing production Python systems at scale
• Deep experience with distributed ML training or inference on GPU clusters
• Experience with: Multi-node jobs, Collective communication failures, Log and metric correlation across ranks and nodes
• Strong hands-on PyTorch experience
• Production familiarity with large-scale ML stacks such as: Megatron-LM, DeepSpeed, TorchTitan, vLLM, Equivalent technologies
• Proven ML debugging and performance engineering experience across multiple failure modes, including: Throughput regression, Memory errors, Numerical instability, Misconfiguration, Profiler or trace analysis
• Demonstrated experience delivering LLM agent or orchestration systems in production or near-production environments
• Experience with: Tool routing, Structured outputs, Reliability, Testability
• Track record of delivering complex software on schedule
• Strong capabilities in: Clean, maintainable code, Automated testing, Code review, Technical communication
• Ability to work independently, prioritize ambiguous requirements, and align weekly with a technical lead
• Open-source ML development, including: Upstream framework repositories, Community CI patterns, OSS tooling integration
• ML performance optimization, including: Parallelism, Throughput/MFU tuning, Roofline analysis, Profiler-driven investigation
• ROCm and GPU cluster operations, including: Metrics, Profilers, Health monitoring, Slurm, Kubernetes job environments
• Security-aware agent deployment, including: Secret handling, Egress control, Customer VPC constraints, On-premises deployment constraints