Senior Software Development Engineer

Trilyon, Inc. is a leader in global workforce solutions, specializing in Cloud Technology, AI/ML, Software Development, Technical Writing, and Digital Transformation. They are seeking a Senior Software Development Engineer to leverage extensive experience in software engineering and machine learning to deliver high-quality solutions in production environments.

Responsibilities:

• Deep experience with distributed ML training or inference on GPU clusters: multi-node jobs, collective communication failures, log and metric correlation across ranks and nodes
• Strong hands-on PyTorch background and production familiarity with large-scale stacks (Megatron-LM, DeepSpeed, TorchTitan, vLLM, or equivalent)
• Proven ML debugging and performance engineering across multiple failure modes: throughput regression, memory errors, numerical instability, misconfiguration, and profiler or trace analysis
• Demonstrated delivery of LLM agent or orchestration systems in production or nearproduction settings: tool routing, structured outputs, reliability, and testability
• Track record of shipping complex software on schedule: clean code, automated tests, code review discipline, and clear technical communication
• Ability to work independently, prioritize across ambiguous requirements, and align weekly with a technical lead

Requirements:

• Bachelor's degree or higher in Computer Science, Computer Engineering, Electrical Engineering, or a related field; equivalent practical experience accepted
• 10+ years of professional software engineering experience, including 5+ years on production Python systems at scale
• Deep experience with distributed ML training or inference on GPU clusters: multi-node jobs, collective communication failures, log and metric correlation across ranks and nodes
• Strong hands-on PyTorch background and production familiarity with large-scale stacks (Megatron-LM, DeepSpeed, TorchTitan, vLLM, or equivalent)
• Proven ML debugging and performance engineering across multiple failure modes: throughput regression, memory errors, numerical instability, misconfiguration, and profiler or trace analysis
• Demonstrated delivery of LLM agent or orchestration systems in production or near-production settings: tool routing, structured outputs, reliability, and testability
• Track record of shipping complex software on schedule: clean code, automated tests, code review discipline, and clear technical communication
• Ability to work independently, prioritize across ambiguous requirements, and align weekly with a technical lead
• Open-source ML development: upstream framework repos, community CI patterns, and integration with OSS tooling
• ML performance optimization: parallelism, throughput/MFU tuning, roofline analysis, and profiler-driven investigation
• ROCm and GPU cluster operations: metrics, profilers, health monitoring, Slurm or Kubernetes job environments
• Security-aware agent deployment: secret handling, egress control, and customer VPC or on-prem constraints