Expert: Interactive Coding, Site Reliability Engineering (Contract, Remote)

Patronus AI hires an expert contractor to work with its research team on production-style site reliability and infrastructure scenarios. The role involves investigating and resolving sandboxed incidents, explaining operational reasoning, reviewing scenario realism, and creating evaluation and training data for AI models.

Responsibilities:

• Work through incident, debugging and infrastructure-change scenarios live with a researcher: reading dashboards and logs, writing PromQL and LogQL, fixing Helm and Kubernetes configuration, rolling back bad deploys and writing post-mortems
• Explain your reasoning as you work, including the fixes you considered and rejected
• Review scenarios we have drafted and say plainly what is unrealistic, too easy or underspecified
• Help write the acceptance criteria for each scenario, including the tempting wrong moves an engineer under pressure might make
• Write short notes after each session on how you approached the problem and what you verified
• Work in reviewer feedback within agreed turnaround times, and flag guideline gaps or edge cases you run into

Requirements:

• 3+ years running production systems in at least one of the areas above, including time on an on-call rotation within the last two years. You did the work yourself; managing the team is not enough
• Hands-on with Kubernetes, Prometheus and Grafana (or Datadog or New Relic), a log stack such as Loki, ELK or Splunk, infrastructure as code such as Terraform and Helm, and a CI/CD system
• Comfortable in Python, Go or Bash and at reading unfamiliar codebases
• Able to explain your reasoning out loud while you work, in clear professional English
• Reliable availability for scheduled live sessions and the habit of showing the evidence behind every conclusion
• Experience writing runbooks, post-mortems, SLO definitions or incident training material
• Familiarity with how AI coding agents fail on operations work