Senior Site Reliability Engineer

Peraton Labs is hiring a Senior Site Reliability Engineer to own end-to-end reliability of its enterprise application. The role defines reliability objectives, develops observable and resilient Python and Rust services, operates cloud infrastructure and monitoring, and leads incident response and reliability improvements.

Responsibilities:

• Own end-to-end reliability of the enterprise application; define SLIs/SLOs with product and engineering, enforce error-budget policy, and use burn rates to guide release and prioritization decisions
• Write, review, and ship production Python and Rust code to fix performance, concurrency, and failure-handling issues and make services observable and resilient
• Instrument services with metrics, structured logging, and distributed tracing (OpenTelemetry or equivalent)
• Own monitoring and alerting (CloudWatch, Prometheus/Grafana): build low-noise, SLO-based burn-rate alerts, manage alert routing and on-call hygiene, and design service, SLO, and executive dashboards
• Build in-application health checks and synthetic user-journey tests that exercise real auth flows and critical transactions
• Lead incident response, run blameless postmortems, and drive corrective actions to closure
• Operate Amazon EKS/Kubernetes (Helm, RBAC, pod identity, GitOps, upgrades) and core AWS services, managed through OpenTofu/Terraform
• Maintain GitHub Actions/GitLab CI pipelines with reliability and performance gates, supporting trunk-based, MR-driven Agile delivery and progressive (canary/blue-green) rollouts
• Enforce secrets management and least privilege, and produce audit evidence for NIST 800-171, CMMC, FedRAMP, or DoD/ATO environments
• Drive load testing, capacity planning, and chaos engineering, and mentor engineers on reliability practices

Requirements:

• Bachelor's Degree and 8+ years in SRE, platform, or software engineering roles with production reliability ownership
• Strong software engineering in Python
• Comfortable reading, debugging, and shipping changes to production application code, including performance, concurrency, and failure-handling fixes
• Working effectively in unfamiliar codebases, including with AI-assisted development tools, counts
• Judgment and code review rigor matter more than language tenure
• SLOs and error budgets. Defining SLIs with product and engineering, setting SLO targets, running error-budget policy, and using burn rates to drive release and prioritization decisions
• Monitoring and alerting ownership. CloudWatch, Prometheus/Grafana, or equivalent; building actionable, low-noise alerts tied to SLOs and multi-window burn rates; owning alert routing and on-call hygiene
• Designing service, SLO, and executive-level dashboards in Grafana or equivalent that answer real operational questions
• Application instrumentation. Adding metrics, structured logging, and distributed tracing (OpenTelemetry or equivalent) directly in Python and Rust services
• Synthetic and transactional health checks. Building health-check capabilities into the application and scripted user-journey checks (CloudWatch Synthetics, Grafana k6, Checkly, or equivalent) that exercise real auth flows and critical transactions
• Incident management. Leading incident response, running blameless postmortems, and driving corrective actions to closure
• Strong Amazon EKS / Kubernetes operations background: Helm, RBAC, pod identity, GitOps, upgrades, troubleshooting
• Solid AWS experience across core services (VPC, IAM, S3, RDS, CloudTrail, CloudWatch, KMS)
• CI/CD experience in GitHub Actions and/or GitLab CI, including reliability and performance gates in pipeline
• Infrastructure-as-code (OpenTofu / Terraform)
• Agile delivery: trunk-based development, MR-driven change, automated quality gates
• Security and compliance awareness: secrets management, least privilege, audit evidence, ideally in NIST 800-171, CMMC, FedRAMP, or DoD environments
• US Citizenchip is required
• Rust experience strongly preferred
• Production Rust experience (async/Tokio, FFI with Python via PyO3, or similar)
• Load and performance testing (k6, Locust, or equivalent) and capacity planning
• Chaos engineering and fault-injection testing
• Progressive delivery (canary/blue-green with automated rollback on SLO or health-check failure)
• Experience with Big Bang (Platform One), IronBank, or hardened-image pipelines
• Prior work on systems pursuing or holding an ATO
• Contributions to observability tooling or Rust/Python open-source projects