Senior Cloud and mlops platform Engineer

The company is a data science organization that provides a secure and compliant software platform for running machine learning workloads at scale, including synthetic data applications for defense. The Senior Cloud and MLOps Platform Engineer will build and operate AWS GovCloud machine learning infrastructure, synthetic data pipelines, model training workflows, and auditable lineage systems, with later support for air-gapped high-performance computing environments.

Responsibilities:

• Write Terraform modules, configuration and runbooks that a partner team uses to build and monitor a SageMaker-based environment in GovCloud, and review the result against the definition
• Wrap a partner's physics simulation engine as containerized SageMaker Processing jobs and orchestrate generation with Step Functions
• Build training, tuning and inference pipelines on GPU instances within fixed monthly run budgets
• Build the constraint framework: hard caps, admission control and pre-flight validation, checkpointing and resume, early stopping, and an experiment registry
• Implement lineage: signed manifests linking every dataset and model to its inputs, code, container digest and configuration
• Build container images in GitLab CI with offline dependencies, so they also run under Apptainer on a batch-scheduled cluster with no internet access
• Write clear runbooks and contribute to monthly technical reports

Requirements:

• Five or more years in cloud or platform engineering, including two or more building machine learning infrastructure on AWS
• Production Terraform, including modules written for other teams to apply
• Amazon SageMaker (Processing, Training, hyperparameter tuning, endpoints) and Step Functions
• Docker and CI/CD (GitLab preferred), with images pinned by digest and dependencies vendored
• AWS security fundamentals: IAM least privilege, KMS customer managed keys, VPC endpoints and PrivateLink, Secrets Manager
• Experience in a regulated environment: AWS GovCloud, FedRAMP High, DoD IL4/IL5, or CUI under NIST SP 800-171
• Strong Python
• U.S. citizenship required
• eligibility to obtain one required
• HPC experience: Slurm or PBS, Apptainer or Singularity, Lustre, offline software stacks
• MLflow or a comparable experiment tracker; SageMaker Model Monitor
• AWS cost controls and FinOps for GPU workloads
• Multi-account AWS Organizations under Control Tower, Lake Formation, API Gateway
• DoD Risk Management Framework documentation or CMMC
• AWS certifications (Solutions Architect Professional, Machine Learning, Security)
• An active Secret clearance is a bonus