Senior Machine Learning Engineer

Candid is a nonprofit that provides data and insights about the social sector. The Senior Machine Learning Engineer will take operational ownership of deployed machine learning and AI services, improve deployment practices, observability, and inference performance, and enable reliable collaboration between data scientists, data engineers, product teams, and software engineering teams.

Responsibilities:

• Take operational ownership of Candid’s deployed machine learning and AI services — monitoring for degradation, managing retraining cadences, coordinating handoffs from data scientists, and serving as the accountable point of contact for models in production
• Improve inference performance for deployed models, including complex graph inference models, applying techniques such as quantization, artifact slimming, batching, and efficient serialization
• Design and operate experiment tracking, model versioning, and artifact management so data scientists have a consistent, low-friction way to hand off work
• Build and maintain observability for ML and AI services — centralized logging, metrics, dashboards, and alerting for the services that matter most — and proactively reduce incidents through good deployment hygiene
• Establish and iteratively improve a repeatable deployment path for new ML services on AWS, including CI/CD integration and infrastructure-as-code patterns
• Monitor and actively manage AWS spend across ML workloads, including Bedrock token usage, compute sizing, S3 lifecycle, and publish cost visibility to the team
• Work closely with data scientists to understand model behavior, surface operational insights back to them, and translate research code into production-ready deployments
• Serve as the primary technical liaison between the Data Science team and Candid’s product and software engineering teams when ML or AI services are integrated into products and systems, defining integration contracts, APIs, latency and reliability expectations, and input/output schemas, and supporting those teams through integration
• Build and operate Amazon Bedrock-backed services and integrations where they are the right tool for the job
• Contribute to Candid’s secure-by-default posture for ML services, including IAM scoping, secrets handling, and compliance-aligned tagging
• Participate in technical planning and roadmap discussions for the Data Science team

Requirements:

• Applicants for this position must be legally authorized to work in the United States without employer sponsorship, now or in the future
• 4+ years of professional software engineering experience, including at least 2 years in a role where production ML systems were a primary responsibility (MLOps, ML engineering, or production-ML-focused data science)
• Strong proficiency in Python, including writing production-quality service code
• Hands-on, operational experience with experiment tracking and model lifecycle tooling in production, such as MLflow, Weights & Biases, or comparable systems
• Hands-on experience deploying PyTorch models in production, including familiarity with serving optimization techniques such as quantization, batching, ONNX Runtime, model serialization, container/artifact slimming, or cold-start mitigation on Lambda/Fargate
• Demonstrated experience deploying and monitoring ML models in production, with an understanding of model degradation, drift signals, retraining triggers, and artifact management
• Working knowledge of deploying Python-based ML services on AWS including Lambda, ECS/Fargate, S3, IAM, and CloudWatch. Deep infrastructure expertise is not required; comfort with the AWS ML deployment stack is
• Experience building or operating CI/CD pipelines for ML services
• A demonstrated track record of improving production reliability through observability and disciplined deployment and maintaining healthy systems over time
• Ability to work closely with data scientists, understanding their outputs, inheriting their work, and communicating operational decisions back to them clearly
• Demonstrated ability to work cross-functionally with software or product engineering teams, translating ML service capabilities into integration-ready contracts and supporting those teams through adoption
• Comfort owning work independently
• Strong written and verbal communication
• Sensitivity and respect for racial, gender, sexual orientation, and cultural differences
• Champions and represents Candid's core values: We're driven, direct, accessible, curious, and inclusive
• Production graph ML inference experience (PyTorch Geometric, DGL, or comparable)
• Experience with optimized CPU inference runtimes (ONNX Runtime, TorchScript) or serving frameworks
• Experience with Amazon Bedrock or another managed LLM service (Anthropic API, Azure OpenAI, Vertex), and/or operating RAG systems or agentic AI workflows
• Familiarity with infrastructure as code (CloudFormation, CDK, Terraform)
• Experience with high performance programming languages for ML serving (Rust, Go, etc.) is a nice to have
• Experience at a nonprofit or mission-driven organization, and mentoring data scientists on production engineering practices