Senior/Staff Software Engineer, Data Platform

Carrum Health is a healthcare technology company that connects employers and specialty care providers through a value-based care marketplace. The Senior/Staff Software Engineer, Data Platform will build and modernize production data platform services, automation, infrastructure, observability, and AI-assisted ingestion systems while reducing operational toil across healthcare data workflows.

Responsibilities:

• Own and extend the data pipeline platform: The core data pipeline is a Ruby/Rake/Bash orchestration system with 230+ active carrier integrations. Your job is to drive its evolution: adding automation, testing, structured observability, and configuration-driven abstractions that reduce the cost of every new integration. Long-term we are moving toward Python and Spark-based compute; you will help shape that path
• Build production platform services: Write the Python and SQL for batch jobs, Lambda functions, and data services that ingest, transform, validate, and route data. The output is software with tests, contracts, monitoring, and on-call ownership, not a collection of scripts
• Build platform tooling that creates leverage: Some of the highest-value work here is building tools for the team: code generators for eligibility automation, integration testing infrastructure, classification Lambdas, data-fix utilities, scaffolding for new data models. You think in systems, not one-offs
• Treat infrastructure as a first-class engineering discipline: Design and extend Terraform modules. Build environment promotion, secret management, IAM patterns, and deployment pipelines. Migrate operational config from runbooks into Terraform-managed resources. The platform's reliability is a function of its infrastructure design
• Build observability into the platform: Structured JSON logging, Datadog monitors and log pipelines, SLOs, data quality checks (including statistical anomaly detection like EWMA control charts), and automated alerting, designed in at the architectural level, not bolted on after
• Lean into AI as a force multiplier: Bedrock (Claude) is in production today, used for job failure analysis and summarization. Our engineering culture is built around multi-agent AI workflows: every engineer uses AI tooling daily and we have a full agentic development setup in active use. Extend this foundation into ingestion tooling, data classification, and transformation assistance. We want someone who is hands-on with this technology, not just curious
• Eliminate operational toil through software, not heroics: The team today carries meaningful operational load: recurring eligibility and claims loads, SFTP setup, PGP key management, data provider switches, launch automation. You will take on some of this work in your first months, not as a tax, but as the curriculum. You will learn exactly where the system is fragile and use that knowledge to drive what you build next. The explicit goal is to make this class of work disappear
• Drive platform modernization: Evaluate and integrate Databricks as a near-term roadmap item. Drive the architectural decisions that come with it: storage layout, compute model, governance, and migration path
• Raise the engineering bar: Define patterns for testing, code review, CI/CD, deployment, and incident response. Document architectural decisions. Write code the next engineer wants to extend
• Partner across the organization: Work with analytics engineers, data scientists, software engineers, and client-facing teams. Translate ambiguous requirements into well-scoped platform capabilities
• Learn the domain: Develop fluency in healthcare data: medical claims, insurance eligibility, HIPAA, and build a platform that is operationally and clinically correct

Requirements:

• 5+ years of professional software engineering experience, with a track record of owning production systems end to end
• Strong Python and SQL. The batch jobs and Lambda functions that do the real work here are Python. SQL is Trino/Athena dialect via dbt. You write clean, tested, well-structured code in both
• Production-grade software engineering practices. Strict testing discipline requiring comprehensive automated test coverage (unit, integration, contract) on all new code
• CI/CD comfort (GitHub Actions, CircleCI, or comparable)
• A commitment to shipping code that holds up under load and over time
• Deep AWS experience. ECS, Batch, Lambda, S3, RDS, Athena, Step Functions, EventBridge, Transfer Family, IAM, Secrets Manager, and CloudWatch. You have operated and debugged production systems on this stack
• Terraform fluency. Not "I've used Terraform" — you have designed or extended Terraform modules that other engineers built on. You understand environment promotion, secret management, and IAM patterns
• dbt proficiency. Modeling, testing, documentation, and the discipline to deprecate what shouldn't exist
• A platform engineering mindset. You think in terms of systems, contracts, and reusable abstractions rather than one-off solutions. You can point to tooling or platform work you have built that created leverage for engineers beyond yourself
• Strong debugging and incident response instincts. When something breaks repeatedly, you fix the system that allows it to break, not just the symptom
• High bias for action and self-unblocking. A proven track record of driving stalled tasks to resolution through proactive cross-team communication rather than waiting for direction
• Daily AI usage in your engineering workflow. This is not a nice-to-have. We expect you to use AI tools every day to ship code, design systems, debug, and document. You hold full responsibility for understanding and defending your code submissions
• Maintain rigorous technical documentation. Own your features from start to finish: centralize project decisions, technical notes, and action-oriented blocker histories (naming specific dates and next steps) in Jira. Write Technical Design Documents before implementation begins on large projects
• LLM/RAG experience. Bedrock (Claude) is in production. We are actively building AI-assisted ingestion and classification tooling. Experience with LLM APIs, RAG architectures, vector stores (pgvector, FAISS), or agentic frameworks (LangChain, LlamaIndex) is a meaningful advantage
• Experience with Databricks or comparable lakehouse platforms (Delta Lake, Spark, Iceberg), directly relevant to our near-term roadmap
• PostgreSQL depth: data modeling, indexing, query performance, and understanding of how application-layer data flows into the warehouse
• Experience with streaming or CDC architectures (Kafka, Kinesis, Flink, Debezium, or similar), real-time data delivery is a current need
• Experience in healthcare, insurance, or another highly regulated industry. Familiarity with claims data formats (837P/I), eligibility files (834/820), HIPAA compliance, and the operational realities of working with insurance carriers
• Familiarity with SFTP-based data exchange, PGP encryption, and integrating with external data providers
• Experience with data quality frameworks (dbt tests, Great Expectations, Monte Carlo) and proactive observability with Datadog or comparable tooling
• Comfort reading and navigating Ruby/Rake. Our current data pipeline is Ruby-based. You don't need to be a Ruby engineer, but you need to be able to read it, debug it, and extend it while we migrate toward Python