Senior/Staff Software Engineer (Platform and Execution Model)

Trase Systems develops enterprise AI solutions that simplify the deployment, management, and optimization of AI in regulated environments. The Senior/Staff Software Engineer will build and own critical distributed systems and platform primitives for Trase OS, focusing on workflow execution, reliability, scalability, correctness, security, observability, and architectural leadership.

Responsibilities:

• Design, build, and own critical components of the core execution model (state machine, lifecycle, resource model, failure semantics)
• Build reliable distributed systems that behave predictably across retries, restarts, partial failures, and concurrent execution
• Develop platform APIs/SDKs connecting workflows, agents, tools, and product surfaces; drive versioning & compatibility
• Guarantee correctness via idempotency, deterministic replays, compensating actions, and data integrity
• Engineer reliability at scale: concurrency controls, rate limits, backpressure, sharding/partitioning, and workload isolation
• Build security & governance into the core: RBAC/ABAC, policy enforcement, fine-grained audit & lineage
• Deliver observability: distributed tracing, structured logs, metrics, and evaluation hooks; build an “explainable trail” of agent actions
• Own quality: design reviews, test strategy (unit, property, chaos), performance baselines, SLOs, incident response, and postmortems
• Mentor engineers, contribute to design reviews, and help establish strong engineering patterns across the platform

Requirements:

• 8+ years of experience building distributed/platform systems, including significant experience defining architecture across teams or domains
• 4+ years owning mission-critical runtimes or workflow/orchestration systems
• Deep expertise with durable execution (e.g., state machines, event sourcing, saga/compensation, idempotency, exactly/at-least-once semantics)
• Proven track record with security & governance in production systems (auth, RBAC, audit, policy)
• Hands-on with observability (Grafana or equivalent), including trace correlation across async boundaries
• Strong systems design across storage, queues, schedulers, and evented architectures; performance tuning under load
• Excellence in a modern language (e.g., Go, Rust, Java, or TypeScript) and cloud-native stacks (containers, CI/CD, IaC)
• Comfortable operating in regulated or high-assurance environments; bias toward correctness, clarity, and documentation
• Strong technical judgment and the ability to influence design decisions within a team and across closely related engineering areas
• Ability to incorporate advance LLM capabilities into system design and platform architecture decisions where appropriate
• Prior work on workflow engines (Temporal/Cadence/AWS Step Functions, Argo, Airflow) or serverless runtimes
• Experience with policy engines (OPA), secrets/KMS, or data-handling controls (PII/PHI)
• ML/LLM evaluation frameworks, tool/plugin architectures, or embedding model governance into execution
• Government or healthcare experience (HIPAA, audit readiness) and multi-tenant isolation