Staff Software Engineer, Search & Distributed Systems

ACV Auctions is a technology company revolutionizing the automotive industry by creating a trusted digital marketplace for buying and selling cars online. They are seeking a Staff Software Engineer who will be responsible for the scalability, reliability, and data resiliency of their Search infrastructure, particularly focusing on Elasticsearch. The role involves designing and architecting systems to prevent failures, managing data lifecycles, and driving engineering excellence across the organization.

Responsibilities:

• Architect for Scale: Design, configure, and scale our Elasticsearch clusters. You will define our global strategies for shard routing, Index Lifecycle Management (ILM), heap tuning, and data tiering to support massive auction throughput
• Master the Failure Modes: Anticipate and engineer away points of failure. You will design circuit breakers, implement backpressure mechanisms, and tune asymmetric timeouts to prevent retry storms between our BFFs, K8s services, and the Search layer
• Expert Troubleshooting & IR: Act as the ultimate technical escalation point for complex, cross-system performance degradation. You will dive deep into JVM metrics, Garbage Collection pauses, K8s network bottlenecks, and slow logs to uncover and remediate root causes
• Holistic System Ownership: Manage the entire data lifecycle. You will optimize the ingestion pipelines syncing our event datastreams driven by producers and consumers (Kafka) to Elasticsearch, ensuring eventual consistency and data integrity at scale
• Drive Engineering Excellence: Draft authoritative architectural Blueprints, SOPs, and Runbooks. You will elevate the surrounding engineering culture by coaching teams on distributed systems design, observability best practices, and incident management
• Modernize & Innovate: Scan the horizon for emerging technologies. You will help evaluate and integrate next-generation search capabilities (e.g., Vector Search, RAG architectures) to support our broader AI and machine learning initiatives

Requirements:

• 8+ years of software engineering experience, with at least 3+ years operating at a Senior or Staff level focusing on distributed systems and high-throughput platforms
• Deep, authoritative knowledge of Elasticsearch internals. You have managed large-scale clusters and deeply understand mapping, analysis, query optimization, cluster state management, and split-brain mitigation
• Proficiency in the systems upstream and downstream of Search. You have hands-on experience with Kubernetes (EKS/GKE), API Gateway/BFF architectures, and event streams (Kafka)
• A proven track record of implementing fault-tolerant patterns (retries, rate limiting, circuit breaking, dead letter queues) in microservice architectures
• Expert-level ability to instrument systems and diagnose complex performance issues using modern observability stacks (Datadog, Prometheus, Grafana, OpenTelemetry)
• Strong communication skills with a proven ability to influence cross-functional teams, build consensus around architectural decisions (the Knoster model!), and mentor mid-level and senior engineers
• Experience with Infrastructure as Code (Terraform, Helm) for stateful applications
• Familiarity with FinOps practices, specifically optimizing Elasticsearch compute and storage costs
• Experience integrating AI-assisted development tools into your daily workflow