Software Engineer I - LLM Observability

Honeycomb builds observability tools, and this Software Engineer I role focuses on observability for AI and agents. The engineer will develop end-to-end capabilities, data infrastructure, and product experiences that help customers understand and debug AI systems.

Responsibilities:

• Deliver AI observability capabilities end to end, from ingestion to in-product visualizations
• Contribute to the data model and infrastructure that let us observe agents and their ecosystems across the Honeycomb product
• Build systems that analyze and surface AI-specific signals, such as cost, token usage, errors, and anomalies
• Collaborate with product and design partners to make sure what we build is usable, intuitive, and unlocks meaningful user workflows
• Share what you learn about exploring and understanding AI system behavior across conversations and workloads
• Iterate through fast feedback loops and experimentation, and share both wins and failures

Requirements:

• 3+ years of software engineering experience
• Experience shipping customer-facing features at a SaaS company, beyond internal tooling
• Full-stack range, with proficiency in backend development (Go and/or Python) and frontend development (TypeScript and React)
• A track record of working with product managers and designers
• Regular, hands-on use of AI tools in your day-to-day engineering work
• Experience being on call
• A track record of navigating ambiguity in a fast-moving field where requirements shift and experimentation is a core part of the job
• Bonus: Experience building LLM or agent-powered features in production
• Bonus: Hands-on experience with observability tooling, such as OpenTelemetry