Agent Monitoring Platform
Datadog for AI agents — trace every trajectory, score quality, attribute cost-per-task, and catch failures before customers do.
“Observability and evals for AI agents — see what they do, why they fail, and what they cost.”
Role in the platform
The reliability layer. Ships standalone and as a ContextOS module; gives the Agent Marketplace its trust signals.
Overview
A platform to monitor, debug, evaluate, and cost AI agents in production: trajectory traces (reason → tool → observe steps), a failure taxonomy, an eval harness (golden tasks + LLM-as-judge), cost-per-task, and replay.
Framework-agnostic across LangGraph, CrewAI, the OpenAI SDK, and the Claude Agent SDK via an SDK + OpenTelemetry.
The problem
Everyone is shipping agents; almost no one can see what they do, why they fail, or what they cost.
LLM-call observability exists (LangSmith, Langfuse, Helicone) but true agent-trajectory observability + evals is under-served.
What it is
A recording SDK instruments an agent; spans flow over OpenTelemetry into ingest and Postgres, then surface as a trajectory viewer, an eval harness, cost-per-task, and a failure taxonomy.
Built for the family's own agents first, then sold — standalone and as a ContextOS module.
Key features
Tech stack
Why it's different
- Agent-trajectory observability (multi-step plans, tool-call graphs) versus LLM-call logging.
- Evals + failure taxonomy + cost attribution in one place.
- Framework-agnostic — not tied to one orchestration library.