An agent plans, calls tools and works over many steps before it answers. Every one of those steps is a place to fail, and the answer at the end can look fine regardless. This is the feature type where the standard eval, input in and final output out, is furthest from being enough.
What typically breaks in an agent after a model change
The agent picks a different tool
Tool selection is a model behaviour. After an update the agent reaches for search where it used to reach for the database, or calls a tool twice where it called it once. Sometimes the answer survives. Sometimes it is right by accident, which is worse, because the run passes and the path is broken.
Arize Phoenix describes reconstructing two signals from an agent trace (the ordered record of one run, every step with its timing, arguments and return): tool_path, the ordered tool calls with their arguments, and tool_io, what each tool was called with and what it returned, and warns that a correct-looking answer can come from a broken path. [Arize AI]
The right tool is called with wrong arguments
A date range off by a month. A filter dropped. An identifier from the wrong entity. The tool returns something, the agent uses it, and the answer is confidently derived from the wrong slice of your data. Argument construction is the single most checkable agent failure, because arguments have a shape, and it is the one most often left unscored.
A tool errors and the run continues
The tool returns a 500, an empty result or a timeout. A well-built agent retries or reports. A model update can change which of those it does. When the agent proceeds as if the call succeeded, the failure is invisible at the endpoint: there is an answer, it reads well, and it rests on nothing.
The agent pursues the wrong goal with every step looking reasonable
Future AGI describes scoring an agent at four resolutions[src], end to end, trajectory (the ordered path the agent took, judged as a path rather than by its last step), step and turn, and writes that a high end to end score can hide bad tool calls, while strong per-step scores can still miss that the agent pursued the wrong goal. [Future AGI] Both directions are real, which is why the set keeps more than one resolution rather than picking the cheapest.
The trajectory grows and nobody notices until the bill does
Runs get longer. More tool calls, more retries, more tokens, and more spans (one recorded step inside a request, for example a single tool call, with its inputs and outputs) to score. Future AGI writes that agent traces in contemporary frameworks are routinely 15-40 spans[src] deep, naming OpenAI Agents SDK, LangGraph, CrewAI, AutoGen and Google ADK. [Future AGI] Against that depth, a few extra spans per run is not visible in any output and is very visible in a monthly invoice.
An action lands that should never have been taken
The AI Incident Database records incident 1152, dated 18 July 2025[src], in which an LLM-driven Replit agent reportedly executed destructive commands during a code freeze, deleted production data, generated fabricated records and incorrectly claimed rollback was impossible. [AI Incident Database (Responsible AI Collaborative)] That record is compiled from reported sources rather than an operator's own postmortem, and the database's own wording is reportedly. It is on this page as a published example of a trajectory failure that an end-point score would not have surfaced.
Which scorers apply
Arize Phoenix writes that end-to-end evaluation only inspects the two endpoints, the input and the final output, and that the interesting failures happen in the middle, in reasoning, tool selection and the decision path, where those endpoints cannot show them. [Arize AI] So the scorer (one automated check that turns an output or a step into a number or a pass and fail verdict) set for an agent is a stack, not a single measure. LangSmith's four evaluator types still apply inside each layer: human review, rule based code functions, LLM as judge (a model that scores another model's output against a written standard) in reference free and reference based variants, and pairwise comparison. [LangChain]
- End to end outcome scorer. Catches a run that failed to achieve what the case asked for. Misses everything about how it got there, including a right answer reached by a broken path.
- Trajectory scorer over the tool path. Catches a path that diverges from the expected sequence of tool calls. Misses cases where a different path was legitimate, so the case needs an expected path or an allowed set, not a single script.
- Step level tool selection scorer. Catches the wrong tool at a named step. Misses whether the step should have existed at all.
- Turn level scorer. Catches a bad exchange inside a multi-turn run, where the agent talks to a user between tool calls. Misses purely internal steps with no turn attached.
- Tool argument code check. Catches malformed, missing or out-of-range arguments against the schema of each tool. Misses arguments that are well-formed and semantically wrong, which is where the judge comes back in.
None of these can read a trace that does not carry tool arguments and returned values. Where your application does not emit them, the set falls back to end-to-end scoring and the gap is written into the report rather than left implied. The full method is EV-AGENT, and selection is EV-SCORE.
What a baseline includes for this feature type
The eval baseline for an agent starts with a definition, because without one there is nothing to score consistently.
- A written definition of one run: where it starts, where it ends, and which spans belong inside it.
- Real traces captured from your framework, with span names, tool arguments and tool returns, rather than one log line per request.
- An expected outcome per case plus an expected tool path or an allowed set of paths, labelled by your domain expert.
- Failure cases on purpose: a tool that errors, an empty result, a timeout, an ambiguous instruction, with the behaviour you want written down for each.
- A schema per tool, so argument checks run in code and need no calibration.
- threshold (the number agreed with you at which a measured drop counts as a finding on this dataset) values per resolution rather than one for the feature, since an end-to-end drop and a tool selection drop are different findings. Galtea recommends a tiered release gate where blocking failures such as safety or policy violations trigger a deploy block at any rate, while regressions (a case that passed on the previous run and fails on this one) gate on the change in fail rate. [Galtea] The gating design is EV-THRESH.
Which plan fits
Up to three features at $4,400 a month is the first plan that includes span-level evals, and for an agent that is the whole point. On One feature you would get the end-to-end layer and the rerun triggers, and you would not get the middle of the run scored, which is where agents fail. If several teams ship agents through a shared platform and you want the suite wired into a pull request gate, that starts on Up to eight features. The honest limit is the same one that applies everywhere on this site: an eval reports what changed against an agreed dataset before a change ships, and it does not stand between your agent and your production systems. Permissions, confirmations and rollback are yours. The bounds are written out in what we cannot do, and the document-fetching half of agent behaviour is covered on the RAG hub.