Evaluating agents: trajectories, not just final answers
Everything so far in TRACE grades a final answer: one input, one output, one judgment. An agent breaks that shape. It takes several steps, calls tools, and changes things in the world before it ever produces a final message, so grading only the last message can score a broken agent as a perfect one.
The failure a final-answer grader cannot see
An agent says "booking confirmed" and your judge marks it correct because the sentence is well-formed and polite. But it called the booking tool with the wrong date, then silently retried three times against the wrong customer record before giving up and fabricating a confirmation. The transcript looks fine. The trajectory (Anthropic's term for the full sequence of thinking, tool calls, and observations) tells the real story, and the outcome (was a booking actually created, with the right fields, in the real system) is the only thing your user experiences.
Ready when
Three things to grade, not one
Final answer (what it said), trajectory (the steps it took to get there), and world state / outcome (what actually changed). Grading only the first is the single most common way agent evals give false confidence.
What this unlocks
- Catching agents that reach the right answer by an unreliable or unsafe path (a fluke, not a competence).
- Catching agents that say the right thing but never actually completed the action (a lie, not a success).
- Debugging *where* in a multi-step run things went wrong, not just *that* they went wrong.
The two Deep Dives on this page go further: Aligning an LLM-as-judge to human labels covers making a judge you can trust on semantic calls, and Agent trajectory and world-state grading covers the mechanics of scoring the steps and the outcome, not just the words.
Watch out
Common misconceptions
- Running your existing single-turn eval suite against an agent and assuming it covers agent-specific failure modes.
- Trusting a polite, confident final message as proof the underlying action succeeded.
- Skipping trace capture for tool calls because "the final answer is what matters to the user" - it is not what tells you the system is reliable.
Go deeper (optional)