Agent trajectory and world-state grading

Deep diveSelf-paced

Agent trajectory and world-state grading

A trajectory is the full sequence of an agent's states and actions, not the final message: every model turn, every tool call with its arguments, every observation, in order. Grading a trajectory means scoring the *path*, and grading world state means checking what actually changed in the environment, independent of what the agent claims happened.

Trajectory grading approaches

Exact-match trajectory

Strict

Compare the agent's tool-call sequence against one reference path. Simple to compute, brittle, most real tasks have more than one valid route to success.

In-order subset match

Common

Check that the required tool calls happened, in a valid relative order, allowing extra exploratory steps around them. Matches how tasks are usually actually solved.

Step-wise LLM judge

Flexible

An LLM judge scores each step (or the whole trajectory) against a rubric: was this tool call necessary, was the argument correct, was this a redundant retry.

Metrics worth tracking per trajectory

Tool-call precision - of the tools it called, how many were actually needed for this task.
Tool-call recall - of the tools a correct solution needed, how many did it actually call.
Redundant steps - repeated identical calls, usually a sign of a confused retry loop, not resilience.
Steps to completion - a proxy for cost and latency; rising step counts on the same task class over time is an early warning sign.

World-state grading: check the environment, not the transcript

For any agent that mutates state (books, sends, writes, deletes), the ground truth is not what the agent said; it is the state of the system afterward. Snapshot the relevant state before the run, run the agent, snapshot again, and diff against an expected state rather than parsing the agent's own claim of success.

world-state diff (sketch)

def grade_world_state(before: dict, after: dict, expected_after: dict) -> dict:
    """Compare actual post-run state against the expected outcome for this task,
    independent of anything the agent said in its final message."""
    diffs = {
        key: {"actual": after.get(key), "expected": expected_after.get(key)}
        for key in expected_after
        if after.get(key) != expected_after.get(key)
    }
    return {
        "world_state_correct": len(diffs) == 0,
        "mismatches": diffs,
    }

# Example: a booking agent task
before = {"booking:B-42": None}
after = query_real_system("booking:B-42")   # what actually got written
expected_after = {"booking:B-42": {"date": "2026-08-14", "customer": "Northwind Robotics"}}
result = grade_world_state(before, after, expected_after)

Critical

A confident transcript is not evidence

"Booking confirmed" in the final message proves the model produced that sentence. It proves nothing about whether the booking tool was called correctly, or at all. Ground every claim of a completed action in a state check on the real system, not the model's own report of itself.

Building this into TRACE

Trajectory and world-state checks slot into Codify as an additional grader type alongside your code assertions and LLM judges, and into Enforce the same way: run them on every capstone change that touches agent logic, and watch tool-call precision/recall and world-state correctness the same way you already watch pass rate.

Watch out

Common misconceptions

Grading only the final text message for a task that mutates external state.
Requiring an exact-match trajectory when the task genuinely has multiple valid solution paths.
Never actually querying the system-of-record to confirm the claimed outcome happened.
Ignoring redundant tool-call loops because the task "eventually succeeded."

Go deeper (optional)