Trace and Read: error analysis
Trace means capturing full records of what your system did: input through every step to output. Read means going through those traces by hand and journaling what went wrong - open-ended, like qualitative research.
What a full trace contains
- System prompt and user turns
- Tool calls and tool results
- Retrieved context (for RAG)
- Final output (and, for agents, environment outcome - not only what the model said)
Ready when
Transcript vs outcome (Anthropic)
The transcript is the message history. The outcome is final state in the world (row written, ticket created, flight booked). Grade both when your agent mutates state - a polite "done" in the transcript can still be a failed outcome.
Open coding
Read traces one by one. Write free-text notes. No metrics yet. Even as an engineer, do not skip this: reading tells you which failures are actually happening, not the ones you imagined.
Tip
How many?
Aim toward ~100 over a real product lifecycle; stop adding categories when ~20 new traces add none. You can start in a CSV - the tool is optional, the reading is not.
Ready when
Homework sample pack
No capstone traffic yet? Use the Harmony Apartments 20-trace pack for Path A (JSONL + CSV + knowledge base). Answer key only after you finish open coding.