TRACE: Trace and Read (error analysis is your job)
TRACE stands for Trace, Read, Analyze, Codify, Enforce. This lesson teaches the first two steps, which are the most important, and which are squarely product work rather than engineering - this is your part of the loop, not something to wait on an engineer for.
Trace means capturing what your product actually did. A trace is the full record of a single interaction from the user's input through to the final output, including any steps in between. You cannot evaluate what you cannot see, so the first move is always to collect real examples of your product in action. If you have no real usage yet, dogfood it yourself across realistic cases.
Read means reading those traces one by one and noting what went wrong - open-ended notes, no metrics yet. This is qualitative research, and it is the highest-ROI activity in evaluation.
Ready when
Transcript vs outcome
Anthropic's language: the transcript is what the assistant said; the outcome is whether the real-world result is correct (row updated, booking made). When your product takes actions, check both.
Why is this your job? Reading traces is about judgment - knowing what "good" means for your users. An engineer can measure a failure; deciding which outputs are failures requires product thinking.
Ready when
Homework sample pack
No product traffic yet? Use the Harmony Apartments 20-trace pack (CSV is easiest in Sheets). Answer key only after you finish open coding.