Week 4 assignment support: Run TRACE on your product
1. Start here: what this assignment is
This guide has two paths. Path A is enough to pass. Do that first. Path B is stretch - tooling and an engineer brief - only if you have time.
Week 4 is TRACE, shared with the Engineering cohort. You own Trace, Read, and Analyze. You ship a must-pass checklist and at least one fix. Engineers go deeper on Codify and Enforce; you learn enough to lead that work.
Ready when
No product traces yet? Start with the sample pack
Download the Harmony Apartments SMS leasing-bot pack (20 traces + knowledge base). Path A is fully valid on this dataset. Use your own product traces too when you have them.
- Knowledge base
- Traces CSV (easiest in Sheets)
- Traces JSONL
- Pack README
- Answer key (only after you annotate): harmony-apartments-answer-key.jsonl
Tip
Session resources
Shared TRACE deck (Engineering): Evaluation, Monitoring & Shipping. Method: Hamel. Agents: Anthropic demystifying evals. Goldens / hill climbing: Addy Osmani. Tooling: Braintrust · Langfuse.
Critical
Stuck?
WhatsApp a concrete blocker ("on this input my product does X, I expected Y"), not "how are my evals looking."
Path A - enough to pass
Basic submission: enough to pass
- Review all 20 Harmony sample traces (CSV recommended) and/or 15-20 of your own product outputs.
- Write free-text notes on what went wrong (open coding, no scores yet). Do not open the answer key until you finish.
- Rank a short list of failures by frequency × impact (specific labels, not "quality issues").
- Build a small binary must-pass checklist (≥5 cases) covering top failures: mix of rule-checkable and judgment cases.
- Ship at least one fix aimed at a top failure (on your product, or a clear rewritten reply/prompt rule for the sample bot).
- Short report: what you found + the fix (Maven / share post).
2. Trace and Read (your job)
Capture what the product actually did: user input → steps → output. Read each run and journal. This is product work: deciding what "good" means for your users.
- 1
Open the sample CSV (default)
Skim the knowledge base first, then fill your_open_coding_notes, your_pass_fail, and your_failure_label for each of the 20 rows.
- 2
Open code
For each: what is good, what failed, what would a careful user notice. No metrics yet.
- 3
Optional: your product too
If you have live usage, repeat on 10+ of your own runs (screenshots or a simple sheet).
Watch out
Do not peek early
The answer key is for self-check after your taxonomy exists. Open coding first is the whole point.
Tip
Transcript vs outcome
Anthropic's language: the transcript is what the assistant said; the outcome is whether the real-world result is correct (row updated, email sent). Check both when your product takes actions.
3. Analyze - rank what matters
Cluster notes into failure types. Count. Prioritize by frequency × impact. Keep judgments binary (pass/fail). Fix cheap obvious issues on the spot so your checklist focuses on hard, recurring failures.
Think "hill climbing" (Addy Osmani): baseline → see failures → one change → re-check the must-pass list. That is how you improve without guessing.
4. Must-pass checklist
Each case is a specific input + what must always be true (binary). Example: "When a user asks for next appointment, it returns the correct one - pass/fail." Mark which cases a simple rule can check vs which need human (or later, validated AI) judgment.
Ready when
Discipline
Re-run the checklist every time you change the product. A list you write once and forget is useless.
5. Fix one thing
Change a prompt, workflow step, or guardrail that addresses a top failure. Re-run the relevant must-pass cases. Note what improved.
Path B - stretch (pick 1–2)
Annotate in Braintrust or Langfuse
StretchLog or paste traces, add open-coding notes and binary labels (e.g. grounding pass/fail) in the UI so eng can pick them up.
Engineer brief for Codify/Enforce
StretchOne page: ranked failures, must-pass cases, ask for binary checks tied to those failures, validated LLM judges (TPR/TNR - not raw agreement), and a dashboard where a failure rate goes down after a fix. Do not ask for generic "quality metrics."
Draft judge criteria
StretchWrite one binary judge prompt for a semantic failure and describe how you would measure TPR/TNR against your labels - without needing full CI.
6. What to submit
Maven / share
- Ranked short list of failures
- Must-pass checklist (screenshot or doc)
- Evidence of one fix shipped
- Optional Path B: tool annotations and/or engineer brief
7. Watch on this page
How to Build AI Evals in 2026
Mastering AI Evaluation: Playground to Production