1. Start here: what this assignment is

This guide has two paths. Path A is enough to pass. Do that first. Path B is stretch: pick one or two only if you have time.

Session 4 is TRACE: Trace, Read, Analyze, Codify, Enforce. Path A proves you can find real failures, turn them into automated checks, fix one, and show a metric move from a Streamlit UI.

Path A - enough to pass

2. Trace and Read (45-90 min)

A trace is the full record: user input, system prompt, tool calls, retrieved context, intermediate state, and final output. Anthropic also distinguishes the transcript (what happened in the messages) from the outcome (final state in the environment, e.g. did a booking actually exist).

1. 1

   Load the sample pack (default)

   Open the knowledge base, then the JSONL or CSV. Each row is one SMS leasing-bot run: user_input, retrieved_context, tool_calls, assistant_output.

2. 2

   Open code all 20

   For each trace, write a free-text note: what is good, what failed, what surprised you. Fill your_open_coding_notes / your_pass_fail / your_failure_label in the CSV if you like. No scores yet.

3. 3

   Optional: your capstone too

   If you have live runs, repeat on 10+ of your own. Log in Braintrust or Langfuse if you want annotations next to the span tree.

3. Analyze - taxonomy and rank

Cluster related notes into specific failure modes. Prefer "ignored bathroom-not-connected constraint" over "temporal issues." Count occurrences. Prioritize by frequency × impact. Keep judgments binary (pass/fail), not Likert 1–5.

• Instruction / constraint miss
• Grounding failure / hallucination
• Formatting error (markdown in SMS, over-long dump)
• Missing handoff (bot fakes an action it cannot take)
• Tool-use error (wrong tool, bad args, ignored tool output)
• Safety / injection compliance

This ranked list is the spec for your eval suite - same idea as Addy’s "hill climbing": baseline → analyze failures → one change → re-measure.

4. Codify - code assertions first

Reach for deterministic checks before LLM-as-judge. Examples: valid JSON / schema; required fields; no markdown characters in a plain-text channel; length budget; URL allowlist; forbidden PII/secrets; latency/token ceilings.

I have a capstone AI app. Here is my failure taxonomy and 3 example traces (paste). Help me:
1) Store traces as JSONL (input, context/tools, output, human notes).
2) Implement two pytest (or Promptfoo) assertions for [failure A] and [failure B].
3) Add a Streamlit page that runs the suite and shows pass/fail counts.
4) Keep everything binary pass/fail with a one-line reason on fail.

5. Fix one thing and show the metric move

Change the prompt, retrieval, tool, or guardrail that addresses your top failure. On the sample pack, that can be a revised system prompt + re-scoring the 20 outputs (or a scripted "fixed" output column) so pass rate moves. On your capstone, change the real system and re-run. Screenshot before/after pass rates (or failure counts). That is your Maven evidence.

Path B - stretch (pick 1–2)

6. What to submit

7. Watch on this page