Session 4: TRACE Evals
11 lessons · Back to full syllabus
What you keep
The full TRACE loop as a system you build - Trace, Read, Analyze, Codify, Enforce. Code assertions first; validate every LLM judge with TPR/TNR.
You ship
An eval suite your capstone runs against, visible from a Streamlit UI.
Live session resources
Week 4 session: Evaluation, Monitoring & Shipping
TRACE live deck - golden sets, rule checks, LLM-as-judge, observability, and shipping discipline.
Open resourceWeek 4 assignment build guide
Path A / Path B TRACE homework: taxonomy, code assertions, one fix, Streamlit; stretch validated judge and goldens.
Open resourceSample traces pack (Harmony Apartments)
20 SMS leasing-bot traces + knowledge base for Path A when you do not have capstone traffic yet.
Open resourceHow to Build AI Evals in 2026 (YouTube)
Step-by-step evals framing - watch before or after the live session.
Open resourceMastering AI Evaluation: Playground to Production (YouTube)
From ad-hoc playground checks to production eval loops.
Open resourceBraintrust
Primary classroom tooling - traces, human labels, scorers.
Open resourceLangfuse
Open-source backup for tracing and evals.
Open resourcePromptfoo
CI/CD evals and OpenAI's migration path off hosted Evals.
Open resourceAnthropic: Demystifying evals for AI agents
Task, trial, grader, transcript vs outcome.
Open resourceSandboxed agentic evaluation
Hermetic fake tools, per-trial isolation, seed→run→snapshot→diff, and when sandboxes must meet the real world.
Open resourceLessons
The vibe-check trap
"It works on the examples I tried" is not evaluation - and for non-deterministic systems it is dangerous.
Trace and Read: error analysis
Capture full records, then read traces by hand - qualitative research on real failures.
Analyze: cluster, count, prioritize
Binary pass/fail judgments, ranked by frequency and impact - the spec for your eval tooling.
Codify: build the eval suite
Code-based assertions plus validated LLM-as-judge - an unvalidated judge is just another vibe check.
Enforce: evals in the loop
Wire evals into the development loop so every change gets measured automatically.
Evaluating agents: trajectories, not just final answers
A polite final message can hide a wrong tool call, a silent retry loop, or an action that never actually happened.
Aligning an LLM-as-judge to human labels
The actual calibration workflow: blind labels, held-out validation, TPR/TNR, and re-validating after every model change.
Agent trajectory and world-state grading
Score the path the agent took and what actually changed in the environment, not just what it said.
Sandboxed agentic evaluation
Evaluate agents where side effects cannot happen: hermetic fake tools, per-trial isolation, and world-state diffs.
Session 4 assignment support: Build and run TRACE on your capstone
Path A: traces, taxonomy, code assertions, one fix, Streamlit. Path B: validated judge, goldens, CI sketch.
Build and run TRACE on your capstone
Trace, read, analyze, codify, and wire evals to run on demand - show a metric moving after a fix.
Lessons in this module