Live
The vibe-check trap
"It works on the examples I tried" is not evaluation. For a system with non-deterministic components it is dangerous: a tweak can silently break cases you never re-tested.
Why this week exists
Success is iteration speed with confidence. That requires real evals - a process, not a vibe. TRACE (Trace, Read, Analyze, Codify, Enforce) is the loop you will build for your capstone.
Tip
Vendor tools move; the method lasts
OpenAI's hosted Evals platform is being deprecated (read-only Oct 31, 2026; shutdown Nov 30, 2026). Their migration path points at Promptfoo. We demo Braintrust (and Langfuse as backup) for classroom UI - not OpenAI Evals. Own TRACE, not a dashboard brand.
Watch: How to Build AI Evals in 2026
Watch out
Common misconceptions
- Asking ChatGPT "was this output good?" and treating a confident yes as evidence.
- Collecting generic helpfulness scores that miss product-specific failures.
- Skipping error analysis and jumping straight to automated metrics.
Go deeper (optional)