TRACE
Evals

Agenda

What we'll cover.

You leave with: the TRACE loop, a ranked failure taxonomy habit, and Path A homework: open-code 20 traces, write assertions, ship one fix with a metric move.

Part 1 · Why Evals

The problem.

Part 1 · Why Evals

What is an AI eval?

Concrete example: Capstone SMS leasing bot. Eval set: 20 real (or sample) user texts. Assertion: "must not invent a unit hold." Score: 14/20 pass, fix prompt, 18/20. That delta is evidence.

Part 1 · Why Evals

Your three scoring tools.

Code-based where possible. Model-based where necessary. Human to calibrate. Own TRACE, not a vendor brand. Today's live tools: Langfuse (see traces) and the OpenAI Evals dashboard (run graders). Hosted OpenAI Evals is sunsetting: still fine for this classroom demo. Promptfoo or Braintrust for lasting setups.

Part 1 · Why Evals

The TRACE loop.

Five steps, in order. Each one produces the input to the next. You cannot skip a letter. Next we define what a trace actually is, then Read and Analyze.

Part 2 · Trace

What is a trace?

In TRACE, the letter T means: capture these records on purpose (Langfuse, Braintrust, CSV. Tool is optional. The record is not). Next: see one live in Langfuse.

Part 2 · Trace

What goes inside a trace.

Transcript vs outcome. The transcript is the message history. The outcome is final state in the world. A polite "done" can still be a wrong refund. Grade both when your agent changes anything.

Part 2 · Trace · Live

Langfuse: see a trace live.

Live now: open Langfuse, pick a recent trace, narrate messages, tools, cost. Scores and LLM-as-judge can wait until Codify. Today this click-path is only the letter T. Method is TRACE. Langfuse (or Braintrust) is a lens. Do not marry a dashboard brand.

Part 2 · Read

Read before you write a check.

Design in the abstract and you test what you imagine. Read real outputs and you test what actually breaks.

Part 2 · Analyze

Count, then rank.

Frequency times impact. Safety is rare (1) but Critical, so it still jumps the queue. Open-coding stays pass or fail. Judges can return a score. Your eval applies the cutoff. This ranked list is the spec for what you Codify next.

Part 3 · Codify

Code-based grader first.

Categorize this support ticket into one of: billing, how_to, bug_report, feature_request, or other. Respond with only the category word, nothing else.

Part 3 · Codify

What happens with harder cases.

"sales_inquiry" is not a valid category. The model says "other," which is correct. The grader says FAIL. The model is right. The eval is wrong. That is Codify-before-Read: a grader for a failure you imagined.

Part 3 · Codify

New task: email quality.

Part 3 · Codify

When you need a judge: score plus cutoff.

You are evaluating a customer support email response.

The customer's issue was:
{{item.input}}

The AI's response was:
{{sample.output_text}}

The response should meet these criteria:
{{item.ground_truth}}

Score from 1 to 7:
7 = Fully meets all criteria, appropriate tone, helpful
4 = Meets some criteria but misses others
1 = Fails to meet criteria, wrong tone, or harmful

Reply with the integer score only, then one short reason.
Be strict. Deduct for missing requirements, wrong tone, "as an AI".

Part 4 · Iterate

Same eval. Same data. Same graders. One targeted change.

You are a customer support agent for CloudApp.
Draft a professional email response.

Rules:
- Be empathetic and acknowledge the customer's situation
- Be specific, don't give generic advice
- For urgent or critical issues, treat them with appropriate urgency
- Do not say "as an AI" or similar hedging
- Keep your response under 150 words
- Sign off as "The CloudApp Support Team"

Part 5 · Agents

Grade more than the final message.

Components of an eval for agents: task, trial, trajectory, outcome. A polite closing does not mean the refund landed. Grade both layers — next slide: LLM-as-judge on full transcripts.

Part 5 · Agents

Agent grading with LLM-as-judge.

You are evaluating an agent transcript (trajectory + outcome).

The full run was:
{{item.input}}

Score against these criteria:
{{item.ground_truth}}

Score from 1 to 7:
7 = Meets all criteria — correct tools, correct final state
4 = Partial — tone OK but tool use or outcome wrong
1 = Fails criteria, unsafe, or worse outcome than before

Grade tool calls and final state, not only the last message.
Reply with the integer score only, then one short reason.

Red team into Codify: indirect injection in docs, tool misuse, state corruption, cascading failures. Every adversarial prompt becomes a future test case.

Part 6 · Enforce

Ship decision matrix.

Regression suites go in CI. Capability evals are what you run when you are trying to get better.

Part 6 · Enforce

From a meeting to a merge gate.

on:
  pull_request:
    paths: ['prompts/**', 'agents/**']

jobs:
  eval:
    steps:
      - run: promptfoo eval --config evals.yaml
      # fails the build below threshold
      # posts pass/fail as a PR comment

Wrap

Path A: after this session.

That is TRACE. Trace, Read, Analyze, Codify, Enforce. · Assignment guide · Harmony pack · Agent transcripts · Hamel · Anthropic agent evals · Braintrust · Promptfoo