What good evals tooling looks like (so you can lead it)
TRACE's last two steps, Codify and Enforce, are where engineers go deep. Your job is to recognise good work and hold a team to it.
Codify turns findings into repeatable checks - rules for structural failures, LLM-as-judge for semantic ones. Enforce wires those checks so every change is measured and you can see a failure rate go up or down.
Classroom tooling: Braintrust (primary - human labels and scorers together), Langfuse (open-source backup), Promptfoo (CI gating - explain-only for you).
Critical
The three leadership questions
Are these checks tied to our real failures? Are they binary? If an AI judges outputs, have we validated it with true positive rate and true negative rate against human labels - not raw agreement?
An unvalidated AI judge is another vibe check. Agreement is a trap when failures are rare: an always-pass judge looks accurate and catches nothing.
Watch: Mastering AI Evaluation - Playground to Production