Codify introduced the idea: validate a judge with TPR and TNR before trusting it. This deep dive is the actual workflow, how you get from "I wrote a judge prompt" to "I have evidence this judge agrees with a human on the calls that matter."

Step 1 - build a labeled set from real failures, not invented examples

Pull 30 to 60 traces from your Trace/Read pass, weighted toward the failure categories from Analyze. Invented "looks about right" examples do not exercise the judge on your system's actual failure distribution, and a judge tuned on easy invented cases will look great and catch nothing in production.

Step 2 - label blind, with a rubric, before you write the judge prompt

• Write the rubric as a yes/no question a human can answer in under 10 seconds ("is every claim in this reply supported by the retrieved context, yes or no").
• Label without seeing what the judge would say - labeling after you have a judge prompt biases you toward agreeing with it.
• If two people can label the same set, measure inter-rater agreement first. A rubric humans cannot agree on cannot be automated reliably.

Step 3 - split before you tune

Hold out roughly a third of your labeled set and do not look at it while iterating the judge prompt. Tune on the rest. Report TPR/TNR only on the held-out split. This is the single biggest thing separating a validated judge from a judge that memorised its own test set.

Step 4 - read the confusion matrix, not just an aggregate score

Step 5 - iterate the prompt, not the rubric

• Force reasoning before the verdict ("reason, then pass/fail") - judges that answer first and justify after are measurably less reliable.
• Give the judge the same context the app had (retrieved chunks, tool results), not just the final text - a judge without the source material is guessing.
• One failure mode per judge. A single judge scoring "is this good" conflates tone, grounding, and formatting into one number you cannot act on.
• Prefer a different model family for the judge than for the app under test, to reduce correlated blind spots.

Step 6 - re-validate when anything upstream changes

A judge validated against GPT-4o is not automatically validated against Gemini 3 or a new prompt version. Model updates shift judge behaviour the same way they shift app behaviour. Re-run the held-out set whenever you change the judge model, the app model, or the prompt, and treat judge drift as a monitored metric, not a one-time checkbox.