Codify turns prioritized failures into automated checks. Anthropic's framing: combine code-based, model-based (LLM-as-judge), and human graders. Reach for code first; add a validated judge only for judgment calls.

Code-based assertions

• Schema / is-json / required fields
• No-markdown for SMS channels
• Length or "max N units listed" budgets
• URL allowlist; PII/secret guards
• Latency and token ceilings

LLM-as-judge (one failure mode per judge)

Best for semantic failures: grounding/faithfulness, constraint adherence, tone, unsupported claims, handoff judgment. Force reasoning before the verdict; keep the label binary; prefer a different model family for the judge than for the app.

You are a strict evaluator. Judge only whether the assistant reply is GROUNDED in the provided knowledge base and tool outputs. If any factual claim is unsupported, contradicted, or invented, it is not grounded.
Answer with JSON: {"reason": "one sentence citing the specific claim", "pass": true or false}. Output the reason first, then the verdict.

Validate the judge - TPR and TNR

An unvalidated judge is another vibe check. Label a human set from error analysis. Measure true positive rate (catch real failures) and true negative rate (correctly pass clean cases). Do not use raw agreement - under class imbalance an always-pass judge looks great and catches nothing.