Deep diveSelf-paced

How to brief an engineer to build the evals you need

Handoffs fail when people ask for "quality metrics" and get generic coherence scores that ignore the job. Bring TRACE raw material instead: traces, ranked failures, must-pass checklist.

Ask for: (1) binary checks tied to those failure modes; (2) if using an AI judge, validation with TPR and TNR against your labels - not raw agreement; (3) a view where a specific failure's rate goes down after a fix.

Tip

Language that helps eng

Use Anthropic's vocabulary when useful: task, trial, grader, transcript vs outcome. Ask for offline golden regression plus (later) online sampling that feeds new cases back into the suite.