Deep diveSelf-paced
How to brief an engineer to build the evals you need
Handoffs fail when people ask for "quality metrics" and get generic coherence scores that ignore the job. Bring TRACE raw material instead: traces, ranked failures, must-pass checklist.
Ask for: (1) binary checks tied to those failure modes; (2) if using an AI judge, validation with TPR and TNR against your labels - not raw agreement; (3) a view where a specific failure's rate goes down after a fix.
Tip
Language that helps eng
Use Anthropic's vocabulary when useful: task, trial, grader, transcript vs outcome. Ask for offline golden regression plus (later) online sampling that feeds new cases back into the suite.