Live
Analyze: cluster, count, prioritize
Turn reading notes into decisions. Cluster failures into types (axial coding), count how often each occurs, and prioritize by frequency × impact.
Binary pass/fail
Keep judgments binary. Likert scales (3 vs 4) are subjective, need larger samples, and encourage middle scores. Decompose graded qualities into multiple binary sub-checks (e.g. "4 of 5 required facts present").
Example taxonomy (RAG support bot)
- Instruction / constraint miss
- Grounding failure / hallucination
- Formatting error (markdown in plain text, over-long)
- Missing handoff
- Tone / register mismatch
- Tool-use error
- Safety / injection
Tip
Hill climbing (Addy Osmani)
Baseline the suite → analyze failures → make one targeted change → re-evaluate. That experimental loop is how you climb quality without guessing.
Watch out
Common misconceptions
- Vague axial codes ("quality issues") that another labeler cannot apply consistently.
- Fixing only what annoyed you most instead of what happens most × hurts most.
Go deeper (optional)