Enforce: evals in the loop
An eval suite you run once is worthless; one that runs on every change is a safety net. Enforce wires checks into the development loop so you can see a specific failure rate go up or down after a change.
Offline vs online
Offline
RegressionGolden dataset from real failures; run before prompt/model merges. Pass-rate thresholds, not "must be 100%".
Online
SamplingSample production traffic, score continuously, route low-score traces to human review, and feed new cases back into the golden set.
CI pattern (explain-only in class)
Promptfoo (or your harness) on PRs that touch prompts/**: post pass/fail, gate below a threshold. Set temperature 0 where possible; use repeat/trials for flakiness (Anthropic: multiple trials per task). Track token cost of model-graded checks.
name: LLM Evals
on:
pull_request:
paths:
- 'prompts/**'
- 'promptfooconfig.yaml'
jobs:
eval:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: promptfoo/promptfoo-action@v1
with:
openai-api-key: ${{ secrets.OPENAI_API_KEY }}
config: promptfooconfig.yamlWatch: Mastering AI Evaluation - Playground to Production
Watch out
Common misconceptions
- Optimizing only the fixed eval set until you overfit (high score, worse product).
- Trusting an unvalidated LLM judge in CI.
- Demanding 100% pass on flaky model-graded checks with no temperature/repeat policy.