The tension
The models were live. The definition of good was still subjective.
MPB already had AI in the customer experience, including Intercom Fin support, natural-language search and personalised recommendations. The harder question was how a mixed team could agree whether an answer was accurate, safe, complete and right for the moment.
Manual review would not scale, and output variance was eroding confidence. A hallucinated return window or the wrong tone on a sensitive query could look fluent and still fail the customer.
The intervention
Build the evaluation in the room, on cases the team recognised.
Every piece of theory was followed by doing. Sixteen participants worked with five real Fin scenarios, mapped seven failure categories across MPB's stack, then built a query classifier, a string-check grader and a model scorer using a 1–7 rubric for urgency, accuracy, helpfulness and safety.
They ran the same dataset and graders against a vague prompt and a more explicit one. The score moved in front of the room, making the relationship between specification and quality visible rather than rhetorical.
The change
A quality contract the whole team could use on Monday.
The group turned the work into an eval contract for the PRD: a gold set, code checks, a model rubric, safety graders and explicit ship thresholds. The shared dashboard view tracked model score, string-check pass rate and safety failures, with zero as the safety target.
The next move was deliberately small: choose the riskiest feature, export 20 real conversations, write the contract, establish a baseline by Friday and review quality every week.
