Assignment
Build and run TRACE on your capstone
Trace and read real outputs, analyze and rank failures, codify a suite of code-based checks (and optionally a validated LLM-as-judge), and wire it to run on demand. Include a Streamlit UI (or extend your existing one) so you can trigger the suite and see scores without living in the CLI.
Path A (enough to pass)
Taxonomy from real traces, ≥2 code assertions, one fix with a metric that moves, Streamlit shows results.
Path B (stretch)
Validated judge with TPR/TNR, golden dataset / Promptfoo sketch, or production-trace → dataset loop. Details in the support guide.
Done when
- You open-coded the Harmony sample pack (20 traces) and/or 15-20 of your own runs
- You have a failure taxonomy from those traces (≥4 specific categories)
- You have an eval suite with ≥2 code-based checks tied to those failures
- You can show a metric moving after a fix
- Streamlit UI can trigger or display eval results (screenshot for Maven)
Submit in Maven: Top failure, the checks you added, and a before/after screenshot from Streamlit (plus any Path B stretch).
Go deeper (optional)