agent-regression-evals

Topics in the source

• Agent Regression Evals — prove the change did not make it worse
• Workflow
• The gate blocks on
• Hard rules
• Scripts
• References and assets

Portability: 3 stdlib scripts, no APIs, no LLM calls. You run the agent; these grade, count and gate. "Pass rate went from 82% to 79%" is not a finding until you know whether three points is outside the noise. Agents are non-deterministic, so one trial per case cannot tell a regression from a coin flip, and a final answer can be right while the path to it called a tool it never should have. Step 1 uses match modes strict | unordered | subset | superset. Default to superset (required steps present, harmless exploration allowed); grade the outcome, not the path, unless order is the requirement. forbidden tools block in every mode. Step 2 reports pass@k (succeeds at least once in k) and pass^k (succeeds all k times). A user meets every trial, so pass^k is the reliability number. Cases marked critical are never quarantine candidates. Step 3 averages trials within each case first (trials of one case are correlated), then tests paired per-case differences with a t-based 95% CI and a one-sided exact McNemar test. When the drop is real but the sample is too small to show it, the gate says how many cases it would take. 1. A statistically significant drop (CI upper bound below zero, or McNem

This is an excerpt. Refer to the complete source and its current requirements.