evals

Topics in the source

• Evals: prove an instruction works
• 1. Frame
• 2. Pick the cases
• 3. Blind the agent
• 4. Grade what it did, not what it says
• 5. Measure the difference
• 6. Keep it safe
• In Claude Code

A test proves code. An instruction (a skill, a rule in AGENTS.md, a prompt) is only a claim until you have watched the agent behave differently because of it. An eval is that proof: the same request, run with the instruction and without it, graded on what came out. Name the change under test and what a user would notice if it worked. Write 3 to 6 criteria a stranger could check, outcome first: the file that exists, the number in the answer, the state the screen shows. Keep the criteria away from the agent being measured. At least four requests where the instruction should change the result, in at least two different shapes. Real requests from your history beat invented ones. At least one where it should do nothing: a plain question it must not load, pitch or slow down. The prompt reads like a real request. No "eval", "test", "judge", "rubric", "benchmark" or "score" in the prompt, folder names or file names. Never ask the agent which skills or rules it followed. It will say what you want to hear. Plant only what a real task would have: a small project, the bug, the files a person would mention. In this order, using the first that can catch the failure: 1. Machine checks: a pattern

This is an excerpt. Refer to the complete source and its current requirements.