eval-genius

Topics in the source

• Eval Genius
• Step 0: Does this need an eval, and where does it go?
• Route the request
• Step 1: Foundation (five minutes, never skipped)
• Step 2: Choose the grader (deterministic first)
• Step 3: Build or adopt
• Step 4: Run under hard rules (a run that breaks one is not a result)
• Step 5: Read the result (in this order)

An eval is a claim you are willing to defend under hostile audit. You measure to earn the right to say "this is better" and have it hold when someone sharp pushes back. Behave like a measurement engineer: state the promise, fix the bar before looking, hold everything else constant, distrust the instrument first, report the number that hurts. Assume the user may be starting from zero; plain language first, jargon when it earns it. -- Three questions decide it (references/00-start-here.md): does the output vary (model, prompt, retriever)? will it change again, and would a quiet regression cost something? is a decision or a public claim coming? No to all: a hand spot check, stop. Yes to any: an eval, sized to the stage the project is in. | Stage the user is at | Instrument | Smallest useful version | |---|---|---| | Exploring prompts and models | Spot check | 10 inputs, eyeball | | First working version | Smoke eval | 20 to 50 real inputs, code-checked; this run is the baseline | | Changing one thing | Paired eval vs baseline | Same items both arms, per-item diff, bar written first | | Merging or shipping | CI gate | Held-out items, three-way outcome, a known-bad item that must fail |

This is an excerpt. Refer to the complete source and its current requirements.