evaluation-independence

Topics in the source

• Evaluation Independence
• When this fires
• The protocol
• Failure modes
• Independence has axes, and they are not interchangeable
• Repeats are not a measurement, for a deterministic component
• Rationalizations that mean stop
• Worked example: a project-long build/judge firewall

Left alone, every builder quietly recruits the judge. You reach for the best examples you have, use them to shape the thing, and then reach for those same examples to prove the thing works — and it does, because you built it to. The grade is real; it just measures nothing. This is train/test leakage, self-review, and benchmark contamination, and all three share one shape: the knowledge that produced the artifact also scored it, so a pass only confirms the artifact agrees with itself. The failure is invisible from inside because each step felt reasonable. The only defense is a standing boundary — declared in writing, enforced by the filesystem — between what may build and what may judge, held for the life of the project. Any time the same project both builds something and measures whether it is good, and the inputs to those two activities could touch. Assembling design principles and a reference set together. Curating training data next to a benchmark. Letting your own judgment tune a rubric that then grades outputs. Pointing a model at its own generations as ground truth. Reporting "it matched the exemplars N/N" when you chose the exemplars. It fires forward while you are still dec

This is an excerpt. Refer to the complete source and its current requirements.