blind-evaluation

Topics in the source

• Blind Evaluation
• When this fires
• The protocol
• Confounds worth expecting
• Reading the result honestly
• Rationalizations that mean stop
• Worked example: a retracted floor-pass
• Boundaries with neighbouring skills

A comparison is only worth what its design is worth. If you look at two outputs, notice that one is yours, and pick it, you have measured your own expectations. The protocol below exists because that failure is invisible from the inside — the reasoning feels sound either way, so the only defense is arranging things so your preference cannot reach the verdict. The expensive failures are rarely "I was biased." They are structural: a control that never could have won, an arm that always sat in the same slot, a defect in one arm that you scored as if both were sound. Any moment you are about to say this one is better on the strength of a comparison — or about to build the comparison that will let you say it. Two configs, two prompts, two libraries, old pipeline vs new, model A vs model B, with-the-thing vs without. It also fires backwards: when a comparison already exists and you are asked whether the result holds, run its design through this and report what the design can and cannot support. 1. Fix the question and the arms before you build anything. Write down what differs between the arms and confirm it is exactly one thing. Both arms get an identical brief — same goal, same audienc

This is an excerpt. Refer to the complete source and its current requirements.