eval-skills

Topics in the source

• Eval Skills
• Inputs to settle before running
• Workflow
• Output
• Rules

Treat a skill like a function under test. Feed it example inputs in a clean room, check the artifacts against what good looks like, and let the failures drive the edits. The eval is only honest if the run is blind: the agent executing the skill must carry none of this conversation's context and must never see the expected output. Leak either and you are teaching to the test. Confirm all three before spawning anything. If any is missing, do not run yet: tell the user exactly which one is missing and what a good version looks like, help them make it concrete, and echo it back. Do not invent cases, guess intent, or eval against a fuzzy wish. Target skill — must resolve to a real SKILL.md. If you can't find it, list the skills you can see and ask which one they mean. At least one golden case — a concrete input the skill will actually receive: a screenshot, a prompt, a file, a scene. "Improve write-spec" with no input attached is not a case. The bar per case — the outcome a good artifact achieves and the smells that would make it bad, not an exhaustive parts list. The skill's judgment is what's under test, so do not pre-enumerate every requirement — that turns the eval into a conformanc

This is an excerpt. Refer to the complete source and its current requirements.