eval-harness
Eval-driven development (EDD) framework for AI coding sessions — define capability and regression evals before coding, grade with code-based, model-based, rule, or human graders, and track pass@k and pass^k reliability. Use when defining pass/fail criteria for agent tasks, measuring agent reliability, building regression suites for prompt or agent changes, or benchmarking across model versions.
Target: Software development & testing. Automatically collected · unverified.
Excerpt from the author’s documentation
Extracted from a fixed version of SKILL.md in its original language. This is the author’s documentation; installation, compatibility and runtime results have not been verified by this site.
Read the original English text
Topics in the source
- Eval Harness Skill
- When to Activate
- Philosophy
- Eval Types
- Capability Evals
- Regression Evals
- Grader Types
- 1. Code-Based Grader
A formal evaluation framework for Claude Code sessions, implementing eval-driven development (EDD) principles. Setting up eval-driven development (EDD) for AI-assisted workflows Defining pass/fail criteria for Claude Code task completion Measuring agent reliability with pass@k metrics Creating regression test suites for prompt or agent changes Benchmarking agent performance across model versions Eval-Driven Development treats evals as the "unit tests of AI development": Define expected behavior BEFORE implementation Run evals continuously during development Track regressions with each change Use pass@k metrics for reliability measurement Test if Claude can do something it couldn't before: Ensure changes don't break existing functionality: Deterministic checks using code: Use Claude to evaluate open-ended outputs: Flag for manual review: "At least one success in k attempts" pass@1: First attempt success rate pass@3: Success within 3 attempts Typical target: pass@3 > 90% "All k trials succeed" Higher bar for reliability pass^3: 3 consecutive successes Use for critical paths Write code to pass the defined evals. Creates eval definition file at .claude/evals/feature-name.md Runs curren
This is an excerpt. Refer to the complete source and its current requirements.
- Source repository
- affaan-m/ECC
- Source file
- skills/eval-harness/SKILL.md
- Fixed revision
- d2ef2680594f9a17526a6e1a2e18b3d242c5d17b
Sources and evidence
This record was collected automatically. Its documentation, suitability and runtime behavior have not been manually verified.
Author: affaan-m/ECC · original repository metadata. Documentation review: Not reviewed. Last source check: 2026-11-09T15:01:43.733Z.
Categories are inferred from names, descriptions and paths. They do not establish compatibility.
License and attribution
MIT. Third-party content and copyright remain with their authors. Refer to the original license and terms.
Original license ↗Synchronization, copying and user-reported success do not mark a resource as runtime-tested.
Limitations and notes
- Collecting metadata does not mean this site has installed, adapted, runtime-tested or security-reviewed the resource.