eval-harness

Topics in the source

• Eval Harness Skill
• When to Activate
• Philosophy
• Eval Types
• Capability Evals
• Regression Evals
• Grader Types
• 1. Code-Based Grader

A formal evaluation framework for Claude Code sessions, implementing eval-driven development (EDD) principles. Setting up eval-driven development (EDD) for AI-assisted workflows Defining pass/fail criteria for Claude Code task completion Measuring agent reliability with pass@k metrics Creating regression test suites for prompt or agent changes Benchmarking agent performance across model versions Eval-Driven Development treats evals as the "unit tests of AI development": Define expected behavior BEFORE implementation Run evals continuously during development Track regressions with each change Use pass@k metrics for reliability measurement Test if Claude can do something it couldn't before: Ensure changes don't break existing functionality: Deterministic checks using code: Use Claude to evaluate open-ended outputs: Flag for manual review: "At least one success in k attempts" pass@1: First attempt success rate pass@3: Success within 3 attempts Typical target: pass@3 > 90% "All k trials succeed" Higher bar for reliability pass^3: 3 consecutive successes Use for critical paths Write code to pass the defined evals. Creates eval definition file at .claude/evals/feature-name.md Runs curren

This is an excerpt. Refer to the complete source and its current requirements.