agent-eval

Topics in the source

• Agent Eval Skill
• When to Activate
• Installation
• Core Concepts
• YAML Task Definitions
• Git Worktree Isolation
• Metrics Collected
• Workflow

A lightweight CLI tool for comparing coding agents head-to-head on reproducible tasks. Every "which coding agent is best?" comparison runs on vibes — this tool systematizes it. Comparing coding agents (Claude Code, Aider, Codex, etc.) on your own codebase Measuring agent performance before adopting a new tool or model Running regression checks when an agent updates its model or tooling Producing data-backed agent selection decisions for a team Note: Install agent-eval from its repository after reviewing the source. Define tasks declaratively. Each task specifies what to do, which files to touch, and how to judge success: Each agent run gets its own git worktree — no Docker required. This provides reproducibility isolation so agents cannot interfere with each other or corrupt the base repo. | Metric | What It Measures | |--------|-----------------| | Pass rate | Did the agent produce code that passes the judge? | | Cost | API spend per task (when available) | | Time | Wall-clock seconds to completion | | Consistency | Pass rate across repeated runs (e.g., 3/3 = 100%) | Create a tasks/ directory with YAML files, one per task: Execute agents against your tasks: Each run: 1. Creates a

This is an excerpt. Refer to the complete source and its current requirements.