evaluation-bigcode-evaluation-harness

Topics in the source

• BigCode Evaluation Harness - Code Model Benchmarking
• Quick Start
• Common Workflows
• Workflow 1: Standard Code Benchmark Evaluation
• Workflow 2: Multi-Language Evaluation (MultiPL-E)
• Workflow 3: Instruction-Tuned Model Evaluation
• Workflow 4: Compare Multiple Models
• When to Use vs Alternatives

BigCode Evaluation Harness evaluates code generation models across 15+ benchmarks including HumanEval, MBPP, and MultiPL-E (18 languages). Installation: Evaluate on HumanEval: View available tasks: Evaluate model on core code benchmarks (HumanEval, MBPP, HumanEval+). Checklist: Step 1: Choose benchmark suite Python code generation (most common): HumanEval: 164 handwritten problems, function completion HumanEval+: Same 164 problems with 80× more tests (stricter) MBPP: 500 crowd-sourced problems, entry-level difficulty MBPP+: 399 curated problems with 35× more tests Multi-language (18 languages): MultiPL-E: HumanEval/MBPP translated to C++, Java, JavaScript, Go, Rust, etc. Advanced: APPS: 10,000 problems (introductory/interview/competition) DS-1000: 1,000 data science problems across 7 libraries Step 2: Configure model and generation Step 3: Run evaluation Step 4: Analyze results Results in results/starcoder2-humaneval.json: Evaluate code generation across 18 programming languages. Checklist: Step 1: Generate solutions on host Step 2: Evaluate in Docker container Supported languages: Python, JavaScript, Java, C++, Go, Rust, TypeScript, C#, PHP, Ruby, Swift, Kotlin, Scala, Perl, Julia

This is an excerpt. Refer to the complete source and its current requirements.