evaluation-lm-evaluation-harness

Topics in the source

• lm-evaluation-harness - LLM Benchmarking
• Quick start
• Common workflows
• Workflow 1: Standard benchmark evaluation
• Workflow 2: Track training progress
• Workflow 3: Compare multiple models
• Workflow 4: Evaluate with vLLM (faster inference)
• When to use vs alternatives

lm-evaluation-harness evaluates LLMs across 60+ academic benchmarks using standardized prompts and metrics. Installation: Evaluate any HuggingFace model: View available tasks: Evaluate model on core benchmarks (MMLU, GSM8K, HumanEval). Copy this checklist: Step 1: Choose benchmark suite Core reasoning benchmarks: MMLU (Massive Multitask Language Understanding) - 57 subjects, multiple choice GSM8K - Grade school math word problems HellaSwag - Common sense reasoning TruthfulQA - Truthfulness and factuality ARC (AI2 Reasoning Challenge) - Science questions Code benchmarks: HumanEval - Python code generation (164 problems) MBPP (Mostly Basic Python Problems) - Python coding Standard suite (recommended for model releases): Step 2: Configure model HuggingFace model: Quantized model (4-bit/8-bit): Custom checkpoint: Step 3: Run evaluation Step 4: Analyze results Results saved to results/llama2-7b-eval.json: Evaluate checkpoints during training. Step 1: Set up periodic evaluation Evaluate every N training steps: Step 2: Choose quick benchmarks Fast benchmarks for frequent evaluation: HellaSwag: 10 minutes on 1 GPU GSM8K: 5 minutes PIQA: 2 minutes Avoid for frequent eval (too slow): MMLU: 2

This is an excerpt. Refer to the complete source and its current requirements.