huggingface-community-evals

Topics in the source

• Overview
• When To Use Which Script
• Prerequisites
• Core Workflow
• Quick Start
• Option A: inspect-ai with local inference providers path
• Option B: inspect-ai on Local GPU
• Option C: lighteval on Local GPU

This skill is for running evaluations against models on the Hugging Face Hub on local hardware. It covers: inspect-ai with local inference lighteval with local inference choosing between vllm, Hugging Face Transformers, and accelerate smoke tests, task selection, and backend fallback strategy It does not cover: Hugging Face Jobs orchestration model-card or model-index edits README table extraction Artificial Analysis imports .evalresults generation or publishing PR creation or community-evals automation If the user wants to run the same eval remotely on Hugging Face Jobs, hand off to the hugging-face-jobs skill and pass it one of the local scripts in this skill. If the user wants to publish results into the community evals workflow, stop after generating the evaluation run and hand off that publishing step to /code/community-evals. All paths below are relative to the directory containing this SKILL.md. | Use case | Script | |---|---| | Local inspect-ai eval on a Hub model via inference providers | scripts/inspectevaluv.py | | Local GPU eval with inspect-ai using vllm or Transformers | scripts/inspectvllmuv.py | | Local GPU eval with lighteval using vllm or accelerate | scripts/ligh

This is an excerpt. Refer to the complete source and its current requirements.