hf-mem

Topics in the source

• When to use?
• What are the requirements?
• How to run?
• Examples

hfmem estimates the required memory for inference, including model weights and an optional KV cache, for Safetensors and GGUF for models on the Hugging Face Hub using HTTP Range requests i.e., without downloading or loading any weights locally. User asks how much VRAM or memory a model needs to run User wants to know if a model fits on their GPU or a given instance User references a Hugging Face model ID or URL and asks about inference requirements uv installed (for uvx) HFTOKEN env var or --hf-token flag (for gated or private models only) Run with --model-id pointing to the Hugging Face Hub repository which will check that it either contains Safetensors (via model.safetensors, model.safetensors.index.json if sharded, or modelindex.json for Diffusers) or GGUF model weights within. If the repository contains GGUF model weights in multiple precisions / quantizations, the estimations will be on a per-file basis, whereas for inference you won't load all of those but rather only a single precision. This being said, for GGUF you might as well need to provide --gguf-file to target the specific file (or path if sharded) you want to run. Additionally, hf-mem comes with an --experimental fla

This is an excerpt. Refer to the complete source and its current requirements.