optimization-gguf

Topics in the source

• GGUF - Quantization Format for llama.cpp
• When to use GGUF
• Quick start
• Installation
• Convert model to GGUF
• Quantize model
• Run inference
• Quantization types

The GGUF (GPT-Generated Unified Format) is the standard file format for llama.cpp, enabling efficient inference on CPUs, Apple Silicon, and GPUs with flexible quantization options. Use GGUF when: Deploying on consumer hardware (laptops, desktops) Running on Apple Silicon (M1/M2/M3) with Metal acceleration Need CPU inference without GPU requirements Want flexible quantization (Q2K to Q80) Using local AI tools (LM Studio, Ollama, text-generation-webui) Key advantages: Universal hardware: CPU, Apple Silicon, NVIDIA, AMD support No Python runtime: Pure C/C++ inference Flexible quantization: 2-8 bit with various methods (K-quants) Ecosystem support: LM Studio, Ollama, koboldcpp, and more imatrix: Importance matrix for better low-bit quality Use alternatives instead: AWQ/GPTQ: Maximum accuracy with calibration on NVIDIA GPUs HQQ: Fast calibration-free quantization for HuggingFace bitsandbytes: Simple integration with transformers library TensorRT-LLM: Production NVIDIA deployment with maximum speed | Type | Bits | Size (7B) | Quality | Use Case | |------|------|-----------|---------|----------| | Q2K | 2.5 | 2.8 GB | Low | Extreme compression | | Q3KS | 3.0 | 3.0 GB | Low-Med | Memory co

This is an excerpt. Refer to the complete source and its current requirements.