optimization-hqq

Topics in the source

• HQQ - Half-Quadratic Quantization
• When to use HQQ
• Quick start
• Installation
• Basic quantization
• Quantize full model with HuggingFace
• Core concepts
• Quantization configuration

Fast, calibration-free weight quantization supporting 8/4/3/2/1-bit precision with multiple optimized backends. Use HQQ when: Quantizing models without calibration data (no dataset needed) Need fast quantization (minutes vs hours for GPTQ/AWQ) Deploying with vLLM or HuggingFace Transformers Fine-tuning quantized models with LoRA/PEFT Experimenting with extreme quantization (2-bit, 1-bit) Key advantages: No calibration: Quantize any model instantly without sample data Multiple backends: PyTorch, ATEN, TorchAO, Marlin, BitBlas for optimized inference Flexible precision: 8/4/3/2/1-bit with configurable group sizes Framework integration: Native HuggingFace and vLLM support PEFT compatible: Fine-tune quantized models with LoRA Use alternatives instead: AWQ: Need calibration-based accuracy, production serving GPTQ: Maximum accuracy with calibration data available bitsandbytes: Simple 8-bit/4-bit without custom backends llama.cpp/GGUF: CPU inference, Apple Silicon deployment HQQ uses BaseQuantizeConfig to define quantization parameters: The core quantized layer that replaces nn.Linear: HQQ supports multiple inference backends for different hardware: Backend selection guide: | Backend | Be

This is an excerpt. Refer to the complete source and its current requirements.