optimization-awq

Topics in the source

• AWQ (Activation-aware Weight Quantization)
• When to use AWQ
• Quick start
• Installation
• Load pre-quantized model
• Quantize your own model
• AWQ vs GPTQ vs bitsandbytes
• Kernel backends

4-bit quantization that preserves salient weights based on activation patterns, achieving 3x speedup with minimal accuracy loss. Use AWQ when: Need 4-bit quantization with <5% accuracy loss Deploying instruction-tuned or chat models (AWQ generalizes better) Want 2.5-3x inference speedup over FP16 Using vLLM for production serving Have Ampere+ GPUs (A100, H100, RTX 40xx) for Marlin kernel support Use GPTQ instead when: Need maximum ecosystem compatibility (more tools support GPTQ) Working with ExLlamaV2 backend specifically Have older GPUs without Marlin support Use bitsandbytes instead when: Need zero calibration overhead (quantize on-the-fly) Want to fine-tune with QLoRA Prefer simpler integration Requirements: Python 3.8+, CUDA 11.8+, Compute Capability 7.5+ Timing: 10-15 min for 7B, 1 hour for 70B models. | Feature | AWQ | GPTQ | bitsandbytes | |---------|-----|------|--------------| | Speedup (4-bit) | 2.5-3x | 2x | 1.5x | | Accuracy loss | <5% | 5-10% | 5-15% | | Calibration | Minimal (128-1K tokens) | More extensive | None | | Overfitting risk | Low | Higher | N/A | | Best for | Production inference | GPU inference | Easy integration | | vLLM support | Native | Yes | Limited

This is an excerpt. Refer to the complete source and its current requirements.