optimization-gptq

Topics in the source

• GPTQ (Generative Pre-trained Transformer Quantization)
• When to use GPTQ
• Quick start
• Installation
• Load pre-quantized model
• Quantize your own model
• Group-wise quantization
• Quantization configurations

Post-training quantization method that compresses LLMs to 4-bit with minimal accuracy loss using group-wise quantization. Use GPTQ when: Need to fit large models (70B+) on limited GPU memory Want 4× memory reduction with <2% accuracy loss Deploying on consumer GPUs (RTX 4090, 3090) Need faster inference (3-4× speedup vs FP16) Use AWQ instead when: Need slightly better accuracy (<1% loss) Have newer GPUs (Ampere, Ada) Want Marlin kernel support (2× faster on some GPUs) Use bitsandbytes instead when: Need simple integration with transformers Want 8-bit quantization (less compression, better quality) Don't need pre-quantized model files How GPTQ works: 1. Group weights: Divide each weight matrix into groups (typically 128 elements) 2. Quantize per-group: Each group has its own scale/zero-point 3. Minimize error: Uses Hessian information to minimize quantization error 4. Result: 4-bit weights with near-FP16 accuracy Group size trade-off: | Group Size | Model Size | Accuracy | Speed | Recommendation | |------------|------------|----------|-------|----------------| | -1 (per-column) | Smallest | Best | Slowest | Research only | | 32 | Smaller | Better | Slower | High accuracy needed | |

This is an excerpt. Refer to the complete source and its current requirements.