optimization-bitsandbytes

Topics in the source

• bitsandbytes - LLM Quantization
• Quick start
• Common workflows
• Workflow 1: Load large model in limited GPU memory
• Workflow 2: Fine-tune with QLoRA (4-bit training)
• Workflow 3: 8-bit optimizer for memory-efficient training
• When to use vs alternatives
• Common issues

bitsandbytes reduces LLM memory by 50% (8-bit) or 75% (4-bit) with <1% accuracy loss. Installation: 8-bit quantization (50% memory reduction): 4-bit quantization (75% memory reduction): Copy this checklist: Step 1: Calculate memory requirements Estimate model memory: Step 2: Choose quantization level | GPU VRAM | Model Size | Recommended | |----------|------------|-------------| | 8 GB | 3B | 4-bit | | 12 GB | 7B | 4-bit | | 16 GB | 7B | 8-bit or 4-bit | | 24 GB | 13B | 8-bit or 70B 4-bit | | 40+ GB | 70B | 8-bit | Step 3: Configure quantization For 8-bit (better accuracy): For 4-bit (maximum memory savings): Step 4: Load and verify model QLoRA enables fine-tuning large models on consumer GPUs. Copy this checklist: Step 1: Install dependencies Step 2: Configure 4-bit base model Step 3: Add LoRA adapters Step 4: Train with standard Trainer Use 8-bit Adam/AdamW to reduce optimizer memory by 75%. Step 1: Replace standard optimizer Manual optimizer usage: Step 2: Configure training Compare memory: Step 3: Monitor memory savings Use bitsandbytes when: GPU memory limited (need to fit larger model) Training with QLoRA (fine-tune 70B on single GPU) Inference only (50-75% memory reduction)

This is an excerpt. Refer to the complete source and its current requirements.