fine-tuning-peft

Topics in the source

• PEFT (Parameter-Efficient Fine-Tuning)
• When to use PEFT
• Quick start
• Installation
• LoRA fine-tuning (standard)
• QLoRA fine-tuning (memory-efficient)
• LoRA parameter selection
• Rank (r) - capacity vs efficiency

Fine-tune LLMs by training <1% of parameters using LoRA, QLoRA, and 25+ adapter methods. Use PEFT/LoRA when: Fine-tuning 7B-70B models on consumer GPUs (RTX 4090, A100) Need to train <1% parameters (6MB adapters vs 14GB full model) Want fast iteration with multiple task-specific adapters Deploying multiple fine-tuned variants from one base model Use QLoRA (PEFT + quantization) when: Fine-tuning 70B models on single 24GB GPU Memory is the primary constraint Can accept 5% quality trade-off vs full fine-tuning Use full fine-tuning instead when: Training small models (<1B parameters) Need maximum quality and have compute budget Significant domain shift requires updating all weights | Rank | Trainable Params | Memory | Quality | Use Case | |------|-----------------|--------|---------|----------| | 4 | 3M | Minimal | Lower | Simple tasks, prototyping | | 8 | 7M | Low | Good | Recommended starting point | | 16 | 14M | Medium | Better | General fine-tuning | | 32 | 27M | Higher | High | Complex tasks | | 64 | 54M | High | Highest | Domain adaptation, 70B models | | Method | Trainable % | Memory | Speed | Best For | |--------|------------|--------|-------|----------| | LoRA | 0.1-1% | Low |

This is an excerpt. Refer to the complete source and its current requirements.