emerging-techniques-knowledge-distillation

Topics in the source

• Knowledge Distillation: Compressing LLMs
• When to Use This Skill
• Installation
• Quick Start
• Basic Knowledge Distillation
• MiniLLM (Reverse KLD)
• Response Distillation
• Core Concepts

Use Knowledge Distillation when you need to: Compress models from 70B → 7B while retaining 90%+ performance Transfer capabilities from proprietary models (GPT-4) to open-source (LLaMA, Mistral) Reduce inference costs by deploying smaller student models Create specialized models by distilling domain-specific knowledge Improve small models using synthetic data from large teachers Key Techniques: Temperature scaling, soft targets, reverse KLD (MiniLLM), logit distillation, response distillation Papers: Hinton et al. 2015 (arXiv 1503.02531), MiniLLM (arXiv 2306.08543), KD Survey (arXiv 2402.13116) Source: arXiv 2306.08543 (2024) Innovation: Use reverse KLD instead of forward KLD for better generative model distillation. Why reverse KL? Forward KL (standard): Student learns to match teacher's mean Reverse KL (MiniLLM): Student learns to cover all teacher's modes Better for diverse text generation Purpose: Soften probability distributions to expose teacher's uncertainty. Rule: Use T=2-5 for distillation (2 is common default). Hinton et al. 2015 (Foundational): https://arxiv.org/abs/1503.02531 MiniLLM (Reverse KLD): https://arxiv.org/abs/2306.08543 KD Survey for LLMs (2024): https://arxiv

This is an excerpt. Refer to the complete source and its current requirements.