emerging-techniques-speculative-decoding

Topics in the source

• Speculative Decoding: Accelerating LLM Inference
• When to Use This Skill
• Installation
• Quick Start
• Basic Speculative Decoding (Draft Model)
• Medusa (Multiple Decoding Heads)
• Lookahead Decoding (Jacobi Iteration)
• Core Concepts

Use Speculative Decoding when you need to: Speed up inference by 1.5-3.6× without quality loss Reduce latency for real-time applications (chatbots, code generation) Optimize throughput for high-volume serving Deploy efficiently on limited hardware Generate faster without changing model architecture Key Techniques: Draft model speculative decoding, Medusa (multiple heads), Lookahead Decoding (Jacobi iteration) Papers: Medusa (arXiv 2401.10774), Lookahead Decoding (ICML 2024), Speculative Decoding Survey (ACL 2024) Idea: Use small draft model to generate candidates, large target model to verify in parallel. Algorithm: 1. Draft model generates K tokens speculatively 2. Target model evaluates all K tokens in parallel (single forward pass) 3. Accept tokens where draft and target agree 4. Reject first disagreement, continue from there Performance: Speedup: 1.5-2× with good draft model Zero quality loss (mathematically equivalent to target model) Best when draft model is 5-10× smaller than target Source: arXiv 2401.10774 (2024) Innovation: Add multiple prediction heads to existing model, predict future tokens without separate draft model. Architecture: Training: Medusa-1: Freeze base LLM,

This is an excerpt. Refer to the complete source and its current requirements.