optimization-flash-attention

Topics in the source

• Flash Attention - Fast Memory-Efficient Attention
• Quick start
• Common workflows
• Workflow 1: Enable in existing PyTorch model
• Workflow 2: Use flash-attn library for advanced features
• Workflow 3: H100 FP8 optimization (FlashAttention-3)
• When to use vs alternatives
• Common issues

Flash Attention provides 2-4x speedup and 10-20x memory reduction for transformer attention through IO-aware tiling and recomputation. PyTorch native (easiest, PyTorch 2.2+): flash-attn library (more features): Copy this checklist: Step 1: Check PyTorch version If 512 tokens. Step 4: Test accuracy matches baseline For multi-query attention, sliding window, or H100 FP8. Copy this checklist: Step 1: Install flash-attn library Step 2: Modify attention code Step 3: Enable advanced features Multi-query attention (shared K/V across heads): Sliding window attention (local attention): Step 4: Benchmark performance For maximum performance on H100 GPUs. Step 1: Verify H100 GPU Step 2: Install flash-attn with FP8 support Step 3: Convert inputs to FP8 Step 4: Run with FP8 attention Use Flash Attention when: Training transformers with sequences >512 tokens Running inference with long context (>2K tokens) GPU memory constrained (OOM with standard attention) Need 2-4x speedup without accuracy loss Using PyTorch 2.2+ or can install flash-attn Use alternatives instead: Standard attention: Sequences 2K tokens: 3-4x speedup Check sequence length is sufficient. Issue: RuntimeError: CUDA error Verify G

This is an excerpt. Refer to the complete source and its current requirements.