distributed-training-megatron-core

Topics in the source

• Megatron-Core - Large-Scale LLM Training
• Quick start
• Common workflows
• Workflow 1: Train LLaMA-style model with 3D parallelism
• Workflow 2: Configure Mixture of Experts (MoE) training
• Workflow 3: Optimize for maximum throughput
• When to use vs alternatives
• Common issues

Megatron-Core trains LLMs from 2B to 462B parameters with up to 47% Model FLOP Utilization on H100 GPUs through advanced parallelism strategies. Installation: Simple distributed training: Copy this checklist: Step 1: Choose parallelism configuration Model size determines parallelism strategy: | Model Size | GPUs | Tensor Parallel | Pipeline Parallel | Data Parallel | Context Parallel | |------------|------|-----------------|-------------------|---------------|------------------| | 7B | 8 | 1 | 1 | 8 | 1 | | 13B | 8 | 2 | 1 | 4 | 1 | | 70B | 64 | 4 | 4 | 4 | 1 | | 405B | 128 | 8 | 8 | 2 | 2 | Step 2: Configure training hyperparameters Step 3: Launch distributed training Step 4: Monitor performance metrics Key metrics to track: For sparse MoE models like Mixtral. Step 1: Configure expert parallelism Step 2: Set MoE hyperparameters Step 3: Launch training with EP Expert parallelism distributes different experts across GPUs, reducing memory while maintaining capacity. Achieve 47% MFU on H100. Step 1: Enable optimizations Step 2: Use FP8 precision (H100 only) Result: 1.5-2x speedup on H100 vs BF16. Step 3: Optimize micro-batch size Find largest micro-batch that fits in memory: Typical v

This is an excerpt. Refer to the complete source and its current requirements.