model-architecture-torchtitan

原文涵盖的主题

• TorchTitan - PyTorch Native Distributed LLM Pretraining
• Quick start
• Common workflows
• Workflow 1: Pretrain Llama 3.1 8B on single node
• Workflow 2: Multi-node training with SLURM
• Workflow 3: Enable Float8 training for H100s
• Workflow 4: 4D parallelism for 405B models
• When to use vs alternatives

TorchTitan is PyTorch's official platform for large-scale LLM pretraining with composable 4D parallelism (FSDP2, TP, PP, CP), achieving 65%+ speedups over baselines on H100 GPUs. Installation: Download tokenizer: Start training on 8 GPUs: Copy this checklist: Step 1: Download tokenizer Step 2: Configure training Edit or create a TOML config file: Step 3: Launch training Step 4: Monitor and checkpoint TensorBoard logs are saved to ./outputs/tb/: Step 1: Configure parallelism for scale For 70B model on 256 GPUs (32 nodes): Step 2: Set up SLURM script Step 3: Submit job Step 4: Resume from checkpoint Training auto-resumes if checkpoint exists in configured folder. Float8 provides 30-50% speedup on H100 GPUs. Step 1: Install torchao Step 2: Configure Float8 Add to your TOML config: Step 3: Launch with compile Step 1: Create seed checkpoint Required for consistent initialization across PP stages: Step 2: Configure 4D parallelism Step 3: Launch on 512 GPUs Use TorchTitan when: Pretraining LLMs from scratch (8B to 405B+) Need PyTorch-native solution without third-party dependencies Require composable 4D parallelism (FSDP2, TP, PP, CP) Training on H100s with Float8 support Want interoperab

这里只展示节选，请以完整原文和当前要求为准。