model-architecture-torchtitan

Topics in the source

• TorchTitan - PyTorch Native Distributed LLM Pretraining
• Quick start
• Common workflows
• Workflow 1: Pretrain Llama 3.1 8B on single node
• Workflow 2: Multi-node training with SLURM
• Workflow 3: Enable Float8 training for H100s
• Workflow 4: 4D parallelism for 405B models
• When to use vs alternatives

TorchTitan is PyTorch's official platform for large-scale LLM pretraining with composable 4D parallelism (FSDP2, TP, PP, CP), achieving 65%+ speedups over baselines on H100 GPUs. Installation: Download tokenizer: Start training on 8 GPUs: Copy this checklist: Step 1: Download tokenizer Step 2: Configure training Edit or create a TOML config file: Step 3: Launch training Step 4: Monitor and checkpoint TensorBoard logs are saved to ./outputs/tb/: Step 1: Configure parallelism for scale For 70B model on 256 GPUs (32 nodes): Step 2: Set up SLURM script Step 3: Submit job Step 4: Resume from checkpoint Training auto-resumes if checkpoint exists in configured folder. Float8 provides 30-50% speedup on H100 GPUs. Step 1: Install torchao Step 2: Configure Float8 Add to your TOML config: Step 3: Launch with compile Step 1: Create seed checkpoint Required for consistent initialization across PP stages: Step 2: Configure 4D parallelism Step 3: Launch on 512 GPUs Use TorchTitan when: Pretraining LLMs from scratch (8B to 405B+) Need PyTorch-native solution without third-party dependencies Require composable 4D parallelism (FSDP2, TP, PP, CP) Training on H100s with Float8 support Want interoperab

This is an excerpt. Refer to the complete source and its current requirements.