model-architecture-nanogpt

Topics in the source

• nanoGPT - Minimalist GPT Training
• Quick start
• Common workflows
• Workflow 1: Character-level Shakespeare
• Workflow 2: Reproduce GPT-2 (124M)
• Workflow 3: Fine-tune pretrained GPT-2
• Workflow 4: Custom dataset
• When to use vs alternatives

nanoGPT is a simplified GPT implementation designed for learning and experimentation. Installation: Train on Shakespeare (CPU-friendly): Output: Complete training pipeline: Config (config/trainshakespearechar.py): Training time: 5 minutes (CPU), 1 minute (GPU) Multi-GPU training on OpenWebText: Config (config/traingpt2.py): Training time: 4 days (8× A100) Start from OpenAI checkpoint: Example config (config/finetuneshakespeare.py): Train on your own text: Train: Use nanoGPT when: Learning how GPT works Experimenting with transformer variants Teaching/education purposes Quick prototyping Limited compute (can run on CPU) Simplicity advantages: 300 lines: Entire model in model.py 300 lines: Training loop in train.py Hackable: Easy to modify No abstractions: Pure PyTorch Use alternatives instead: HuggingFace Transformers: Production use, many models Megatron-LM: Large-scale distributed training LitGPT: More architectures, production-ready PyTorch Lightning: Need high-level framework Issue: CUDA out of memory Reduce batch size or context length: Issue: Training too slow Enable compilation (PyTorch 2.0+): Use mixed precision: Issue: Poor generation quality Train longer: Lower temperature

This is an excerpt. Refer to the complete source and its current requirements.