model-architecture-litgpt

Topics in the source

• LitGPT - Clean LLM Implementations
• Quick start
• Common workflows
• Workflow 1: Fine-tune on custom dataset
• Workflow 2: LoRA fine-tuning on single GPU
• Workflow 3: Pretrain from scratch
• Workflow 4: Convert and deploy model
• When to use vs alternatives

LitGPT provides 20+ pretrained LLM implementations with clean, readable code and production-ready training workflows. Installation: Load and use any model: List available models: Copy this checklist: Step 1: Download pretrained model Models are saved to checkpoints/ directory. Step 2: Prepare dataset LitGPT supports multiple formats: Alpaca format (instruction-response): Save as data/mydataset.json. Step 3: Configure training Step 4: Run fine-tuning Training saves checkpoints to out/finetune/ automatically. Monitor training: Most memory-efficient option. Step 1: Choose base model For limited GPU memory (12-16GB): Phi-2 (2.7B) - Best quality/size tradeoff Llama 3 1B - Smallest, fastest Gemma 2B - Good reasoning Step 2: Configure LoRA parameters LoRA rank guide: r=8: Lightweight, 2-4MB adapters r=16: Standard, good quality r=32: High capacity, use for complex tasks r=64: Maximum quality, 4× larger adapters Step 3: Train with LoRA Step 4: Merge LoRA weights (optional) Merge LoRA adapters into base model for deployment: Now use merged model: Train new model on your domain data. Step 1: Prepare pretraining dataset LitGPT expects tokenized data. Use preparedataset.py: Step 2: Configure m

This is an excerpt. Refer to the complete source and its current requirements.