post-training-simpo

Topics in the source

• SimPO - Simple Preference Optimization
• Quick start
• Common workflows
• Workflow 1: Train from base model (Mistral 7B)
• Workflow 2: Fine-tune instruct model (Llama 3 8B)
• Workflow 3: Reasoning-intensive tasks (lower LR)
• When to use vs alternatives
• Common issues

SimPO is a reference-free preference optimization method that outperforms DPO without needing a reference model. Installation: Training (Mistral 7B): Config (mistral-7b-base-simpo.yaml): Launch training: Config (llama3-8b-instruct-simpo.yaml): Launch: For math/code tasks: Use SimPO when: Want simpler training than DPO (no reference model) Have preference data (chosen/rejected pairs) Need better performance than DPO Limited compute resources Single-node training sufficient Algorithm selection: SimPO: Simplest, best performance, no reference model DPO: Need reference model baseline, more conservative PPO: Maximum control, need reward model, complex setup GRPO: Memory-efficient RL, no critic Use alternatives instead: OpenRLHF: Multi-node distributed training, PPO/GRPO TRL: Need multiple methods in one framework DPO: Established baseline comparison Issue: Loss divergence Reduce learning rate: Reduce beta: Issue: Model forgets capabilities Add SFT regularization: Issue: Poor preference separation Increase beta and margin: Issue: OOM during training Reduce batch size: Enable gradient checkpointing: Loss functions: See references/loss-functions.md for sigmoid vs hinge loss, mathematical f

This is an excerpt. Refer to the complete source and its current requirements.