post-training-openrlhf

Topics in the source

• OpenRLHF - High-Performance RLHF Training
• Quick start
• Common workflows
• Workflow 1: Full RLHF pipeline (SFT → Reward Model → PPO)
• Workflow 2: GRPO training (no critic model needed)
• Workflow 3: DPO training (preference optimization)
• When to use vs alternatives
• Common issues

OpenRLHF is a Ray-based RLHF framework optimized for distributed training with vLLM inference acceleration. Installation: PPO Training (Hybrid Engine): GRPO Training (Group Normalized Policy Optimization): Step 1: Train reward model (DPO): Step 2: PPO training: Memory-efficient alternative to PPO: Key GRPO parameters: --advantageestimator groupnorm - Enables GRPO --useklloss - KL loss from GRPO paper --klestimator k3 - Loss function (k2 ≈ k1) --noadvantagestdnorm - Disables std normalization Simpler alternative without reward model: Use OpenRLHF when: Training large models (7B-70B+) with RL Need vLLM inference acceleration Want distributed architecture with Ray Have multi-node GPU cluster Need PPO/GRPO/RLOO/DPO in one framework Algorithm selection: PPO: Maximum control, best for complex rewards GRPO: Memory-efficient, no critic needed RLOO: Modified PPO with per-token KL REINFORCE++: More stable than GRPO, faster than PPO DPO: Simplest, no reward model needed Use alternatives instead: TRL: Single-node training, simpler API veRL: ByteDance's framework for 671B models DeepSpeedChat: Integrated with DeepSpeed ecosystem Issue: GPU OOM with large models Disable model colocation: Issue:

This is an excerpt. Refer to the complete source and its current requirements.