trl-training

Topics in the source

• TRL
• Dataset formats
• SFT: the fields that matter
• GRPO: online RL
• CLI

Each method pairs a Trainer class with a Config dataclass. Configs extend transformers.TrainingArguments, so all of its arguments work in any trainer config. | Trainer | Dataset type | |---|---| | SFTTrainer | language modeling or prompt-completion | | DPOTrainer | preference (chosen/rejected pairs) | | GRPOTrainer | prompt-only + reward function(s) | | DistillationTrainer | prompt-only + a teacher model (on-policy distillation) | | KTOTrainer | unpaired preference (per-sample bool label) | | RewardTrainer | preference (chosen/rejected pairs); trains a scalar reward model, not a policy | Many more trainers (OnlineDPO, ORPO, CPO, GKD, …) live in trl.experimental with unstable APIs: https://huggingface.co/docs/trl/experimentaloverview Pass model as a string and route loading kwargs through modelinitkwargs (e.g. {"dtype": "bfloat16", "attnimplementation": "kernels-community/flash-attn2"}) instead of calling frompretrained yourself. The tokenizer/processor is inferred from the model; pass processingclass only when it differs. For LoRA, pass peftconfig=LoraConfig(...). Conversational: {"messages": [{"role": ..., "content": ...}]} (language modeling) or {"prompt": [...], "completion": [.

This is an excerpt. Refer to the complete source and its current requirements.