safety-alignment-constitutional-ai

Topics in the source

• Constitutional AI - Harmlessness from AI Feedback
• Quick start
• Common workflows
• Workflow 1: Supervised learning phase (self-critique + revision)
• Workflow 2: RL phase (RLAIF - RL from AI Feedback)
• Workflow 3: Chain-of-thought critique
• When to use vs alternatives
• Common issues

Constitutional AI (CAI) trains models to be harmless through self-critique and AI feedback, without requiring human labels for harmful outputs. Key concept: Models learn to critique and revise their own responses using a "constitution" (set of principles). Two phases: 1. Supervised Learning (SL): Self-critique + revision 2. Reinforcement Learning (RL): RLAIF (RL from AI Feedback) Constitution example: Step 1: Generate initial responses: Step 2: Self-critique with constitution: Step 3: Revision based on critique: Step 4: Fine-tune on revised responses: Step 1: Generate comparison pairs: Step 2: AI preference evaluation: Step 3: Train preference model (reward model): Step 4: RL training with RLAIF: Enable reasoning transparency: Use Constitutional AI when: Want safety alignment without human labels Need explainable AI decisions Want to avoid evasive refusals Have a clear set of principles/constitution Need scalable safety training Principles: RLAIF: AI-generated preferences (scalable, no human labels) RLHF: Human preferences (more accurate, expensive) Self-critique: Iterative improvement Chain-of-thought: Reasoning transparency Use alternatives instead: RLHF (PPO): Need human-validat

This is an excerpt. Refer to the complete source and its current requirements.