spinning-up-deep-rl

Topics in the source

• Spinning Up in Deep RL
• How to Use This Skill
• Core Frameworks & Mental Models
• The RL problem
• The two branching questions
• Policy optimization vs Q-learning — the central trade-off
• The policy gradient template
• The policy-gradient loss is not a loss function

Author: Joshua Achiam (OpenAI) | Source: spinningup.readthedocs.io, MIT | Chapters: 20 | Generated: 2026-08-25 No argument — load the core frameworks below A topic — ask about advantage function, target networks, entropy regularization; I resolve it through the Topic Index and read that chapter file chNN — I load that chapter's summary "what chapters do you have?" — the full index When you ask about something not in Core Frameworks, I read the relevant chapter file before answering rather than guessing from the index. -- pi = argmaxpi J(pi), where J(pi) = E{taupi}[R(tau)]. Every algorithm approximates this; where it substitutes a different objective (a Bellman residual, a surrogate), that substitution is the source of its failure modes. Four value functions — V^pi, Q^pi, V, Q — all obey Bellman self-consistency, and a(s) = argmaxa Q(s,a) is why Q-learning is a viable family at all. Advantage A^pi(s,a) = Q^pi(s,a) - V^pi(s) is the relative-quality signal policy gradients run on. (ch07) Place any algorithm by asking: does it have or learn a model, and what does it learn (policy, Q-function, value function, model). That generates the whole landscape. (ch08) Policy optimization is prin

This is an excerpt. Refer to the complete source and its current requirements.