What the connections mean
This is a modular computation map, not a new RL algorithm. A shaded region names a responsibility; it never breaks a connection. Each purple box is a learner: a learned function together with its training procedure. A query returns a prediction without changing parameters. A training call consumes suitable data and runs the objective, gradients, and optimizer internally. Dashed red arrows carry training data, not gradients. Each state, dataset, and learner appears once.
Act → imagine → act
The actor returns a distribution; a fixed selector produces an action. Alternatively, a planner queries the learned dynamics repeatedly, scores rollouts, and sends its chosen action to the selector. The prediction-to-planner feedback is essential: model outputs do not select actions on their own. A Q-based controller queries the critic at the current state and sends Q-values to the ε-greedy selector, bypassing the actor.
Experience → targets → updates
Real transitions feed the sampler directly. Optional simulated rollouts join there. The critic evaluates the batch; a fixed target builder combines appropriate value estimates with rewards and episode information. The resulting learning package retains the batch alongside its targets and advantages. The actor receives states, actions, advantages, and any required behavior-policy metadata; the critic receives states or state–action pairs with return targets. Real transitions train the dynamics learner directly. Each learner computes its current predictions and updates its own parameters internally.
¹ Recording is bookkeeping. The downward edge stores the previous state and action together with the environment’s returned reward, next state, and termination. Data boxes store information; processing happens in the named functions or explicitly labeled edge operations (record, observe/reset, collect).
Inside a learner: two calls, one learned function
predict(inputs) → outputs
Run the current function. An actor returns a policy distribution, a critic returns a value, and a dynamics learner returns a transition prediction. This call does not update parameters.
learn(training_data)
Evaluate the function on the training inputs, compute the configured loss against the supplied targets or learning signals, differentiate, and take optimizer steps. The graph hides this machinery inside each learner.
Where imagined transitions go
There are two distinct uses of imagination. Imagined replay sends simulated transitions through the batch sampler to train a learner (the Dyna-style branch). Planning keeps simulations in a search procedure, where they improve the next real action. A planner can use this second path without producing a synthetic training batch.
MuZero: learn to support search
In the MuZero view, h encodes real observation history into a latent state, g advances that state for a candidate action and predicts reward, and f predicts policy and value. MCTS calls g and f repeatedly, backs up values, and selects a real action from root visit counts. These imagined latent states need not reconstruct future observations. Schrittwieser et al., 2020.
Real trajectories retain actions, rewards, and observations together with the root search policy and value. Training samples a sequence, unrolls g along its recorded actions, and jointly trains h, g, and f against observed rewards, return targets, and search policies. Losses and optimization stay inside the learner. The target builder uses terminal returns or bootstrapped returns as appropriate; root values stored with the trajectory can supply bootstraps. DeepMind’s MuZero explanation.
The feedback loop is search → policy targets → learner → better search. Vanilla MuZero does not treat every imagined tree edge as a replay transition. This view groups latent states, model predictions, and action selection with their owning modules to keep the connections legible. Reanalyze and stochastic-model extensions are outside this diagram’s scope.
Choose the algorithm, then the paths
| Method | Acting | Learning data & target | Learned functions |
|---|---|---|---|
| PPO with a V-critic | Actor → distribution → sampler | Fresh real trajectories; commonly GAE + return targets. Preserve rollout order and old log-probabilities. | Actor and V-critic; no model required. |
| Deep Q-learning | Current-state Q query → ε-greedy selector | Real replay; bootstrapped TD target, commonly using a lagged target network. | Q-critic; no actor required. |
| Model-based planning | Planner ↔ model → chosen action | Fit dynamics to real transitions. A planner can act without training an actor. | Dynamics; actor / critic optional. |
| Dyna-style learning | Policy or Q-based controller | Real and model-generated transitions train a value function or policy; real data fit the model. | Dynamics + value / policy functions as chosen. |
| MuZero | Latent dynamics + policy/value predictions → MCTS → real action | Real sequences plus root search statistics; unrolled reward, value, and search-policy targets. | Representation h, dynamics g, prediction f; trained jointly. |
² PPO needs behavior-policy information. Store log πold(a | x) with real rollouts; the actor learner reevaluates πθ(a | x) on the same samples internally. Arbitrary replay or imagined data are not standard PPO inputs. The optional imagined route requires an algorithm designed to use it.
Targets are computed, not learned
One-step value target
δₜ = yₜ − V(xₜ)
For Q-learning, replace V̄(x′) with maxₐ′ Q̄(x′, a′). The bar denotes the value source used for bootstrapping, often a lagged target network in deep Q-learning. The graph groups online and bootstrap evaluations in the critic node; target-network copying / averaging is an implementation detail omitted here.
Advantage and regression
Lvalue = mean[(Vφ(xₜ) − yₜ)²]
The GAE sum stays within the sampled trajectory, with boundary masks. GAE estimates advantages; a corresponding return target is Âₜ + Vold(xₜ). Monte Carlo uses observed discounted returns, without a bootstrap at a true terminal state. Targets are held fixed in the usual semi-gradient value update.
Termination is not truncation. Here d means a true terminal transition. A time-limit cutoff generally still permits bootstrapping from the final observation, although the advantage recursion must not continue into a reset episode. x means the state or history representation supplied to the learner; an observation alone need not be Markov.
Scope. Planner scoring may additionally use a terminal critic value or actor proposals; those optional calls are omitted to keep the core paths legible. Networks may share parameters. Dynamics can be probabilistic or an ensemble. Learners encapsulate algorithm-specific objectives and autodifferentiation. Receiving arbitrary data does not guarantee learning or improvement: the training call needs a specified objective, appropriate data, and an optimization rule. Those are hidden inside the module, not assumed to disappear. A fixed algorithm can call a learned function without itself becoming a learned network.
Sources & reading
The connected layout and animation are an explanatory synthesis of the supplied diagrams. Algorithm distinctions follow these primary sources:
- OpenAI Spinning Up — Key concepts in RL: policies, value functions, and model-based versus model-free methods.
- Schulman et al. — Proximal Policy Optimization Algorithms (2017): sampled trajectories, policy-ratio objectives, and repeated optimization.
- Schulman et al. — Generalized Advantage Estimation (2015): advantage estimates from discounted temporal-difference residuals.
- OpenAI Spinning Up — The Q-learning side of DDPG: replay, terminal masks, Bellman targets, and target networks.
- David Silver — Integrating Learning and Planning: model learning, simulated experience, and Dyna.