tanh.xyz← Research notes
Research note 01 · Reinforcement learning

Shared representations.
One connected learning loop.

The environment produces experience. Fixed algorithms turn it into decisions and learning signals. Each learner predicts when queried and learns when given training data. Its loss and optimizer stay inside.

One graph, shared representations. Actor, critic, and dynamics are functional roles: they may share parameters and train jointly. Single-line arrows carry data. Double-line arrows import a function: the destination can call it repeatedly with its own inputs. Every arrow has one direction.

Function import → destination can call the source functionEnvironmentDataLearner (function + training)Fixed algorithmTraining data →Data flow →
Unified reinforcement learning: a connected computation graphThe graph connects action selection, planning, experience recording, target construction, and learning. Single arrows carry data; double-line arrows import callable functions; dashed red arrows supply training data. Learners may share parameters. Line crossings are not junctions. The explanation follows the diagram.ACTPOLICY AND VALUEIMAGINE · MODEL-GENERATED DATARECORD · ALIGN · BATCHselector → action: selectselectaction → env: stepstepenv → state: next observation + reward + terminationnext observation + reward + terminationstate → planner: same xₜseed planning with xₜbuilder → targets: buildtargetsbatch → builder: batch: rewards, masks, trajectory order + metadataB + metadatatargets → actor: actor training data²actor training²targets → critic: critic: states / actions + return targetscritic trainingplanner → search-results: generated datagenerated datasearch-results → recorder: training fieldstraining fieldsstate → recorder: step datastep dataaction → recorder: actual action aₜactual action aₜactor → planner: function import; the planner invokes this callable on its own inputs and uses its returned predictionsimport policy functioncritic → planner: function import; the planner invokes this callable on its own inputs and uses its returned predictionsimport value functionmodel → planner: function import; the planner invokes this callable on its own inputs and uses its returned predictionsimport dynamicsrecorder → batch: sample / batchsample / batchbatch → model: dynamics trainingdynamics trainingCritic → Target builder: imports the value function to evaluate sampled states or state-action pairs internally; prediction does not update parametersimport value functionActor → Action selector: imports the policy function, evaluates it on the current state, and samples or selects an action internallyimport policy functionCurrent step data → Action selector: current state x for invoking the imported policycurrent state xCritic → Action selector: optional Q-function import; selector evaluates candidate actions at the supplied current state internallyimport Q (optional)Planner → Action selector: imports the planning function; selector calls it on the current state and uses its returned action or action distributionimport plannerCurrent observation / step data: xₜ with the preceding transition’s reward rₜ₋₁ and termination dₜ₋₁ when available. Policy and planning consume x; the recorder consumes step data.DATACurrent step dataxₜ · rₜ₋₁ · dₜ₋₁Actor learner: πθ(a | x)LEARNER ROLEActor learnerπθ(a | x)Action selector: sample / ε-greedy / planFIXED ALGORITHMAction selectorsample / ε-greedy / planAction: aₜDATAActionaₜEnvironment: step(aₜ) → x′, r, dENVIRONMENTEnvironmentstep(aₜ) → x′, r, dPlanner / rollout: propose → score → chooseFIXED ALGORITHMPlanner / rolloutpropose → score → chooseDynamics learner: Mψ(x, a)LEARNER ROLEDynamics learnerMψ(x, a)Experience batch: sampled transitions or sequences, with observed rewards, termination flags and optional model-generated fields; target construction has not yet been appliedDATAExperience batchsampled transitions / sequencesCritic learner: Vφ(x) or Qω(x, a)LEARNER ROLECritic learnerVφ(x) or Qω(x, a)Target builder: TD / n-step / GAE / MCFIXED ALGORITHMTarget builderTD / n-step / GAE / MCTargets + experience: sampled experience paired with return targets, advantages, and applicable policy targets, ready for actor and critic trainingDATATargets + experienceexperience · returns · advantagesModel-generated data: predicted transitions and/or search policy and value estimates. The algorithm determines which fields are produced and consumed.DATAModel-generated datatransitions · policy · valueExperience recorder: stores and aligns records, selects eligible samples, and builds algorithm-appropriate training batchesFIXED ALGORITHMExperience recorderstore · align · sample · batch

Select a phase to trace its arrows. Search connections include MuZero-style latent planning. On small screens, scroll the diagram sideways. Straight arrows follow nearby dependencies; crossings are not junctions. Motion illustrates dependencies, not execution timing.

The complete graph

Every region belongs to the same system. The experience recorder stores, aligns, samples, and batches experience. Experience batches feed dynamics learning and target construction; the resulting targets plus experience feed actor and critic learning. Each learner contains its own loss and optimizer. Optional paths describe alternatives, not a single algorithm that uses everything.

One interface, several RL algorithms

This graph unifies reinforcement learning around functional roles and the data they use. It is a computation map, not a single algorithm: each method uses the functions, record types, and training objectives it needs. Actor, critic, and dynamics roles may share representations, parameters, and a joint training procedure.

Read the arrows

Single solid arrows carry data. Double-line arrows import a callable function: the destination supplies inputs and uses the returned result internally. Importing a function does not copy its parameters or train it. Dashed red arrows supply training data. Each learner computes its predictions, objective, gradients, and optimizer steps internally. Crossings are not junctions.

Select an action

The action selector receives the current state and can invoke the actor, the critic’s Q-function, or the planner. It samples from an actor distribution, selects from action values, or uses the planner’s returned action or distribution. These alternatives share the same action output and environment loop.

Plan with imported functions

The planner imports the learned dynamics function and, when needed, policy and value functions. It calls them on hypothetical states and actions to generate and score possible futures. The selector imports the planner itself, so its internal calls and returns need no additional prediction boxes.

Record once, batch inside the recorder

Current step data carries xₜ and, after an environment step, the preceding transition’s reward rₜ₋₁ and termination flag dₜ₋₁. Decision functions consume x; the recorder receives the step data and selected action. By retaining the previous state and action, it assembles (xₜ, aₜ, rₜ, xₜ₊₁, dₜ) when the next result arrives.

The recorder stores and aligns records, samples eligible transitions or sequences, and outputs an Experience batch. Sampling rules are internal to this fixed algorithm. The shared batch interface does not require all learners to use identical samples. Initial/reset observations start an episode; retain terminal observations before reset and never join records across episode boundaries.

One model-generated data interface

Model-generated data can contain predicted transitions, search policies, and value estimates. Dyna-style learning uses predicted transitions as training samples. MuZero uses search policies and value information alongside observed trajectories. These are different uses of one interface; an algorithm need not produce or consume every field.

The recorder can store a synthetic transition as a record or attach a search annotation to its matching environment timestep. Their sources remain explicit: a search estimate does not become an observed outcome because both are stored together. Search runs before the actual action and summarizes hypothetical futures.

Build targets, then learn

The target builder receives an Experience batch and imports the critic function. It evaluates the states or state–action pairs needed for returns, bootstraps, or advantages internally. The resulting Targets + experience package retains the samples and their appropriate learning signals.

The actor uses policy-learning signals such as advantages or search-policy targets. The critic uses return targets. The dynamics learner receives experience batches with the transition or sequence supervision its objective requires. Shared parameters may receive a joint update across these roles; separate arrows do not require independent optimizers.

Inside a learner: prediction and learning

predict(inputs) → outputs

Evaluate the current function without updating parameters. The caller chooses the input states or state–action pairs.

learn(training_data)

Compute predictions and the configured objective, differentiate, and apply optimizer steps. An appropriate objective and valid training signals are still required.

How algorithms fit

MethodAction selectionTraining
PPO with a V-criticSelector imports the actor.Fresh trajectories, commonly GAE and return targets; preserve trajectory order and old action log-probabilities.
Q-learningSelector imports Q for greedy or ε-greedy control.Observed transitions and bootstrapped Q targets; deep variants commonly use replay and a lagged target network.
Model-based planningSelector imports the planner, which invokes learned dynamics and optional policy/value functions.Fit the model to experience; actor training is optional.
Dyna-style learningPolicy or Q-based selection.Model-generated transitions supplement environment experience through the common recorder and batching interface.
MuZeroSelector invokes search over learned latent dynamics, policy, and value functions.Observed sequences plus search annotations supervise jointly trained representation, dynamics, policy, and value functions.

MuZero and shared representations

MuZero’s representation function h can live inside the learner roles: it maps observation history to a latent state. Dynamics g advances that latent state, and prediction f supplies policy and value outputs. Search repeatedly invokes these functions; latent states need not reconstruct future observations. Training unrolls dynamics along recorded actions and jointly supervises rewards, values, and policies. Schrittwieser et al., 2020.

Search-tree transitions are not automatically independent replay samples. Search policy and root-value annotations can instead accompany the corresponding observed trajectory. The graph captures this through the same Model-generated data interface used for synthetic transitions, while preserving their different training roles. DeepMind’s MuZero explanation.

² PPO behavior information. Store log πold(a | x) with rollouts. The actor recomputes its current action probabilities during learning. Arbitrary replay and model-generated transitions are not interchangeable with standard PPO rollouts.

Targets are computed, not learned

One-step value target

yₜ = rₜ + γ(1 − dₜ) V̄(xₜ₊₁)
δₜ = yₜ − V(xₜ)

For Q-learning, replace V̄(x′) with maxₐ′ Q̄(x′, a′). The bar denotes the value source used for bootstrapping, often a lagged target network in deep Q-learning. The graph groups online and bootstrap evaluations in the critic node; target-network copying / averaging is an implementation detail omitted here.

Advantage and regression

Âₜ = Σₗ (γλ)ˡ δₜ₊ₗ
Lvalue = mean[(Vφ(xₜ) − yₜ)²]

The GAE sum stays within the sampled trajectory, with boundary masks. GAE estimates advantages; a corresponding return target is Âₜ + Vold(xₜ). Monte Carlo uses observed discounted returns, without a bootstrap at a true terminal state. Targets are held fixed in the usual semi-gradient value update.

Termination is not truncation. Here d means a true terminal transition. A time-limit cutoff generally still permits bootstrapping from the final observation, although the advantage recursion must not continue into a reset episode. x means the state or history representation supplied to the learner; an observation alone need not be Markov.

Scope. Policy and value function imports are optional planner capabilities; an algorithm uses only the functions it requires. Networks may share parameters. Dynamics can be probabilistic or an ensemble. Learners encapsulate algorithm-specific objectives and autodifferentiation. Receiving arbitrary data does not guarantee learning or improvement: the training call needs a specified objective, appropriate data, and an optimization rule. Those are hidden inside the module, not assumed to disappear. A fixed algorithm can call a learned function without itself becoming a learned network.

Sources & reading

The connected layout and animation are an explanatory synthesis of the supplied diagrams. Algorithm distinctions follow these primary sources:

  1. OpenAI Spinning Up — Key concepts in RL: policies, value functions, and model-based versus model-free methods.
  2. Schulman et al. — Proximal Policy Optimization Algorithms (2017): sampled trajectories, policy-ratio objectives, and repeated optimization.
  3. Schulman et al. — Generalized Advantage Estimation (2015): advantage estimates from discounted temporal-difference residuals.
  4. OpenAI Spinning Up — The Q-learning side of DDPG: replay, terminal masks, Bellman targets, and target networks.
  5. David Silver — Integrating Learning and Planning: model learning, simulated experience, and Dyna.