Reinforcement Learning

Reinforcement learning studies one problem. An agent takes an action, an environment returns a scalar reward and a new state, and the agent must learn behaviour that maximises reward accumulated over time. That framing sounds narrow. It is not. A robot arm, an inventory reorder policy, an auction bidding strategy, a cache eviction rule, a board game, and a language model deciding which token to emit next can all be written as a Markov decision process with states, actions, transition dynamics and a reward function.

What separates RL from supervised learning is the shape of the feedback. Supervised learning hands you the correct answer for every input. RL hands you a number, usually late, usually many actions after the one that earned it, and leaves you to work out which decision deserves the credit. That is the temporal credit assignment problem, and it is why the field developed its own algorithms instead of reusing a loss over labelled pairs.

Two consequences follow, and both are why RL is harder than it looks on a slide.

The first is that the data distribution depends on the policy you are training. Improve the policy and it visits different states, which changes the data, which changes the gradient. Supervised learning optimises against a fixed dataset. RL optimises against a dataset it is itself causing to move, which is why training curves wander, collapse and recover in ways a classification run never does.

The second is that the agent has to gather its own data. That forces the exploration and exploitation trade-off: exploiting the best action you currently know about is how you earn reward now, exploring is the only way to discover something better, and with a finite budget you cannot do both fully.

Multi-armed bandits are the honest place to start, because they strip out state and leave only that trade-off. Once epsilon-greedy, optimistic initialisation, upper confidence bounds and Thompson sampling feel obvious in the bandit setting, the full MDP machinery — returns, discounting, state and action value functions, the Bellman equations, policy evaluation and policy improvement — reads as a generalisation rather than a wall of notation. Most people who bounce off reinforcement learning bounce off because they met the Bellman optimality equation before they had any reason to want it.

Every RL algorithm is a different answer to the same question: how do you estimate the value of behaviour you have not fully observed?

Dynamic programming

Value iteration and policy iteration compute an optimal policy exactly, given the transition model and the reward function. That assumption almost never holds outside a textbook, so dynamic programming is taught as the reference point rather than the tool. Everything after it is an attempt to get the dynamic programming answer without the model.

Monte Carlo and temporal difference

Monte Carlo methods drop the model and learn from complete episodes: average the returns actually observed after visiting a state, and you have an unbiased value estimate. The price is high variance and a hard requirement that episodes terminate.

Temporal difference learning is the compromise that made RL practical. Instead of waiting for the episode to end, TD updates a value estimate towards the immediate reward plus its own current estimate of the next state. That is bootstrapping — learning a guess from a guess. It adds bias, cuts variance sharply, and works in continuing tasks with no natural end. Q-learning and SARSA are the canonical control versions, and the gap between them is the gap between off-policy and on-policy learning. Q-learning learns the value of acting greedily while behaving exploratively. SARSA learns the value of the policy it is actually following, which makes it more cautious near catastrophic states. The classic cliff-walking example exists precisely to make that difference visible.

Function approximation and the deadly triad

Tables stop working the moment the state space is large or continuous, so values get approximated by a neural network. Combining bootstrapping, off-policy learning and function approximation is known as the deadly triad, and the combination can diverge rather than merely converge slowly. The engineering around deep Q-networks — a replay buffer to break correlation between consecutive samples, a slowly updated target network to stop the regression target chasing its own tail — exists to make that combination behave.

Policy gradients and actor-critic

Policy gradient methods attack from the other side. Parameterise the policy directly and push it up the gradient of expected return. REINFORCE is the base case. It handles continuous action spaces and stochastic policies naturally, which value-based methods do not, and it pays with variance. Baselines reduce that variance, advantage estimates reduce it further, and actor-critic methods make the baseline a learned value function.

Advanced policy optimisation is mostly the discipline of not destroying a working policy with one oversized update. TRPO constrains the divergence between the old and new policy. PPO, published by OpenAI, gets a similar effect with a clipped objective that is far simpler to implement, which is why it became the default. Soft actor-critic adds an entropy bonus and is off-policy, which makes it the usual first choice for continuous control where samples are expensive.

The practical trade-off is a triangle. Off-policy value methods reuse data and are sample efficient but fragile. On-policy policy gradients are stable and easy to reason about but discard data after each update. Model-based methods are the most sample efficient and the most sensitive to model error.

Algorithms get the attention. Reward functions decide outcomes.

The reward function is a specification of what you want, written in a language that permits no ambiguity and offers no partial credit for good intentions. An agent optimising it will find every gap between what you wrote and what you meant. This is not a hypothetical concern: OpenAI published a well-known example of an agent in a boat racing game that learned to loop through a lagoon collecting respawning score pickups instead of finishing the race, because the score was the reward and finishing was not. The behaviour was optimal. The specification was wrong.

Sparse rewards create the opposite problem. If reward arrives only on success and success is rare, random exploration will essentially never see a positive signal, and there is nothing for the algorithm to climb. The usual responses are reward shaping, curriculum design, demonstrations, and intrinsic motivation bonuses that reward novelty or prediction error.

Shaping deserves care. Adding a helpful-looking bonus can change the optimal policy, which means you have quietly replaced the task. Potential-based reward shaping, introduced by Ng, Harada and Russell, is the construction that provably preserves the optimal policy, and it is worth learning the form rather than inventing bonuses by feel.

Three habits separate people who ship RL from people who publish plots:

• Keep the training reward and the evaluation metric separate. The training reward is a lever you are allowed to shape. The evaluation metric is what the business actually cares about, and it must never be shaped to make training look better.
• Read the trajectories, not just the curve. Reward going up while the behaviour becomes nonsense is the single most common failure, and a plot cannot show it. Watch episodes. Render them if you can.
• Assume Goodhart's law applies. Any proxy strong enough to be optimised hard will eventually stop measuring the thing you chose it for. Build in periodic re-grounding against the real objective.

Constraints belong in the design too. Many real deployments do not want an unconstrained maximiser, they want a policy that maximises subject to a hard limit on spend, risk or physical safety. Constrained MDPs, Lagrangian penalties and action masking are the standard vocabulary, and choosing between them is a design decision, not a hyperparameter.

Model-free RL needs an enormous number of interactions. In a simulator that is a compute bill. On a physical robot, a warehouse or a live pricing system, it is not a bill at all — it is simply impossible, because you cannot let an untrained policy explore against real customers or real hardware.

Two families exist to deal with that.

Model-based RL

Learn a model of the environment dynamics, then use it. You can plan against it directly with model predictive control, generate synthetic experience to train a model-free learner in the style of Dyna, or learn a latent model whose only job is to be useful for planning, which is the idea behind MuZero from DeepMind. The payoff is sample efficiency, because most of the learning happens inside the model rather than against the world.

The failure mode is compounding error. A dynamics model that is slightly wrong at each step becomes badly wrong over a long rollout, and the planner will happily exploit whatever fictional region of the model looks most rewarding. Short rollouts, ensembles for uncertainty, and penalising plans that stray into uncertain regions are the standard defences.

Offline RL

Offline RL learns a policy from a fixed dataset of logged interactions with no further exploration. That matches a great many real situations: you already have years of logs from whatever heuristic policy is running today.

The core difficulty is distributional shift. The learner will evaluate actions that the logging policy rarely or never took, the value function has no evidence about them, and function approximation tends to be optimistic in exactly those gaps. The result is a policy that looks excellent offline and is terrible in deployment. Conservative approaches deliberately push down the estimated value of out-of-distribution actions to prevent it.

There is also a humbling baseline that offline RL papers report for good reason: plain behaviour cloning of the best segment of the logged data is often hard to beat. If a straightforward imitation baseline matches your offline RL policy, you have learned something valuable about whether the problem needs RL at all.

Simulation-to-reality transfer sits alongside both. Train in a simulator, randomise the parts of the simulator you are least confident about, and you get a policy less likely to depend on details the simulator got wrong. The residual gap is still the main reason robotics timelines slip.

The reason RL is suddenly on every machine learning engineer's roadmap is that it turned out to be the mechanism for aligning language model behaviour with human judgement.

The classic pipeline, described publicly by OpenAI in the InstructGPT work, has three stages. First, supervised fine-tuning on demonstrations of the behaviour you want. Second, a reward model: collect pairwise human preferences over model outputs and fit a model that scores a response, typically with a Bradley-Terry style objective over the pairs. Third, optimise the policy against that reward model with PPO, plus a KL penalty that holds the policy near the supervised reference.

That KL term is not a detail. Without it, the policy drifts into whatever degenerate text the reward model happens to score highly — reward model overoptimisation, where the true quality peaks and then falls while the proxy score keeps climbing. Recognising that curve is one of the practical skills the topic demands.

Since then the space has split into several approaches worth being able to distinguish.

• Direct preference optimisation, introduced by researchers at Stanford, removes the separate reward model and the RL loop entirely, deriving a supervised-style loss directly on preference pairs. The related methods that followed it came out of different groups and vary in what they assume about the data — some drop the paired requirement, some fold the supervised objective back in. Cheaper and more stable than an online loop, and now the usual starting point.
• Reinforcement learning from AI feedback, including the Constitutional AI approach published by Anthropic, replaces or supplements human labels with model-generated critiques against a written set of principles, which changes the economics of collecting preference data.
• Reinforcement learning with verifiable rewards applies to domains where correctness can be checked mechanically — mathematics with a checkable answer, code with a test suite. DeepSeek released open weights and an accompanying paper for a reasoning model trained this way, which made the technique concrete for people outside the large labs. Critic-free variants such as group-relative policy optimisation reduce the memory cost by scoring a group of sampled completions against each other instead of training a separate value network.

The binding constraint in all of this is rarely the algorithm. It is the preference data: who labelled it, whether the labellers agreed, whether the comparisons were on the model's own outputs, and whether the rubric matched the behaviour you actually wanted. A reward model trained on careless labels is an efficient machine for propagating carelessness.

RL research code and RL production systems fail for different reasons.

Seed variance is not noise, it is the result. Runs of the same algorithm with different random seeds can land in visibly different places. Any comparison based on a single run of each method is not evidence. Report multiple seeds, and treat a favourable single seed as a hypothesis rather than a finding.

RL bugs are silent. A sign error in an advantage calculation, an off-by-one in a done flag, or an observation normaliser fitted on the wrong statistics does not raise an exception. It produces a policy that learns something, just not the thing you asked for. The standard defences are unglamorous: unit-test the environment, verify that a random policy gets the return you expect, confirm that a trivially solvable variant of the task is solved quickly, and only then run the real thing.

The environment is part of the model. Change the reward scale, the observation preprocessing, the frame skip or the episode termination rule, and you have changed the problem. Version the environment with the same rigour you version the weights, or your comparisons across weeks are meaningless.

Deployment is non-stationary. Users adapt, competitors adapt, seasons change, and other agents in the system are learning too. A policy that was optimal against last quarter's distribution is now operating off-distribution. Continuous evaluation and scheduled retraining are the norm, not a sign that something went wrong.

Off-policy evaluation is how you avoid shipping a disaster. Before a learned policy touches traffic, you want an estimate of its performance from logged data. That requires having logged the propensities of the policy that generated the logs, which is a decision you have to make before you need it. Teams that did not log action probabilities discover this at the worst possible moment.

Most industrial problems are bandit problems. Contextual bandits with a well-instrumented exploration policy are cheaper, safer, easier to evaluate and easier to explain than full sequential RL, and they are the right answer whenever the action does not meaningfully change the state you see next. Reaching for deep RL when a contextual bandit would do is the most common strategic mistake in the field.

Finally, guardrails belong outside the learned policy. Action masking, rate limits, spend caps and a hard kill switch are not admissions that the policy is untrustworthy; they are what makes it deployable at all.

A quick self-check. You are ready for this material if you can explain why a validation set exists, write a PyTorch training loop from memory, and describe what an expectation is without reaching for a definition. You are not ready if neural network training is still mysterious — RL adds a second source of instability on top of the first, and debugging both at once is genuinely miserable.

The most reliable path through the subject looks like this.

1. Implement tabular methods yourself. Policy iteration, Monte Carlo control and Q-learning on a small grid world, with no framework. It takes an afternoon and it removes most of the notation anxiety permanently.
2. Move to function approximation on a standard benchmark. The Gymnasium environments maintained by the Farama Foundation are the common language here; classic control tasks train on a laptop CPU.
3. Implement one policy gradient method end to end. REINFORCE with a baseline, then advantage actor-critic, then PPO. Writing PPO once teaches more than reading ten explanations of the clipped objective.
4. Pick a direction. Continuous control leads to soft actor-critic and robotics. Language leads to preference optimisation and verifiable rewards. Operations leads to contextual bandits, constrained MDPs and off-policy evaluation.

For reference material, the Sutton and Barto textbook remains the standard for the foundations, and it is freely available from the authors. For the deep RL side, reading the original algorithm papers alongside a known-good implementation is more useful than any secondary summary, because the implementation details that make these algorithms work are frequently in the code and not in the paper.

If your actual goal is to change a language model's behaviour rather than to control an agent, the shortest useful route is to understand the preference optimisation family first and treat the online RL loop as an advanced option you may never need.