Reinforcement learning basics: states, actions, rewards, values
In reinforcement learning, an agent takes actions in an environment, receives rewards, and learns a policy — what to do in each state — that maximises the total (discounted) reward over time.
Why it matters
Many problems have no labelled "right answers", only outcomes: win or lose a game, finish a task, keep a robot upright. Reinforcement learning learns from those outcomes by trial and error. It powers game-playing systems, robotics, recommendation tuning, and the RLHF and reasoning training of modern language models.
The idea
In reinforcement learning (RL), an agent takes actions in an environment and gets rewards. It learns a policy, meaning what to do in each state, that collects the most reward over time. A value function says how good a state is. Q-learning learns how good each action is in each state.
Training a dog with treats. Nobody shows the dog the right move. It tries things, and good actions earn treats.
A dog gets its treat straight away. In RL the reward often comes much later, like winning a game after 100 moves, so the agent must work out which earlier actions deserve the credit.
How it works
- The agent sees a state s, takes an action a, and the environment returns a reward r and a new state s′. This repeats over an episode.
- The return is the sum of future rewards, discounted by γ (between 0 and 1) per step, so sooner rewards count more.
- The value V(s) is the expected return from state s when following the policy; Q(s, a) is the expected return after taking action a in s.
- Values satisfy the Bellman equation: the value of a state = the reward for the best next step + γ × the value of where it leads.
- Value iteration applies that rule repeatedly; value spreads back from rewards. In a grid with γ = 0.8, the square next to the goal is worth 1, the one before that 0.8, then 0.64.
- Q-learning learns Q from experience without knowing the rules: Q(s, a) ← Q(s, a) + α·(r + γ·max Q(s′, ·) − Q(s, a)).
- Balance exploration (try new actions) and exploitation (use the best known one), for example with ε-greedy: random action with probability ε.
Key points
- Agent, environment, state, action, reward, policy.
- Return = discounted sum of future rewards; γ sets how far ahead the agent cares.
- V(s) values states; Q(s, a) values actions in states.
- Bellman equation: a value equals immediate reward plus discounted next value.
- Credit assignment: working out which earlier actions earned a delayed reward.
- Exploration versus exploitation is a constant trade-off.
Common mistakes
- No exploration, so the agent locks onto the first thing that worked.
- γ = 0, so the agent ignores everything but the immediate reward.
- Reward functions that can be "hacked": the agent maximises the number, not what you meant.
Not true: An agent will do what you meant, as long as its reward keeps going up.
In fact: An agent maximises the number it is given, not your intention. A carelessly designed reward can be collected in ways you never wanted.
Not true: With γ = 0, an agent plans far into the future.
In fact: γ = 0 makes every future reward count for nothing, so the agent cares only about the next one. A γ close to 1 makes it look far ahead.
Go deeper: Why value iteration always converges
- One sweep of value iteration replaces every value: new V(s) = the best over actions of [reward + γ × the average of V over the next states]. Call one sweep T, so the update is V ← T(V).
- Take any two tables of values, V and U, and let D be the largest difference between them over all states.
- Inside the brackets, V and U only enter through "γ × the average of the next values". Averages of numbers that differ by at most D differ by at most D, so for every action the two brackets differ by at most γD.
- Taking the best action on each side cannot widen the gap: two lists that differ by at most γD, entry by entry, have maxima that differ by at most γD. So T(V) and T(U) differ by at most γD.
- Now let U be the true optimal values V*, which a sweep leaves unchanged: T(V*) = V*. Then every sweep shrinks the distance to V* by the factor γ. After k sweeps the error is at most γᵏ times the starting error.
- With γ = 0.8 the error falls below 1% of its start in 21 sweeps. With γ = 0.99 it takes 459: caring about the far future makes planning slower.
A sweep of value iteration is a contraction: it shrinks the distance to the optimal values by γ every time. So it converges from any starting values, always to the same answer.
Part of Unit 7: Generative models and reinforcement learning in the free ML Foundations course.