An introduction to reinforcement learning

Most introductions to reinforcement learning are written by people in academic RL. They start with Markov Decision Processes, the Bellman equation, value functions, and a small army of Greek letters. By page three you’ve met γ, π, and Q, and you’ve been told that “we want to maximise the expected discounted return” without being told why you should care.

This is not that introduction.

This is the first post in a five-part series for working engineers, and its job is the simplest possible one: make sure you have the right mental model and vocabulary before the rest of the series goes into algorithm internals. The next post, Designing environments for RL, is about formalising your problem. The three after that are about the algorithms that operate on it.

If you’re already comfortable with the vocabulary, skip this. It’s the orientation, not the substance.

What RL is, in one sentence

Reinforcement learning is the family of algorithms that learn what to do in a system by trying things and observing what happens.

That’s it. Everything else is implementation detail.

Three other ML families help frame what RL is by saying what it isn’t:

• Supervised learning has labels. You have (input, correct_output) pairs and you fit a function that predicts output from input. Classification, regression, image labelling, most of NLP. The signal is “here’s what the answer should be.”
• Unsupervised learning has no labels. You have inputs and you find structure — clusters, embeddings, generative models. The signal is “here are some examples, figure out the regularities.”
• Bandits are RL’s stateless cousin. You pick an action, you get a reward, you pick another action. There’s no state that evolves; nothing the action does affects future opportunities. A/B testing is the canonical example.

RL is what you reach for when the system responds to your actions and what you do now changes what’s available later. The reward signal is the only feedback you have, and it’s often sparse, delayed, and silent on whether your specific action was the right one.

The contrast with supervised learning is the one worth holding onto, because it’s the framing most engineers arrive with:

The jump from “optimise a prediction” to “optimise an outcome” is the one that reshapes everything downstream — it’s why RL needs states, transitions, and discounting at all, and why the rest of this series spends most of its time on the environment rather than the algorithm.

If your problem has labels, use supervised learning. If it has no state, use bandits. If the dynamics are well-understood and modellable, use classical control. RL is for the messy middle: a system that responds to actions, no labels for individual decisions, dynamics you don’t fully understand.

The vocabulary, on one example

Most RL writing introduces vocabulary in the abstract. We’ll do it on a concrete example — email send-time optimisation — and watch each piece appear naturally.

You’re trying to learn, per user, the time of day to send a message that maximises opens.

• The agent is your scheduling code. It picks send times. It’s the thing that’s learning.
• The environment is everything else — the user, their inbox, the way they interact with email. The agent doesn’t see the environment directly; it only sees what the environment chooses to expose.
• A state s is everything about the situation the agent gets to look at when making a decision. Here it might be (user_id, last_open_recency, segment, day_of_week, message_type).
• An action a is the choice the agent makes from a state. For send-time optimisation: a discrete send-time slot, like “now” or “in 4 hours” or “tomorrow at 9am.”
• A transition is what happens after the action: the user opens (or doesn’t), some time passes, the system arrives at a new state.
• A reward r is the scalar feedback the agent gets after each action. Maybe +1 if the user opens within 24 hours, -50 if they unsubscribe, 0 otherwise. The agent’s whole job is to learn behaviours that produce more reward.
• A policy π is the agent’s strategy — a function from states to actions. After training, the agent’s policy is “for this user state, send at this time.”
• An episode is one full trajectory through the system. For send-time optimisation, an episode might be “one campaign per user” — agent makes a decision, user responds, episode ends.
• An MDP — Markov Decision Process — is the formal name for the bundle of (states, actions, transitions, rewards, episode structure). It’s the input contract every RL algorithm expects.
• The discount factor γ is a number between 0 and 1 that controls how much the agent values future reward versus immediate reward. γ = 0.9 says a reward ten steps from now is worth 0.9¹⁰ ≈ 0.35 of an immediate reward. It exists for mathematical reasons (so infinite-horizon sums converge) but it also has a real engineering meaning — how far ahead the agent plans.

Two more terms you’ll meet often:

• A value function V(s) is the expected total discounted reward you’d accumulate starting from state s and acting according to your current policy. It answers “how good is this situation, given how I currently know how to behave?”
• An action-value function Q(s, a) is the same idea but for state-action pairs. “How good is it to take this specific action from this specific situation, then act optimally?” Q is what lets the agent choose actions; V only tells it how good situations are.

That’s the entire core vocabulary. Everything else is built on these ten or so words.

The Bellman idea, conceptually

The one equation that runs through everything in RL is the Bellman equation. In its simplest form, it says:

The value of a state equals the immediate reward you get from it, plus the discounted value of whatever state you end up in next.

Symbolically: V(s) = R(s) + γ · V(s’).

That’s not an algorithm. It’s a consistency relationship. It says: once we know the right value for every state, the values have to be consistent with each other in this way. Solving for them is the job of the algorithms — value iteration, Q-learning, deep RL — and each one is a different way of satisfying this constraint.

The third post in this series, Inside the Bellman loop, is about how that satisfaction actually happens at runtime. The headline finding, in advance: it’s much more like spreadsheet recalculation than recursive tree search.

When to reach for RL — and when not to

A few rules of thumb worth knowing before you commit to RL on a real problem.

Reach for RL when:

• The system responds to actions in ways that change future opportunities.
• You have a reward signal — something measurable that you’d like the agent to maximise.
• You can either tolerate exploration cost in production, build a simulator, or learn from logs.
• The problem genuinely benefits from sequential decision-making, not just one-shot picks.

Reach for something else when:

• You have labels for individual decisions. That’s supervised learning, and it’ll work better and converge faster.
• There’s no state — your decisions are stateless. That’s a bandit problem, and bandits have stronger theoretical guarantees than RL.
• The dynamics are well-modelled and you can solve the optimal-control problem analytically or with a planner. Classical control will be more reliable.
• You can’t define a scalar reward. Multi-objective optimisation, preference learning, or human-in-the-loop systems are probably what you want.
• Exploration is catastrophic and you can’t simulate. Medical dosing, financial trading without a sandbox, anything where one bad action is unrecoverable. Bandit-with-safety-constraints is a better starting point than RL.

The honest summary: RL is overused. A lot of problems that get reached for with RL would be better served by bandits, supervised learning, or hand-rolled heuristics. The cases where it genuinely shines are sequential, state-rich, and have a clean reward signal — recommendation, routing, scheduling, games, robotics, agentic LLM workflows.

The series

Five posts, in reading order:

1. An introduction to reinforcement learning (this post) — the orientation. What RL is, the vocabulary, when to reach for it.
2. Designing environments for RL — the engineering discipline. How to formalise your problem as an MDP: state representation, action discretisation, reward design, sample sources.
3. Inside the Bellman loop — the value iteration algorithm. Why it looks recursive but acts like graph relaxation.
4. Inside Q-learning — the move from value estimation to action selection. Why argmax over a Q-table is a policy for free.
5. Inside deep RL — what happens when the table is too big to fit in memory. Function approximation, target networks, replay buffers, the deadly triad.

If you’re an engineer who wants to apply RL to a real system, the most important reading order is 1 → 2 → 3 → 4 → 5. Posts 1 and 2 are the bits most introductions skip and the bits that determine whether your project ships.

If you already know the vocabulary, skip to post 2. If you already know the vocabulary and how to design environments, the algorithm posts stand on their own.