Designing environments for RL

This is the second post in the Inside RL series. The first post is the conceptual orientation — what RL is, when to reach for it, the vocabulary. This one is the engineering discipline that sits between “we should use RL for this” and “OK, now we can pick the algorithm.”

Most RL writing skips it. Pick the algorithm, tune it, ship it. The implicit assumption is that the environment — the formalisation of your problem into states, actions, rewards, and transitions — is somehow given. In practice it almost never is. You have a queue with a backlog, or a user with an inbox, or a cache with a fill rate, and somebody is asking you to apply reinforcement learning to it. The work of converting what you have into something the algorithm can read is most of the project, and it’s the part that decides whether the algorithm works or quietly produces garbage.

Algorithms are commodities. Well-formed environments aren’t.

The MDP as a target

The algorithms downstream — value iteration, Q-learning, deep RL — all consume the same thing: a Markov Decision Process, or MDP. Five formal parts:

• State s. The information you give the agent at each step. Has to satisfy the Markov property — see the next section.
• Action a. The choice the agent makes from state s.
• Transition P(s’ | s, a). The probability distribution over next states given the current state and action. May be deterministic.
• Reward R(s, a) or R(s, a, s’). The scalar feedback the agent receives for taking a from s.
• Discount γ ∈ [0, 1). How much future reward matters relative to immediate reward.

Plus an episode structure: when does the agent start? When does a trajectory end? Or is the task continuing?

In code, this collapses behind a small interface. The de facto contract is the Gymnasium API (formerly OpenAI Gym):

class Environment:
    def reset(self) -> Observation: ...
    def step(self, action: Action) -> tuple[
        Observation, Reward, terminated: bool, truncated: bool, info: dict,
    ]: ...

Five parts of design effort, two methods of code. The disproportion is what trips engineers up — most of the work isn’t in the implementation, it’s in the decisions baked into what those methods return. The rest of this post is about those decisions.

State representation

This is where most RL projects die quietly.

The Markov property says: the future depends only on the current state, not on the history that produced it. The catch is that the Markov property is not a fact about your problem — it’s a requirement on your state representation. You have to design state such that the property holds, by including everything that materially affects future dynamics or rewards.

When you don’t, the algorithm will be silently wrong in proportion to how much the hidden information matters. Some examples of getting it wrong:

• Email send-time optimisation. If day-of-week affects open rates and you only feed in (user_id, recency), you’re hiding a load-bearing variable. The agent will learn an averaged policy that’s wrong for weekends and holidays.
• Pricing. If inventory level affects optimal price (low stock → raise price), and you don’t include inventory in state, the agent can’t learn that pattern. It will set the same price regardless of whether you’re about to stock out.
• Cache eviction. If recency-of-access affects future access probability, and you only feed in current cache contents, you’ve discarded the signal that makes the problem solvable.

The diagnostic question to ask of every potential state variable: would knowing X change which action is best? If yes, X is part of state. If no, leave it out.

There are two ways to get state wrong other than just omission:

State aliasing. Multiple distinct world-situations collapse into the same observed state. The agent can’t tell them apart, and its policy has to be a compromise. Common when you use too few features or too coarse a discretisation — three customer segments when ten would be needed, hourly bins on a phenomenon that varies by the minute.

Partial observability. Sometimes you genuinely can’t see all the relevant variables — the user’s mood, the competitor’s next move, the upstream system’s queue depth. The honest framing is that you have a POMDP (partially observable MDP), not an MDP. The cleanest fix is to use a history of recent observations as state — frame-stacking in Atari games, a sliding window of recent events for a queue, the last N tool calls for a coding agent. Recurrent policies (LSTM or Transformer over observation history) are the deep RL response.

The general advice: err on the side of including too much in state and let the algorithm figure out what matters. Modern function approximators are good at ignoring irrelevant features. They are not good at recovering missing ones.

Action space

Three design decisions, in roughly decreasing order of consequence.

Discrete vs continuous. This is the single most consequential choice in environment design because it determines which algorithm families apply. Discrete action spaces let you use Q-learning and its descendants (DQN, Rainbow) — the argmax over a finite set is cheap. Continuous action spaces force you into policy-based methods (DDPG, SAC, TD3, PPO) because there’s no closed-form argmax over a function. If you’re not sure, start discrete; you can almost always discretise a continuous action space into a small number of useful levels.

Action set size. Q-learning’s table has |S| × |A| cells. Deep Q-networks output |A| values per forward pass. PPO’s policy network outputs a distribution over |A|. All of them scale linearly or worse with action-set size. If you have a thousand possible actions, you’ve made every algorithm slow and most of them statistically inefficient.

Action masking. Real systems often have actions that are illegal in certain states — “you can’t apply this discount if the user isn’t in the loyalty program,” “you can’t evict this cache entry because it’s pinned,” “you can’t bid above your remaining budget.” The cleanest way to express this isn’t a penalty in the reward (that teaches the agent to avoid the illegal action slowly, while burning steps on it). It’s a mask applied at policy time: zero out the probability of illegal actions before the agent samples. Both Q-learning (argmax over the legal subset) and PPO (renormalised distribution over the legal subset) support this trivially, and most production RL uses it.

When the action space is genuinely too big to enumerate — picking one of ten thousand products to recommend, choosing from millions of bid landscapes — the standard workaround is hierarchical actions: split one big decision into a sequence of smaller ones. “Which user segment, then which template, then which time-slot” instead of “which one of N campaigns.” Hierarchical RL is its own subfield; for production purposes, a manually-designed hierarchy usually works fine.

Reward design

This is the part that gets RL projects shut down.

The Bellman equation does arithmetic on the reward signal. Every assumption you bake into how reward is computed gets amplified by the algorithm into a policy. Get the reward wrong and the algorithm will faithfully optimise the wrong thing.

Four principles, in order of importance.

Scalar requirement. The Bellman update sums and discounts rewards into a single number per state. If you have multiple objectives — “maximise opens AND revenue AND user satisfaction” — you have to collapse them into one scalar somehow. The options are: a weighted sum (you pick the weights, which is just hiding the trade-off in the weight choice), constrained optimisation (one objective, others as constraints — algorithmically harder), or multi-objective RL (returns Pareto fronts rather than single policies; rarely used in production). Most engineers eventually pick the weighted sum and iterate on the weights. That’s fine. Just notice that you’ve made a value judgement in the reward function.

Dense vs sparse. A dense reward gives signal at every step — “did the user open this email? +1 if yes, 0 if no.” A sparse reward gives signal only at episode end — “did the user convert after the 7-touch campaign?” Dense rewards are easier to learn from because there’s signal on every transition. Sparse rewards are often closer to what you actually care about. The standard tactic is reward shaping — adding small intermediate rewards for behaviours you believe to be useful, even when they don’t directly contribute to the final outcome.

Reward shaping has a well-known footgun: if the shaping reward isn’t potential-based (in the technical sense from Ng, Harada & Russell 1999), it can change the optimal policy. The agent will faithfully optimise the shaped reward, which may not match the true reward you cared about. In practice this means: pick shaping rewards carefully, and treat shaping as a tool you’ll have to remove and validate against periodically.

Reward hacking, or Goodhart’s law for RL. “When a measure becomes a target, it ceases to be a good measure.” If you reward an agent for “user opens email,” it’ll optimise for opens — including by sending more emails (annoying), sending at unsociable hours when curiosity peaks (annoying), or learning clickbait subject lines that have nothing to do with content (lawsuit). Every reward function has a corresponding exploit. The designer’s job is to think about the exploit before the agent finds it.

The mitigation is usually one or more of: a per-episode budget (no more than N sends per user per week), penalty terms on suspicious behaviour (negative reward for unsubscribes), or human-in-the-loop validation of agent decisions during training. DeepMind keeps a running catalog of reward hacks caught in the wild; it’s worth a read before designing any non-trivial reward function.

Inverse RL. Sometimes you genuinely can’t write the reward function — you’d know a good policy when you saw one, but can’t write down what makes it good. Inverse reinforcement learning (Ng & Russell 2000, Ziebart et al. 2008) recovers a reward function from demonstrations of good behaviour. It’s the escape hatch when reward design is intractable. RLHF — the thing that powers ChatGPT, Claude, and Gemini — is a descendant of this approach: the reward function is learned from human preference comparisons rather than written down.

Three ways environments fail

Each of the three axes above — state, action, reward — has a signature failure mode. They’re worth seeing side by side, because in practice they present identically: the algorithm runs fine, the agent learns something, and the something is wrong in a way no amount of hyperparameter tuning will fix.

Missing state — the agent can’t know better. A delivery robot whose state omits battery level can’t learn energy-efficient behaviour; it has no variable that tells it when conserving charge matters, so it acts as if charge were infinite. This is the state-representation failure from earlier, restated as a symptom: anything that changes the optimal action and isn’t in state produces a policy quietly averaged over the values you hid.

Missing actions — the agent can’t act better. A trading agent with only Buy and Sell and no Hold cannot express “do nothing right now” — frequently the optimal move. No amount of training recovers an action that isn’t in the space. The same ceiling catches a recruitment-outreach agent with no “wait and re-engage later” action, or a pricing agent that can only move in £5 steps when the right move is £2. The action space is a hard upper bound on the policies the agent can even represent.

Bad rewards — the agent has no reason to do better. Reward a recruitment agent for meetings booked rather than placements made and it learns to book meetings that never convert: high-volume, low-quality outreach, which is to say spam. This is reward hacking from the previous section, and the lesson is the same — the reward defines the objective literally, and the gap between the number you reward and the outcome you want is exactly the gap the agent will find and exploit.

The unifying diagnostic, when the agent misbehaves: ask which of the three it is. Did it lack the information to know better (state), the vocabulary to act better (action), or the incentive to do better (reward)? Almost every “the algorithm is broken” report is really one of those three.

Episodes vs continuing tasks

Two design choices: when does an episode end, and is there even an episode boundary at all?

Episodic tasks. Things with natural endings. A campaign concludes. A game ends in win/lose. A robot completes a pick-and-place. Episodes give you a clean place to compute returns and reset state.

Continuing tasks. Things with no natural ending. Server load balancing. Inventory management. A trading strategy. The Bellman equation handles these with γ < 1, which gives infinite-horizon sums a finite value.

In code, the Gymnasium API distinguishes termination (the episode ended for environmental reasons — agent fell off the cliff, game ended) from truncation (you cut the episode short for training-loop reasons — max steps reached). The two are computed differently because terminated episodes should treat the final reward as the true terminal value, while truncated ones should bootstrap from the value function (the episode would have continued, you just decided to stop).

This sounds like a triviality. It’s the single most common bug in production RL code. Treat truncation as termination and you teach the agent that hitting your training-loop limit is a terminal state — which it isn’t.

Where experience comes from

You can’t run any algorithm without (state, action, reward, next-state) tuples. Three sources, each with a different cost structure.

Production (online RL). The agent acts in your live system; every step is real. Realistic, but exploration costs real money. ε-greedy with ε = 0.05 means 5% of your traffic is doing what the agent currently thinks is suboptimal. Fine for a recommendation system; catastrophic for automated trading. The constraint shapes everything — most online RL deployments use a “shadow mode” first (agent’s decisions are logged but not executed) and a “constrained exploration” mode (the agent can only deviate from the current policy by a small amount).

Logs (offline / batch RL). You have historical data of (state, action_taken, reward, next_state) from whatever your system used to do. You can learn Q from it without any new exploration cost — but only over the coverage of states and actions the old policy actually visited. If you never previously sent emails at 3am, you have no signal there, and a naive Q-learning algorithm will extrapolate a confident answer with no data behind it. Modern offline RL (CQL, IQL, BCQ) explicitly penalises actions far from the data distribution. This is where most of the recent RL research lives, because production engineers really want to learn from logs.

Simulator. You build a model of how your environment responds and train the agent inside the model. Fast and safe — you can run a million episodes overnight without touching production. The catch is that the agent’s quality is bounded by the simulator’s faithfulness. The “sim-to-real gap” is the failure mode where the agent learns to exploit modelling artefacts that don’t exist in reality. Common in robotics (the simulated friction isn’t the real friction); also a concern wherever you build a demand model from history to train against. The standard fixes are domain randomisation (randomise simulator parameters during training so the agent generalises to nearby realities) and progressive sim-to-real (start in simulation, fine-tune in production with bounded exploration).

Most production deployments are some hybrid: warm-start from logged data, fine-tune with bounded online exploration, validate in a simulator first. Pure online RL on production traffic is uncommon outside of recommender systems and bandits, because the exploration cost is too high.

Validating your environment

How do you know your environment is correct before the algorithm tells you? Three techniques worth knowing:

Sanity policies. Write a deterministic, hand-coded “obviously good” policy and verify it gets high reward. Write an “obviously bad” policy and verify it gets low reward. If both score about the same, your reward function is wrong before the algorithm even starts.

Oracle comparisons. If you know the optimal policy by construction (a planning algorithm, an analytical solution, a human expert), compare your learned policy against it. Gaps point at either the algorithm or the environment — and surprisingly often, it’s the environment.

Reward hacking detection. Set up dashboards on the agent’s actual behaviour, not just its reward. Is it doing things you wouldn’t expect a sensible policy to do? Is it producing low variance in actions that should vary by context? Is it ignoring features you thought mattered? Reward hacking shows up in behaviour, not in reward — by definition, the agent is making the reward number look great.

The meta-principle is worth stating directly: most RL failures look like algorithm failures and are actually environment failures. When the agent isn’t learning, the first instinct is to switch algorithms or tune hyperparameters. The second instinct should be to check the environment.

Common shapes in production

Five concrete decompositions, briefly, so the pattern is visible across familiar systems.

Email send-time optimisation.

• State: user features (timezone, engagement history, recency, segment), message type, day-of-week.
• Action: discrete send-time options (“now”, “in 1h”, “in 4h”, “tomorrow 9am”).
• Reward: open within 24h (+1), click (+5), unsubscribe (-50).
• Episode: one campaign per user.

Dynamic pricing.

• State: product features, recent sales velocity, inventory level, competitor prices, time-of-day.
• Action: discretised price points (e.g., 20 levels in £1 increments).
• Reward: revenue per impression.
• Episode: a day, an A/B period — depends on how fast demand responds.

Cache eviction.

• State: cache contents + access pattern statistics (recency, frequency, size of each entry).
• Action: which entry to evict on insertion pressure.
• Reward: cache hit rate over the next N requests, or negative latency.
• Episode: continuing.

A/B bandit allocation.

• State: per-arm statistics so far (counts, mean reward, variance).
• Action: which arm to allocate the next user to.
• Reward: the user’s outcome on that arm.
• Episode: continuing.

Ad placement / bidding.

• State: user features, context, recent bid landscape, remaining daily budget.
• Action: bid amount (continuous) or which creative to show (discrete).
• Reward: revenue from impression or conversion.
• Episode: per-impression, or per-day budget cycle.

The pattern is identical every time. Identify what the agent needs to know about the world (state), what choices it has (actions), what scalar number you want it to maximise (reward). Get any of those wrong and the algorithm cannot save you.

Key takeaways

Three lines.

Environment design is the project. Algorithm choice is the easy part — there are six or seven you’d reach for and they’re all well-studied. State representation, action discretisation, and reward design are open-ended and project-specific, and they’re where most RL effort actually goes.

The Markov property is a design requirement. Not a fact about your world. If your state representation hides load-bearing information, the algorithm will be silently wrong. The diagnostic question: “would knowing X change which action is best?” If yes, X is part of state.

Reward hacking is the rule, not the exception. Every reward function has a corresponding exploit. The job of the designer is to find the exploit before the agent does — usually by adding constraints, budgets, or shaping terms that close the gap between what you want and what you’re literally rewarding.

If you want to go further: the Gymnasium documentation is the canonical reference for the environment contract. Ng, Harada & Russell 1999 on potential-based reward shaping is the foundational paper and remains the right thing to read before writing your first shaped reward. DeepMind’s specification-gaming catalog is a great running list of reward hacking caught in the wild, and worth re-reading every time you design a new reward function.

The next post in this series is Inside the Bellman loop — given a well-formed MDP, how the value iteration algorithm actually computes over it. The post after that, Inside Q-learning, extends value iteration to action selection. The last, Inside deep RL, is what happens when the value table is too big to fit in memory and has to become a neural network.