Feature Engineering for RL

In reinforcement learning, the agent does not learn from a neat table of labelled answers. It learns from the signals it receives while acting, so the way a situation is represented can make the difference between discovering a useful policy and seeing every experience as unrelated noise.

What the features do
Feature engineering for RL is the deliberate construction of informative inputs from an environment’s raw observation. Rather than asking a value function or policy to infer every useful pattern from scratch, the designer supplies features that expose quantities relevant to choosing actions. A feature vector might include distances, relative velocities, inventory levels, recent rewards, or indicators for important events. These features are then used by a function approximator, such as a linear value function: V(s) ≈ wᵀφ(s), where φ(s) is the engineered feature representation.

Why RL makes representation especially important
The right features help an agent generalise: experience in one state can improve decisions in similar states it has not visited. They also help with delayed reward by making the parts of the past that matter visible in the current representation. For example, a navigation agent given only its camera image might struggle to learn whether it is moving toward the goal. Adding goal-relative direction and distance gives a compact signal that a linear method, tile coding, or a small neural network can exploit.

Useful designs—and costly mistakes
Common RL features include:

relative positions and speeds for control tasks;
action history or timers when one observation hides relevant context;
scaled, bounded quantities so updates remain numerically well behaved;
interaction features, such as “near obstacle and moving fast,” when risk depends on combinations.

Poor features can create a false version of the environment. If two situations demand different actions but map to the same feature vector, the agent cannot reliably distinguish them; value estimates become contradictory. In a Gymnasium or MuJoCo task, this can produce a policy that scores well in training yet fails after a small change in starting position or dynamics. Deep methods such as DQN and PPO reduce manual feature design, but they do not remove the need for observations that preserve the information needed to act.

Feature Engineering for RL is the design or selection of informative state and action features used by a value function, policy, or model to generalize beyond individual experiences. Features encode task-relevant structure—such as positions, distances, or action-history summaries—into a form the learner can use. It matters because feature quality determines what an approximator can represent, strongly affecting sample efficiency, learning stability, and final control performance.

Imagine teaching a dog to catch a ball. It helps if the dog can notice useful clues: where the ball is, how fast it is moving, and how far away it is. Those clues make the learning problem much easier than asking it to make sense of every tiny detail at once.

Feature engineering for RL means choosing or creating those useful clues for a decision-making AI. Instead of feeding it a raw flood of information, a designer highlights patterns that matter for choosing a good action—such as a robot’s distance from an obstacle or a game player’s remaining health. Good features help the AI learn from rewards faster and make better decisions in unfamiliar but similar situations.