Tile Coding

Tile coding gives a reinforcement-learning agent a practical way to handle continuous inputs such as position, velocity, temperature, or joint angles. Rather than treating every exact state as entirely new, it groups nearby states into overlapping regions so experience can transfer locally.

Imagine laying several transparent grids over a map, with each grid shifted slightly. Each grid is a tiling, and each cell is a tile. A state activates one tile in every tiling. The resulting feature vector is sparse: nearly every entry is zero, while a small fixed number are one. A value function can then be written as a simple linear estimate, such as Q(s,a), by adding the weights of the active tiles for that state-action pair.

A single grid creates sharp boundaries: two nearly identical states on opposite sides of a cell edge share nothing. Offset tilings soften that problem. Nearby states share many active tiles; farther-apart states share few or none. When a temporal-difference update changes the weights of active tiles, it therefore improves estimates for related states too.

• In Mountain Car, states with similar position and velocity can learn from one another.
• In a robot-control task, tile coding can represent continuous sensor readings without needing a huge table for every possible measurement.
• For high-dimensional inputs, implementations commonly hash tile indices into a fixed-size memory table rather than storing every possible tile explicitly.

Tile coding sits between the environment’s observations and algorithms such as semi-gradient SARSA or Q-learning. It makes incremental updates cheap, interpretable, and relatively stable because only a few linear weights change per transition. Its main trade-off is resolution: tiles that are too large blur important distinctions, while tiles that are too small require much more experience and reduce generalisation. Unlike supervised learning with a fixed labelled dataset, the agent must still explore the regions where these features need useful estimates; tile coding shares what it learns, but cannot create evidence for actions it never tries.