Stacked RNN

A single recurrent layer reads a sequence step by step, carrying a small working memory forward. A stacked RNN places several recurrent layers on top of one another, so each layer can refine the sequence representation produced by the layer below.

How the stack works

At time step t, the first layer receives the input vector and updates its hidden state. Its output becomes the input to the second recurrent layer at that same time step; the second layer maintains its own hidden state, and so on. The top layer supplies the model’s final per-step representation or prediction. In practice, stacks usually use LSTM or GRU cells rather than plain RNN cells, because gates preserve useful information across longer spans.

Lower layers can encode local or simple sequential patterns.
Higher layers combine those patterns into more abstract, context-aware features.
Each layer has recurrence across time, while the stack adds depth across layers.

Training and stability

Training uses backpropagation through time across both directions: backward through the stacked layers and backward across sequence steps. This creates long gradient paths. A plain, deep RNN can therefore suffer vanishing gradients, where early layers or distant time steps receive almost no learning signal, or exploding gradients, where updates become unstable. Gated cells, careful initialization, gradient clipping, and a suitable learning rate make deeper stacks practical. More layers also increase parameter count, activation memory, and training time.

Practical design

A two- or three-layer LSTM is a common starting point when one recurrent layer lacks capacity. Dropout between layers helps regularize the stack: during training, some layer-to-layer outputs are removed; during inference, all outputs are used with the learned scaling. In PyTorch, torch.nn.LSTM(input_size, hidden_size, num_layers=3, dropout=0.2) creates this arrangement. Adding layers blindly can hurt: a loss curve that plateaus suggests limited useful signal or optimization difficulty, while loss that shoots upward points to an excessive learning rate or exploding gradients. A stacked RNN earns its complexity when extra depth captures structure that one recurrent state cannot represent cleanly.

A stacked RNN is a recurrent neural network with multiple recurrent layers arranged vertically, where each layer processes the sequence of hidden representations produced by the layer below. This increases representational depth, allowing higher layers to model more abstract temporal patterns. It matters because added depth can improve sequence modeling capacity, but also increases optimization difficulty and the risk of unstable or vanishing gradients during backpropagation through time.

Imagine several readers passing a story from one to the next. The first reader notices simple details, such as individual words. The next reader uses those details to notice short phrases, and another may pick up the story’s larger meaning.

A stacked RNN is like that: several sequence-reading layers placed on top of each other. An RNN is a kind of AI network that reads information in order, while carrying a memory of what came before. Stacking layers lets it build from simple patterns to richer ones. For example, in speech, lower layers may notice sounds, while higher layers help recognize words and the meaning of a sentence.