Recurrent Neural Network (RNN)

A Recurrent Neural Network (RNN) is built for data where order matters. Rather than treating each input as isolated, it carries a compact memory of what it has already processed, allowing earlier steps in a sequence to influence later ones.

State carried through time
At time step t, an RNN combines the current input xₜ with its previous hidden state, hₜ₋₁, to produce a new state, hₜ. The same weights are reused at every step:

hₜ = activation(Wₓ xₜ + Wₕ hₜ₋₁ + b)

This repeated update lets one network process sequences of different lengths. A final hidden state can support one prediction for a whole sequence, while each state can also produce an output at every step. In PyTorch, this basic mechanism appears in torch.nn.RNN.

How it learns—and where it struggles
Training uses backpropagation through time (BPTT): the RNN is conceptually “unrolled” into a chain of copies, one per time step, and gradients flow backward through that chain. This exposes the central weakness of a vanilla RNN:

Vanishing gradients: repeated multiplication by small derivatives makes the learning signal fade before it reaches early steps. The network then cannot retain distant information.
Exploding gradients: repeated multiplication by large values makes updates unstable, causing loss curves to spike or diverge. Gradient clipping limits this damage.

Gates made recurrence practical
LSTMs and GRUs extend the RNN with gates that decide what to keep, write, and forget. Their protected memory paths allow useful gradients to travel much farther, which is why they replaced plain RNNs in many sequence systems. An RNN still costs sequential computation: step 20 cannot be computed until step 19 finishes, unlike a transformer’s parallel processing. Careful initialization, clipped gradients, and gated cells turn recurrence from a fragile short-term memory into a trainable model of evolving context.

A Recurrent Neural Network (RNN) is a neural network that processes a sequence one step at a time while carrying a hidden state forward, allowing each output to depend on current input and prior context. Its parameters are shared across steps and trained with backpropagation through time. RNNs provide a basic mechanism for learning temporal dependencies, though standard forms suffer from vanishing and exploding gradients over long sequences.

Imagine reading a sentence while keeping the earlier words in mind. When you reach “The dog chased the ball because it…,” the word “it” makes sense only because you remember what came before.

A Recurrent Neural Network (RNN) is an AI system designed for that kind of ordered information. Unlike a system that treats every item separately, an RNN carries a small, changing memory of earlier steps as it moves along. This makes it useful for things like speech, written language, music, sensor readings, and weather patterns.

Its key job is to notice how earlier events can shape what happens next. That sense of sequence is why RNNs mattered for teaching AI to handle information that unfolds over time.