Gated Recurrent Unit (GRU)
A Gated Recurrent Unit (GRU) is a recurrent neural-network cell built to carry useful information through a sequence without letting older signals fade away too quickly. Think of it as a small, trainable memory controller: at each step, it decides what to retain from the past and what new information deserves to replace it.
How its memory is controlled
A GRU receives the current input xt and previous hidden state ht−1, then produces a new state ht. Two learned gates, with values between 0 and 1, make that update selective:
- The update gate decides how much of the old state to preserve. A value near 1 keeps existing memory; a value near 0 replaces it with new content.
- The reset gate decides how much past state to use when constructing a candidate new memory. It can let the cell deliberately ignore stale context.
The final state blends the old state and candidate state rather than overwriting memory abruptly. Because this path can preserve information across many time steps, gradients have a clearer route backward during backpropagation through time, reducing the vanishing-gradient problem of a plain RNN.
Why the gates matter in training
For a sequence with long-range dependencies, a vanilla RNN repeatedly transforms its hidden state and can lose an early, important signal. A GRU can keep that signal through high update-gate values, then revise its state only when relevant input arrives. If gates become saturated near zero or one too early, learning can slow because their sigmoid activations have small gradients; sensible initialization, learning rates, and gradient clipping help keep training stable.
GRUs in practice
GRUs are a lighter alternative to LSTMs: an LSTM has separate cell and hidden states plus more gates, while a GRU uses one state and two gates. This means fewer parameters and less compute per step, while retaining strong sequence-memory behavior. In PyTorch, torch.nn.GRU provides efficient stacked and bidirectional GRU layers. During inference, the final hidden state can be carried into the next chunk of a long stream, preserving context without reprocessing its entire history.
A Gated Recurrent Unit (GRU) is a recurrent neural-network cell that maintains a hidden state across sequence steps using update and reset gates. These gates control how much prior information is retained and how new input changes the state. GRUs mitigate vanishing gradients and learn long-range dependencies more reliably than vanilla RNNs, while using fewer parameters than LSTMs.
Imagine someone listening to a long story while carrying a small notebook. They cannot write down every word, so they keep the details that still matter and cross out things that no longer do. A Gated Recurrent Unit (GRU) is like that notebook for an AI handling a sequence, such as a sentence, a conversation, or changing sensor readings.
As each new piece arrives, a GRU decides what to remember from earlier and what to let fade away. This helps it connect “she” with a person mentioned earlier in a sentence, or notice a pattern spread across several moments. Its “gates” are simply built-in choices about keeping, updating, or discarding information.