Update Gate (GRU)

Think of a GRU as a small memory system reading a sequence one step at a time. Its update gate decides how much of the existing memory should survive and how much should be replaced by newly computed information.

How the gate controls memory

At time step t, the update gate produces a vector of values between 0 and 1:

zₜ = sigmoid(Wₓxₜ + Wₕhₜ₋₁ + b)
hₜ = zₜ ⊙ hₜ₋₁ + (1 − zₜ) ⊙ h̃ₜ

Here, hₜ₋₁ is the previous hidden state, h̃ₜ is a proposed new state, and ⊙ means element-by-element multiplication. A value of zₜ near 1 keeps that component of the old state; a value near 0 replaces it with the candidate. Because this happens separately for every hidden-unit dimension, a GRU can retain one feature while updating another.

Why it helps training

The update gate creates a direct, adjustable route from an earlier hidden state to a later one. When the gate preserves memory, gradients can also flow backward through that route with less repeated shrinking or exploding than in a plain RNN. This is crucial when an early part of a sequence must influence a later decision. Unlike an LSTM, a GRU has no separate cell state: its hidden state is both its working representation and its memory.

What it looks like in practice

For rapidly changing input, the gate can close toward 0, allowing the state to track new evidence.
For a persistent pattern, it can remain near 1, protecting information across many steps.
In PyTorch’s torch.nn.GRU, update-gate parameters are learned alongside the reset gate and candidate-state parameters through backpropagation through time.

If update gates saturate too strongly at 0 or 1 too early, learning can become rigid: the model either overwrites useful memory or refuses to revise it. The gate’s learned balance between preservation and replacement is what gives GRUs their practical ability to handle dependencies across time.

In a Gated Recurrent Unit (GRU), the update gate is a learned sigmoid gate that controls how much of the previous hidden state is retained versus replaced by the candidate new state at each time step. By selectively preserving information over long sequences, it helps regulate memory flow and reduces vanishing-gradient problems during training.

Imagine taking notes during a long conversation. Sometimes you should keep an earlier note because it is still important; other times, new information should replace it. The update gate is the part of a GRU—a network that handles sequences such as sentences, speech, or sensor readings—that makes this kind of choice.

At each moment, it helps the network decide how much of its existing “memory” to retain and how much to refresh with what it has just seen. This matters because a word near the start of a sentence, or an earlier event in a time series, may remain relevant much later. The update gate helps the network avoid forgetting useful context too quickly.