Forget Gate

When a sequence keeps unfolding, a network needs a way to decide what deserves to stay in memory and what has become stale. The forget gate is the part of an LSTM that makes this decision: it softly erases selected pieces of its internal memory at every time step rather than blindly carrying everything forward.

How the gate makes its decision
An LSTM maintains a cell state, a running memory vector. Before adding new information, the forget gate examines the current input x_t and the previous hidden state h_t−1. It computes:

f_t = sigmoid(W_f · [h_(t−1), x_t] + b_f)
c_t = f_t * c_(t−1) + i_t * g_t

Each value in f_t lies between 0 and 1. A value near 1 preserves the corresponding element of the old cell state c_t−1; a value near 0 removes it; intermediate values gradually weaken it. This element-by-element control is more useful than a single “remember or forget” switch: an LSTM can retain one feature while discarding another.

Why it helps training
The forget gate creates a relatively direct route through time for both memory and gradients. If a gate stays near 1, information can persist across many steps without being repeatedly squashed by nonlinear transformations—the failure mode that makes a vanilla RNN forget distant context and suffer vanishing gradients. A common initialization gives the forget-gate bias a positive value, encouraging preservation early in training instead of immediate erasure.

What it looks like in practice
In a PyTorch nn.LSTM, the forget gate is built into each LSTM layer rather than exposed as a separate module. For a sequence with changing regimes, it can clear outdated state at a boundary while retaining useful long-term signals. Poorly learned gates cause recognizable problems:

• Gates near zero erase context too aggressively, so the model behaves as if it has only short memory.
• Gates near one retain irrelevant history, making the state slow to adapt.
• Gates that saturate at exactly 0 or 1 receive weak gradients, slowing correction.