Long Short-Term Memory (LSTM)

Sequences have a special difficulty: what happened early can matter much later. An LSTM is a recurrent neural-network unit built to carry useful information across many time steps without letting the learning signal fade away or explode.

A memory path with controlled gates
An LSTM processes one step at a time while maintaining two internal quantities: a hidden state, which is the visible output passed onward, and a cell state, a more protected memory channel. At each step, learned gates—small neural layers using sigmoid activations—decide what to retain and reveal:

The forget gate scales old cell-state values, removing information no longer useful.
The input gate controls how much newly proposed content is written into memory.
The output gate controls which parts of the updated memory become the next hidden state.

Because the cell state has an additive update path, its gradient can travel through many steps more reliably than in a vanilla RNN, whose state is repeatedly transformed and can rapidly erase gradient information.

How it learns across time
Training uses backpropagation through time: the network is unrolled conceptually across sequence positions, and loss gradients flow backward through its recurrent connections. The gates are not hand-written rules; their weights learn when remembering helps reduce loss. For example, a model processing a long stream can learn to preserve a relevant earlier condition while discarding routine intermediate inputs. In PyTorch, torch.nn.LSTM packages these gate computations efficiently, typically receiving tensors shaped as batches of time steps and feature vectors.

Why the design still has trade-offs
LSTMs substantially reduce vanishing gradients, but they do not make unlimited memory free. Each time step computes four gated transformations, so long sequences cost sequential compute and require storing activations for training. Truncated backpropagation through time limits memory use by training on chunks, but prevents gradients from crossing chunk boundaries. Poorly chosen learning rates can still cause unstable loss, and recurrent dropout must be handled carefully: dropout is active during training but disabled at inference, and indiscriminately dropping recurrent state can damage memory. GRUs provide a lighter gated alternative; transformers remove recurrence entirely, but LSTMs remain a clear example of architecture designed around stable information and gradient flow through time.

Long Short-Term Memory (LSTM) is a gated recurrent neural-network unit that maintains a persistent cell state while selectively writing, retaining, and exposing information through input, forget, and output gates. It learns dependencies across long sequences more reliably than a vanilla RNN by controlling gradient flow through time. LSTMs matter because they reduce vanishing-gradient failures, enabling stable training when earlier sequence information must influence later predictions.

Imagine reading a sentence while keeping a few important earlier words in mind. When you reach “she,” you remember who “she” refers to; when a story reveals a twist, you may recall a clue from pages ago. A Long Short-Term Memory, or LSTM, gives an AI a similar kind of working memory.

It is useful for information that arrives in order, such as spoken words, music notes, weather readings, or steps in a video. Unlike simpler systems that quickly lose track of the past, an LSTM can hold onto useful details and let go of distractions. This helps it understand context: not just what happened now, but what earlier events make it mean.