Bidirectional RNN

A Bidirectional RNN reads a sequence in two directions: forward from the beginning and backward from the end. This lets the representation at each position use both what came before it and what comes after it—useful when the meaning of a piece of a sequence depends on its surrounding context.

Two views of the same sequence
A standard recurrent neural network maintains a hidden state while moving left to right. At time t, it combines the current input with a summary of earlier inputs. A bidirectional version runs two separate recurrent layers:

a forward RNN, producing a state based on positions 1 through t;
a backward RNN, producing a state based on positions t through the end.

The two states are usually concatenated, forming a richer vector for that position. In practice, the recurrent units are commonly LSTMs or GRUs, rather than simple RNN cells, because their gates preserve useful information and reduce vanishing-gradient problems during backpropagation through time.

Why future context changes the result
Consider a token whose role is unclear until later in a sequence. A forward-only RNN has to decide using the prefix it has already seen. A BiLSTM can combine the prefix with evidence from the suffix, making its per-position predictions more informed. Frameworks expose this directly: PyTorch’s torch.nn.LSTM supports bidirectional=True, which doubles the output feature size when forward and backward states are concatenated.

Training and practical limits
Both directions are trained jointly from the same loss, so gradients flow through two recurrent chains. This improves contextual representations but roughly doubles recurrent computation and activation memory. It also prevents true step-by-step streaming: the backward state requires the future sequence. Variable-length batches need padding masks or packed sequences so padding does not become fake context. For tasks where a prediction must be made immediately—such as live control or next-step forecasting—a forward-only RNN is the appropriate design; a bidirectional RNN would leak information unavailable at prediction time.

A Bidirectional RNN processes a sequence in both forward and reverse directions using separate recurrent layers, then combines their hidden states at each position. Each output can therefore use context from both earlier and later elements in the sequence. This improves representations when the full sequence is available, but prevents strictly causal, step-by-step prediction because future inputs are required.

Imagine reading a sentence with one eye on the words already seen and the other on the words still to come. A Bidirectional RNN does something similar. It looks at a sequence, such as a sentence, in both directions: from beginning to end and from end to beginning.

This matters because a word’s meaning often depends on its neighbors. In “The bank was beside the river,” the later words clarify that “bank” is not a financial institution. By using context from both sides, a bidirectional RNN can make better sense of language, speech, or other ordered information when the full sequence is available.