Input Gate

An input gate is the part of an LSTM cell that decides how much new information should be written into its long-term memory at the current time step. Think of the memory cell as a notebook: the input gate controls how widely the notebook is opened before the model adds a new note.

How it works
At time step t, the LSTM examines the current input xₜ and the previous hidden state hₜ₋₁. It sends them through a learned linear transformation and a sigmoid activation:

iₜ = sigmoid(Wᵢ · [hₜ₋₁, xₜ] + bᵢ)

The resulting values lie between 0 and 1, one value per memory dimension. A value near 0 blocks a proposed update; a value near 1 permits it. The proposed content is created separately, usually with tanh, and the cell updates its memory through:

cₜ = fₜ ⊙ cₜ₋₁ + iₜ ⊙ c̃ₜ

Here, fₜ is the forget gate, c̃ₜ is candidate memory content, and ⊙ means element-wise multiplication.

Why selective writing matters
Without a gate, every input would overwrite the recurrent state indiscriminately. The input gate lets an LSTM preserve useful context while admitting genuinely relevant evidence. For example, while processing a long sequence, it can keep a feature representing an earlier important event and only update that feature when later input changes its meaning. The gate is not a hand-written rule: its weights are learned through backpropagation through time, based on whether opening or closing it improves the loss.

Training behavior
The input gate works with the forget and output gates to create a controlled path through time. This helps gradients travel through the cell state more reliably than in a vanilla RNN, reducing vanishing-gradient failures. Bad initialization or saturated sigmoid values can still cause trouble: gates stuck near zero prevent new information entering, while gates stuck near one write too aggressively. In frameworks such as PyTorch, torch.nn.LSTM computes these gates internally as a single efficient matrix operation.

In an LSTM, the input gate controls how much newly computed information is written into the cell state at each time step. It outputs values between 0 and 1, which scale the candidate memory update elementwise. By selectively admitting relevant updates and blocking noise, it helps the network preserve useful long-term information and regulate memory during training.

Imagine taking notes during a long meeting. You cannot write down every word, so you decide which new details are worth adding to your notes. An LSTM, a kind of AI designed to handle sequences such as sentences or speech, uses an input gate in a similar way.

The input gate decides how much of the newest information should be added to the network’s running memory. Useful new details can be kept; distractions or irrelevant details can be mostly ignored. This matters because the meaning of a sentence, a conversation, or a song often depends on remembering the right earlier information while making room for what has just happened.