Notes
🔍

Teacher Forcing

Imagine teaching a model to write a sequence one step at a time. During practice, you can show it the correct previous step before asking for the next one. Teacher forcing does exactly this: while training a recurrent network, it feeds the true previous output from the training data into the next time step, rather than feeding the model’s own prediction.

How the training loop works
For a target sequence such as <start> A B C <end>, a decoder RNN, LSTM, or GRU is trained like this:

  • Given <start>, predict A.
  • Feed the true A as the next input, then predict B.
  • Feed the true B, then predict C.

The loss compares every prediction with its known target, and backpropagation through time adjusts the shared network weights. Because each step receives a correct context, the model learns the intended next-step relationship instead of immediately being thrown off by an early mistake. This produces stable, efficient training and lets all target tokens supply direct learning signals.

The training–inference gap
At inference time, the true previous output is unavailable. The model must feed back its own prediction: a wrong token at step two becomes part of the input at step three, and errors can compound. This mismatch is called exposure bias. A model can therefore achieve low teacher-forced training loss yet generate weak long sequences on its own. Scheduled sampling addresses this by gradually replacing some true previous outputs with model predictions during training, although it introduces its own optimization trade-offs. Teacher forcing remains the standard baseline because it makes sequence learning practical; its limitation is that it trains the model under cleaner conditions than it faces when generating autonomously.

Teacher forcing is a training method for autoregressive sequence models in which the network receives the true previous output as input at each step, rather than its own prior prediction. It supplies stable, correct context during learning and enables efficient parallel loss computation, but creates exposure bias: at inference, prediction errors can feed into later steps because true prior outputs are unavailable.

Imagine teaching someone to recite a sentence one word at a time. During practice, when they pause, you give them the correct next word so they can continue smoothly. Teacher forcing does something similar when training an AI that produces sequences, such as text, captions, or translations.

Instead of making the AI rely on its own previous guess while it is still learning, the trainer supplies the correct previous word from the example. This keeps practice on track and helps the AI learn each next-step choice more easily. Later, when used for real, the AI must continue from its own generated words—so mistakes can sometimes build on each other.