BERT4Rec
Reading a user's history strictly left to right is one way to guess what comes next. BERT4Rec takes the other option: look at the whole sequence at once, and learn by filling in deliberately hidden positions.
The architecture and the training task
Items in a user's interaction sequence are embedded like tokens and passed through a stack of bidirectional transformer layers, so the representation at any position is informed by items both before and after it. Training uses the cloze objective: mask a fraction of positions at random and predict the missing items from the surrounding context. At inference a mask token is appended to the end of the sequence and the distribution predicted there becomes the next-item score over the catalogue.
What the bidirectionality buys
A left-to-right model such as SASRec sees each position only in terms of its past, which is the right assumption for a strict causal chain and a poor one for real histories full of detours, shared accounts and accidental clicks. Letting later items inform earlier ones makes the learned representation more tolerant of that noise, and masking doubles as data augmentation — one short session yields many training examples.
Why it matters, with a caveat worth carrying
BERT4Rec is the reference point for self-attentive sequential recommendation and a standard baseline in RecBole, trained on datasets such as MovieLens and Amazon Beauty. The caveat belongs with the method:
- Reported gains depend heavily on how long the model was trained — the masked objective is a slower learner than next-item prediction, and under-trained replications flatter it or SASRec by turns.
- Much of the published comparison rests on sampled metrics, where ranking against a hundred sampled negatives can reorder methods relative to full-catalogue evaluation.
- The inference mask makes serving cost scale with sequence length, which is a real constraint under a latency budget.
BERT4Rec is a sequential recommender that runs a user's interaction history through bidirectional transformer layers and trains with a masked-item (cloze) objective, predicting hidden positions from items on both sides. At serving time a mask appended to the sequence yields the next-item distribution. Seeing both directions makes the representation more robust to noisy, non-linear histories than left-to-right models, though its reported advantage is sensitive to training budget and sampled-metric evaluation.
Think of a sentence with a word blanked out. You work out the missing word from the words before it and the words after it — reading only the first half would make the puzzle much harder.
BERT4Rec learns your taste the same way. It treats the list of things you have watched or bought as a sentence, hides some entries, and trains itself to guess what was hidden using everything on both sides. Once it is good at filling gaps, the next thing you might want is just one more gap — placed at the end of the list. Looking in both directions helps because real histories are messy, full of detours and things clicked by accident.