User Engagement Metric

An offline score tells you whether a model ranked a held-out item highly. It does not tell you whether anyone came back. A user engagement metric is a quantity read off the live system — what people tapped, how long they stayed, whether they returned — chosen to stand in for value the model cannot observe directly.

What gets measured

• Click-through rate: the share of impressions acted on, cheap to measure and easy to game.
• Dwell time or completion rate: how much of the item was actually consumed, the usual guard against clickbait.
• Session depth — items consumed per visit — and return rate over the following days or weeks.
• Composite indices that fold several of these into one number so a single launch decision can be made.

Why it is the metric that decides
It is the only direct evidence that a change helped, and it is read through an A/B test or interleaving rather than from raw traffic, because engagement drifts with the day of the week and the catalogue. Aggregation is per user, not per impression, so heavy users cannot carry the result. NDCG rising offline while click-through falls online is a routine outcome, and the online number wins.

What goes wrong
Every engagement metric is a proxy, and a ranker optimises exactly what is measured. Maximise clicks and you get clickbait and popularity bias; reward dwell time and long items crowd out short ones; measure only the next session and a narrowing feed looks like a success. A primary metric needs guardrails — diversity, coverage, retention over a longer horizon — reported beside it.