Q-Learning
Q-Learning is a way for an agent to build a practical answer to a repeated question: “Given where I am now, which action will lead to the best long-term outcome?” Rather than needing a map of the environment or examples of correct actions, it learns that answer from rewards encountered through trial and error.
What the Q values represent
The “Q” in Q-Learning stands for action value. For every state–action pair, the algorithm stores Q(s, a): its estimate of the total discounted reward it can earn by taking action a in state s, then behaving well afterward. In a small gridworld, this can be a table: rows are locations, columns are moves such as up, down, left, and right.
Learning from each transition
After taking an action, receiving reward r, and reaching state s′, Q-Learning updates its estimate toward:
Q(s, a) ← Q(s, a) + α [r + γ maxₐ′ Q(s′, a′) − Q(s, a)]The bracketed quantity is the temporal-difference error: the gap between the old prediction and a better one-step-ahead target. The max says, “from the next state, assume I choose the best action I currently know.” This lets a reward at the end of a route gradually propagate backward to earlier choices.
Why its off-policy design matters
Q-Learning is off-policy: it can explore with a noisy policy, such as ε-greedy action selection, while learning values for the greedy policy that always chooses the current best action. This separation is useful, but it creates responsibilities:
- Every relevant state–action pair must be explored sufficiently; otherwise an unseen shortcut cannot receive a reliable value.
- With a suitable learning-rate schedule and enough exploration in a finite tabular environment, it converges to optimal action values.
- The maximum over noisy estimates systematically overestimates attractive actions, called maximization bias. Double Q-Learning reduces this by selecting an action with one value table and evaluating it with another.
Q-learning is a model-free, off-policy reinforcement-learning algorithm that learns an action-value function, Q(s, a): the expected discounted return from taking action a in state s and then acting optimally. It updates values from observed reward and the highest estimated value of the next state. It matters because the learned Q-values directly define an optimal policy without requiring a model of the environment.
Imagine learning the quickest way home by trying different streets. Each trip teaches you something: a route that gets you home sooner feels like a win, while a traffic jam feels like a loss. Over time, you remember which choice tends to lead to the best outcome from each place along the way.
Q-Learning is an AI learning approach built on that idea. It helps a program learn which action is likely to pay off in each situation, simply by trying actions and seeing what happens next. It does not need someone to supply the correct move beforehand. This makes it useful for tasks like game-playing, robot navigation, and choosing actions when the best results may only appear later.