Notes
🔍

k-Nearest Neighbors (k-NN)

Imagine trying to guess something about a new case by looking at the most similar cases you’ve seen before. That “learn from your closest examples” idea is exactly what k-Nearest Neighbors (k-NN) does.

How k-NN makes predictions

k-NN is an instance-based, non-parametric method: it doesn’t fit a neat equation during training. Instead, it stores the training data and waits until prediction time. For a new input, it:

  • Computes a distance (commonly Euclidean) from the new point to every training point.
  • Selects the k nearest points (the “neighbors”).
  • Predicts using those neighbors:
    • Classification: majority vote (often with distance weighting so closer neighbors count more).
    • Regression: average (or weighted average) of neighbor targets.

Why details like k and scaling matter

The choice of k controls the bias–variance tradeoff: small k can chase noise; large k can blur important local patterns. Because k-NN relies on distance, feature scaling is crucial—otherwise one large-scale feature (like income) can dominate another (like age). In practice you nearly always pair it with standardization (e.g., scikit-learn’s StandardScaler in a pipeline).

Where you’ll see it in real tasks

  • Spam detection: classify an email by the labels of similar emails in feature space.
  • Credit risk: predict default risk by comparing to “nearby” past applicants.
  • House prices: estimate price from similar homes’ sale prices (k-NN regression).

In scikit-learn, you’ll encounter KNeighborsClassifier and KNeighborsRegressor, where you set n_neighbors, the metric, and optional weights.

k-Nearest Neighbors (k-NN) is a non-parametric, instance-based supervised learning method that predicts an output for a new example by finding the k closest labeled training points in feature space and aggregating their targets (majority vote for classification, mean for regression). It matters because it provides a strong, simple baseline and makes performance directly dependent on distance choice, feature scaling, and local data density.

Imagine you move to a new neighborhood and want to guess what a house is worth. You’d look at a few nearby houses that are most similar—same size, same area, similar condition—and use their prices as a guide. That’s the basic idea behind k-Nearest Neighbors (k-NN).

In supervised learning, k-NN makes a prediction by finding the “k” most similar examples it has already seen (its nearest neighbors). If it’s sorting things into categories—like spam vs. not spam—it goes with the most common label among those neighbors. If it’s predicting a number—like a delivery time—it uses the neighbors’ values as a sensible estimate.