Notes
🔍

Feature Scaling

When your input features live on wildly different numeric scales, a model can get “distracted” by the biggest-looking numbers. Feature scaling is the set of simple transformations that put features onto comparable ranges so learning behaves more predictably.

What feature scaling does

Many supervised models use distances, dot products, or gradient-based optimization. If one feature is “annual income” (0–200,000) and another is “number of late payments” (0–10), the large-scale feature can dominate the math even if it’s not more informative. Scaling changes the units, not the underlying information, so each feature can contribute in a balanced way.

Common scaling methods (and when they help)

  • Standardization (z-score): subtract mean, divide by standard deviation. Great default for Logistic Regression, Linear SVM, and neural nets.
  • Min–max scaling: map values into a fixed range (commonly 0 to 1). Useful when features have known bounds or for some neural network setups.
  • Robust scaling: use median and IQR to reduce the impact of outliers (handy in fraud or income data).
  • Normalization (per-sample): scale each row to unit length; common in text classification with TF-IDF and cosine similarity.

Why it matters in practice

Scaling can dramatically change training speed and quality. Without it, gradient descent can zig-zag and converge slowly; distance-based models like k-NN can become meaningless. Tree-based models (e.g., Random Forest, XGBoost) are mostly scale-insensitive, so scaling is less critical there. In scikit-learn, scaling is typically done with StandardScaler inside a Pipeline so the scaler is fit only on the training set (avoiding data leakage):

from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

model = make_pipeline(StandardScaler(), LogisticRegression())

Feature scaling is a preprocessing step that transforms numeric input features onto comparable ranges (e.g., via standardization to zero mean/unit variance or min–max normalization to a fixed interval). It matters because many supervised models and optimizers are scale-sensitive: scaling stabilizes and speeds training, prevents large-magnitude features from dominating distance or regularization terms, and improves comparability of learned coefficients and model performance.

Imagine you’re comparing two things: someone’s height in centimeters and their income in dollars. The income numbers are so much bigger that they can “drown out” height in the comparison, even if height matters. Feature scaling is like converting everything to a similar kind of measuring stick so no single input wins just because its numbers are larger.

In supervised learning, models learn from many inputs (called “features”), like age, clicks, and time spent on a page. Feature scaling adjusts these inputs to comparable ranges, which often helps the model learn more smoothly and make fairer use of all the information.