Hyperparameter

When you train a supervised model, it learns from data—but you still have to make a few “design choices” about how that learning should happen. Those choices are called hyperparameters, and they can quietly make the difference between a model that generalizes well and one that falls apart on new data.

What a hyperparameter is (and isn’t)

A hyperparameter is a setting you choose before training that controls the model’s behavior or capacity. It is different from a model parameter (like the weights in linear regression or a neural network), which is learned directly from the training data by minimizing a loss function. Hyperparameters shape the learning process: how flexible the model can be, how strongly it resists overfitting, or how optimization proceeds.

Concrete examples you’ll actually encounter

Common hyperparameters show up across popular supervised algorithms:

Regularization strength (e.g., C in scikit-learn’s LogisticRegression): controls the tradeoff between fitting the data and keeping the model simple.
Tree depth and min_samples_leaf in decision trees / random forests: limit how detailed the splits can get.
Learning rate in gradient-boosted trees or neural nets: sets the step size of optimization.
k in k-NN: decides how local the “vote” is.

Why it matters in supervised learning

Hyperparameters directly control bias–variance tradeoff. For example, in credit scoring or fraud detection, a very deep tree can memorize quirks of last month’s data (overfitting), while a shallow one can miss real patterns (underfitting). That’s why hyperparameters are tuned using validation data—commonly via cross-validation with tools like scikit-learn’s GridSearchCV or RandomizedSearchCV—to estimate how well the model will perform on unseen cases.

A hyperparameter is a model or training setting chosen before fitting that controls learning dynamics or model capacity and is not directly learned from the data (unlike parameters such as weights). Examples include learning rate, regularization strength, tree depth, and number of neighbors. Hyperparameters matter because they strongly determine generalization, stability, and compute cost; poor choices can cause underfitting, overfitting, or failed optimization.

Think of training an AI model like baking cookies. The ingredients (your data) matter, but so do the oven settings: temperature, time, and pan choice. Those settings are like hyperparameters—important knobs you choose before the learning starts.

In supervised learning, the model learns patterns from labeled examples (like emails marked “spam” or “not spam”). The model’s internal numbers adjust automatically during training, but hyperparameters are the user-chosen settings that guide how learning happens—like how fast it learns, how complex it’s allowed to be, or how strongly it avoids overfitting (memorizing instead of generalizing).