Maximum Likelihood Estimation (MLE)
Maximum Likelihood Estimation (MLE) is a way to pick model settings so your observed data looks as “unsurprising” as possible under that model. Think of it as tuning the knobs of a probability story until the story best matches what actually happened.
What MLE is doing
Suppose you choose a statistical model with unknown parameters (like a mean, variance, or regression weights). The model says how likely different datasets are. The likelihood is that probability, viewed as a function of the parameters given the data. MLE chooses the parameter values that maximize the likelihood—the values under which the observed data would be most probable.
A concrete example
Imagine exam scores that you model as Normally distributed with unknown mean μ and variance σ². MLE gives:
- μ̂ = the sample mean (the average score)
- σ̂² = the average squared deviation from μ̂ (note: this uses 1/n, not 1/(n−1))
Why it matters in machine learning
Many training objectives are MLE in disguise. When you minimize negative log-likelihood, you’re doing MLE:
- Linear regression with Gaussian noise ⇢ minimizing mean squared error
- Logistic regression ⇢ maximizing Bernoulli likelihood ⇢ minimizing cross-entropy
- Neural networks for classification ⇢ softmax + cross-entropy is an MLE-style objective
Practical notes
MLE often works well with lots of data, but it can overfit in flexible models; adding priors leads to MAP estimation (a regularized cousin). In practice, libraries like scikit-learn and PyTorch optimize log-likelihood-based losses routinely.
Maximum Likelihood Estimation (MLE) is a parameter-fitting method that chooses the model parameters that make the observed data most probable under an assumed probabilistic model. It is central in statistical inference and widely used in ML because it yields principled, often efficient estimators and connects directly to minimizing negative log-likelihood loss. Example: fitting logistic regression by maximizing the likelihood of observed class labels given features.
Imagine you’re trying to guess what kind of loaded die someone is using. You roll it a bunch of times and see the results. The best guess is the die setup that would make your exact rolls most “reasonable” to happen.
That’s the idea behind Maximum Likelihood Estimation (MLE). In statistics and machine learning, we often assume a model has some unknown settings (called parameters), like the average and spread of heights or the weights in a prediction model. MLE picks the parameter values that make the data you observed most likely under the model.