Can we trust a perfect fit?

The price of fitting the data · Bias, variance, and the art of learning less

Can a machine learning model learn too much? A model can match its training data extremely well and still make worse predictions on new observations. We will see why, then explore how Ridge, LASSO, and Elastic Net trade fit for stability.

1. When a close fit misses the point

Last time, we saw that an imputed value is a prediction, not a recovered observation. The same caution applies to a model fitted to a complete dataset: close agreement with observed values does not guarantee that it has learned a relationship that will persist.

The plot below is a simulation. Its hidden dashed curve is known to us because we generated the data. In an actual application, the underlying relationship is unknown.

First, reward the best fit

Which model follows these observations most closely?

For now, look only at the training points. Select the curve that best reproduces the data you have already seen.

Synthetic training sample, n = 20 · select a model to see its fit.

The training sample rewards a close fit. The scientific question is whether the fitted relationship will hold beyond those points.

The degree-12 curve can follow many irregularities in the sample. Those irregularities may be noise. If they do not recur, reproducing them will not help the next prediction.

This is the risk of overfitting: adapting to sample-specific fluctuations as if they were stable patterns.

Training fit measures how closely a model predicts the observations it used for fitting. Generalization is how well it predicts observations it has not seen. These goals are related, but they are not the same.

2. A fitted model depends on its sample

Imagine repeating the same data-generating process and fitting the same model each time. The population relationship stays fixed; the particular sample changes.

Same population · new sample

Do these fitted models change when the sample changes?

Each small panel shows a fit from a different sample drawn from the same simulated population. Switch model families to compare their stability.

Sample 01

Sample 02

Sample 03

Sample 04

Sample 05

Sample 06

Sample 07

Sample 08

Relatively stable. The fits vary, but stay near one another and the shared signal.

Eight samples · n = 20 per sample · simulated illustration. Each mini-plot uses the same scale.

If small changes to the training sample lead to large changes in predictions, the model is unstable. This sensitivity to the sample is variance. Greater flexibility can reduce systematic error, yet make estimates more responsive to random fluctuations.

3. Bias, variance, and prediction error

Assume an outcome is generated by Y = f(X) + ε, where f(X) is the underlying relationship and the new observation’s noise has conditional mean zero and variance σ². A fitted prediction f̂_D(x) depends on training sample D. For squared error at a fixed x:

Pointwise squared-error decomposition

$$E_{D,\mathrm{Y|x}}[{Y-{\hat{f}}_D(x)}^2]={E_D{\hat{f}}_D(x)-f(x)}^2+{\mathrm{Var}}_D{\hat{f}}_D(x)+{\sigma }^2$$

The expectation averages over possible training samples and a new outcome. This identity assumes homoscedastic irreducible noise, independent of the training sample.

• Squared bias: how far the average prediction is from the underlying relationship.
• Variance: how much the prediction changes across training samples.
• Irreducible noise: outcome variation that remains even if the relationship is known.

The last term cannot be eliminated by choosing a more flexible model. An unbiased estimator is not automatically the best predictor either: if its variance is high, a little bias may reduce total prediction error. The Gauss–Markov theorem calls OLS BLUE under its assumptions—best among linear unbiased estimators of the regression coefficients. That is a different target from minimizing expected prediction error.

A balance, not a magic setting

Which error is more visible here?

Choose a model description to see how bias and variance can move in opposite directions. The indicators are qualitative, not measurements.

Squared bias moderate

Variance moderate

Irreducible noise still present

The model can capture the broad pattern while avoiding some sample-specific variation. This is a useful trade-off to seek, not a guarantee of best performance.

Expected prediction errorBias² + Variance + Noise

A little bias may buy a lot of stability. The actual trade-off depends on the data and model family.

These indicators are a useful conceptual picture, not measurements or a universal law. The balance depends on the data, model family, and prediction task.

4. When ordinary least squares becomes unstable

Ordinary least squares chooses coefficients to minimize the sum of squared residuals. Under the classical linear model with full column rank and homoscedastic errors, its conditional coefficient variance is:

Ordinary least squares

$${\hat{\beta }}^{\mathrm{OLS}}=\underset{\beta }{\arg min}{{\Vert y-X\beta \Vert }_2}^2$$

OLS minimizes squared residuals. When XᵀX is nearly singular, some coefficient directions can have high sampling variance.

OLS coefficient variance

$$\mathrm{Var}({\hat{\beta }}^{\mathrm{OLS}}|X)={\sigma }^2({XᵀX}^{\mathrm{-1}})$$

This is the conditional coefficient variance under the classical linear model with homoscedastic errors.

For example, age and years of professional experience may carry overlapping information about income. Multicollinearity means that columns of the design matrix are close to a linear dependence; it is not simply that two columns contain similarly sized numbers. Small sample changes can then produce large changes in the separate coefficient estimates, even when their combined predictions are less affected.

5. Change the objective: regularization

Instead of minimizing training error alone, regularization adds a penalty on the coefficients:

Fit plus penalty

$$\hat{\beta }=\underset{\mathrm{\beta ₀},\beta }{\arg min}[\frac{1}{2n}{{\Vert y-\mathrm{\beta ₀}1-\mathrm{X\beta }\Vert }_2}^2+\lambda P(\beta )]$$

λ ≥ 0 controls penalty strength. The intercept is not penalized; predictors should be on comparable scales before coefficient penalties are applied.

At λ = 0, this returns the unpenalized least-squares objective. Increasing λ gives coefficient control more influence. The point is not to make training fit as poor as possible; it is to accept some bias when the gain in stability may improve predictions.

Ridge · L2

$$P(\beta )=\sum _{\mathrm{j=1}}^p{{\beta }_j}^2$$

Shrink coefficients continuously toward zero.

LASSO · L1

$$P(\beta )=\sum _{\mathrm{j=1}}^p|{\beta }_j|$$

Shrink coefficients; some can become exactly zero.

Elastic Net

$$\begin{array}{c}P(\beta )=\alpha \sum _{\mathrm{j=1}}^p|{\beta }_j|\\ +\frac{1-\alpha }{2}\sum _{\mathrm{j=1}}^p{{\beta }_j}^2\end{array}$$

Combine sparsity with an L2 component for correlated features.

Here 0 ≤ α ≤ 1; conventions for penalty scaling vary across texts and software. Ridge is continuous shrinkage. LASSO can also perform variable selection—but an exact zero is a property of this fitted prediction rule, not proof that a variable is irrelevant or has no causal effect.

Robert Tibshirani introduced LASSO as a way to combine shrinkage with variable selection, comparing it with Ridge and subset selection. The question was whether one estimation procedure could stabilize coefficients and produce a sparse model. (Tibshirani, 1996)

Why does LASSO make exact zeros possible?

The L2 constraint used by Ridge is circular in two dimensions. The L1 constraint used by LASSO is diamond-shaped, with corners on the coordinate axes. When a least-squares error contour first touches an L1 corner, one coefficient is zero.

The geometry of sparsity

Why can LASSO land exactly on zero?

Both methods limit coefficient size. Select a shape to see why an L1 constraint can produce exact zeros.

The contour touches the circle’s boundary. Here the constraint is active, so both solutions lie on its boundary. Ridge’s smooth circle shrinks both coefficients; the LASSO corner lies on an axis, making β₂ exactly zero.

Illustrative geometry, not a fitted dataset. If the constraint is loose enough to contain the OLS solution, it need not bind and the estimate can lie inside the boundary.

Under centered, standardized, orthogonal predictors, LASSO’s one-coefficient solution is soft thresholding. If z_j is the corresponding OLS coefficient, the positive-part operation subtracts the penalty and clips negative results to zero. The orthogonal design matters: this closed form is not a general formula for correlated predictors.

One coefficient · a precise result

Try a coefficient of your own

Set the unregularized coefficient and penalty. See Ridge shrink it continuously while LASSO can switch it off.

OLS coefficient z

0.10

Penalty λ

0.20

Ridge

0.07

A nonzero coefficient gets smaller, but usually stays nonzero.

LASSO

0.00

Below the threshold: this coefficient is exactly zero.

The closed forms below assume centered, standardized predictors with an orthogonal design, and use the lecture’s penalty convention.

Ridge

$${\hat{\beta }}_j=\frac{{{\hat{\beta }}^{\mathrm{OLS}}}_j}{1+2\lambda }$$

LASSO

$${\hat{\beta }}_j=\mathrm{sign}(z_j){(|z_j|-\lambda )}_{+}$$

The sharp corner in the geometry and the flat region in the coefficient map are two views of the same L1 property. Ridge continuously shrinks a nonzero coefficient; LASSO can map a whole interval of small coefficients to exactly zero.

6. Selection is not the same as importance

Suppose age and professional experience are correlated, and LASSO keeps experience but drops age. That does not establish that age has no relationship with income. The predictors can compete to represent overlapping information, and selection can change with the sample or penalty strength.

Prediction, statistical interpretation, and causal explanation are different goals. A zero LASSO coefficient says that this fitted linear prediction does not directly use that feature. It does not establish the absence of a causal effect.

Regularization is not a cure-all. Too much penalty can discard useful signal; Ridge does not repair a misspecified relationship; and Elastic Net does not guarantee recovery of the “correct” variables. Whether a penalty improves prediction has to be evaluated on data not used for fitting.

7. What to carry forward

• Training fit alone cannot tell us whether a model will generalize.
• A model is a random object: its fitted predictions depend on the sample.
• Prediction error reflects bias, variance, and irreducible noise.
• Ridge shrinks coefficients; LASSO can set coefficients exactly to zero; Elastic Net combines L1 and L2 penalties.
• More complexity or more regularization is not automatically better. Predictive reliability must be assessed.

A little bias may buy a lot of stability. But how much penalty is appropriate, and how reliable is an error estimate from one particular split? Those are questions for evaluating models across samples.

Pause and consider

How would you tell whether a model has learned a relationship—or only a coincidence in its training sample?

References

Hoerl & Kennard (1970)

“Ridge Regression: Biased Estimation for Nonorthogonal Problems.” Technometrics, 12(1), 55–67.

Tibshirani (1996)

“Regression Shrinkage and Selection via the Lasso.” Journal of the Royal Statistical Society: Series B, 58(1), 267–288.

Zou & Hastie (2005)

“Regularization and Variable Selection via the Elastic Net.” Journal of the Royal Statistical Society: Series B, 67(2), 301–320.

Hastie, Tibshirani & Friedman (2009)

The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd edition. Springer.