Can we trust our data?
When information is incomplete
An incomplete dataset invites quick fixes. Before choosing one, ask what information is missing, why it is missing, and what a filled-in value would mean.
1. The missing value in the spreadsheet
A company has six customer records, but two values are missing. The manager needs the dataset in 30 minutes.
| Customer | Age | Education | Income (k€) |
|---|---|---|---|
| A | 25 | Bachelor | 35 |
| B | 42 | Master | 85 |
| C | 38 | Master | — |
| D | 51 | PhD | 95 |
| E | 29 | Bachelor | 45 |
| F | — | Master | 75 |
You have 30 minutes
A dataset is missing some customer incomes. Your manager needs it ready. What would you do?
There is no automatic fix: the consequences depend on what is missing and why.
2. Why is the information missing?
Suppose high-income customers are less likely to disclose their income. The observed incomes may then under-represent the customers whose incomes are missing. A sophisticated imputer does not automatically remove that selection bias.
The missingness mechanism matters: an estimate can only be as informative as the observations and assumptions behind it.
3. KNN imputation: who is “similar”?
K-nearest neighbours (KNN) looks for observations that are close on the features available for comparison. To impute a missing value, it uses the corresponding values from those neighbours. For a numerical target, a simple choice is their mean.
For two observations, the distance combines their differences across the features being compared:
Euclidean distance · across p features
d(xi, xj) = √∑k = 1p (xik − xjk)2
Explore the lecture’s example. Move k and watch both the neighbours and the income estimate change.
Interactive graph · K-nearest neighbours
Who is near the customer with missing income?
Customer C is 38 years old, but her income is missing. Here we use age as the only comparison feature and average the incomes of the nearest k customers.
With k = 3: nearest customers B, D, E → estimated income €40,867.
Changing k changes which observations contribute and can change the estimate. This toy graph uses age alone; real distance calculations may use several observed features.
4. Does scale matter?
Age and income are measured in very different units. In the example below, customers differ by 10 years and €100. Squared raw differences contribute 100 for age and 10,000 for income to the distance calculation.
Interactive graph · Distance and scale
Can a large number dominate similarity?
Two customers differ by 10 years of age and €100 of income. Compare the squared contributions to Euclidean distance.
The bars show each contribution relative to the largest one in the selected view. The standardized deviations are assumptions from the slide example.
Standardization expresses each difference in units of its feature’s standard deviation. This can make features more comparable for a distance calculation; it does not decide which features are useful or make the missingness mechanism disappear.
The deck’s minimal Python example standardizes the features before applying KNNImputer with three neighbours, then converts completed values back to the original scale:
import numpy as np
from sklearn.preprocessing import StandardScaler
from sklearn.impute import KNNImputer
X = np.array([
[30, 40000], [39, 40100], [35, np.nan],
[36, 40500], [42, 42000], [24, 39000],
])
scaler = StandardScaler()
Z = scaler.fit_transform(X)
Z_completed = KNNImputer(n_neighbors=3).fit_transform(Z)
X_completed = scaler.inverse_transform(Z_completed)
StandardScaler is a scikit-learn transformer. fit learns each feature’s mean and standard deviation; transform applies that scaling. fit_transform runs both steps on the same data, while inverse_transform returns values to their original units.
KNN imputation is intuitive, can capture local patterns, and does not require specifying a regression relationship. Its result depends on k and the distance features; computation can become expensive, and distances can become less informative in high-dimensional data.
5. From neighbours to relationships
Instead of borrowing values from nearby observations, we can model a target from other variables. Bayesian Ridge uses a linear predictor with a probability model for the target:
Bayesian Ridge · conditional model
Xj | X−j, β, σ2 ∼ Normal(β0 + X−jTβ, σ2)
Its priors regularize the coefficients and support predictive uncertainty, while the mean relationship remains linear in the supplied features.
Bayesian Ridge is a regression estimator; using it for an entire incomplete table requires an additional procedure. And when might a straight-line relationship be a poor description?
Interactive graph · A relationship need not be straight
Can one straight-line relationship describe heating and cooling?
Imagine electricity use is higher in cold weather, lower in mild weather, and higher again in hot weather.
The U-shape captures higher use at both temperature extremes.
A linear model can represent this shape if useful nonlinear features are supplied. The question is what relationship the model is given the ability to express.
The temperature–electricity pattern is schematic, not measured data. A linear model can represent a curved relationship if useful nonlinear features are supplied. More flexibility is a capability, not a guarantee of better imputations.
6. MissForest: flexible relationships, iterative updates
MissForest uses regression forests to impute missing values. It can model potentially nonlinear relationships and interactions, and it can predict numerical or categorical targets. (Stekhoven & Bühlmann, 2012)
When several columns are incomplete, a variable may itself be missing from the predictors. MissForest handles this iteratively: initialize missing entries, order variables by their amount of missingness, predict each variable’s missing values from its observed rows, update the table, and repeat.
Interactive process · MissForest
What if several columns are incomplete?
Initialize. Start with the incomplete table and create initial values for missing numerical and categorical entries.
Simplified teaching view of the paper’s algorithm; the stopping rule compares changes between iterations.
This is a simplified teaching view of Algorithm 1. The paper’s actual stopping criterion compares changes across iterations. Read the original pseudocode.
7. How can researchers evaluate imputed values?
When the original value is known, researchers can hide selected observed values, impute them, and compare estimates with the values that were hidden.
- Numerical variables — normalized root mean squared error (NRMSE):
sqrt(mean((X_true − X_imputed)²) / Var(X_true)). - Categorical variables — proportion falsely classified (PFC): incorrect imputations divided by the number of hidden categorical entries.
For both measures, lower is better. This experiment evaluates artificial missingness in benchmark data. It does not prove that the imputer will perform the same way when people selectively withhold information.
When reading the paper’s results, compare numerical and categorical errors separately, and ask whether MissForest has the lowest error in every setting. A lower imputation error does not automatically imply better downstream predictions. One imputed value is still an estimate, not the recovered ground truth.
8. Take the estimate seriously (but not literally)
- The reason data are missing can shape what the observed data represent.
- KNN estimates depend on the distance, feature scales, and choice of k.
- Flexible methods can represent richer relationships, but flexibility alone does not ensure better results.
- Evaluation on artificially hidden values cannot fully reproduce real-world selective missingness.
Missing information can become a prediction problem. But a prediction is not an observation.
References
Stekhoven & Bühlmann (2012)
“MissForest—non-parametric missing value imputation for mixed-type data.” Bioinformatics, 28(1), 112–118.
Python implementation
See the MissForest Python implementation for code. This implementation reference is separate from the original research paper above.