MACHINE LEARNING · EVALUATION · REPRODUCIBILITY

A high model score can be evidence of a mistake

KEY TERMS data leakage / train-test separation / pipelines / selection bias / cross-validation / reproducibility

A high score is easy to like. It fits neatly in a table, it moves in the right direction, and it gives a project a clean ending.

It can also be evidence that the evaluation has already seen part of the answer.

This is not an argument against metrics. A model needs measurement. It is an argument against treating one number as though it arrived alone. Before asking whether a score is impressive, ask a less flattering question: what was the model allowed to know before it was tested?

A score answers the question you actually asked

Suppose a classifier is evaluated on a held-out test set. That score is meant to estimate how the trained process behaves on data it did not use to make decisions. The phrase “trained process” matters. It includes feature selection, scaling, imputation, dimensionality reduction, hyperparameter choice, and every other operation that learns something from data.

If any of those operations uses the test partition before final evaluation, the score answers an easier question. It describes a process that had a preview of the test data, not one facing genuinely unseen data.

scikit-learn’s guidance is blunt on this point: test data should never be used to make choices about the model, and preprocessing steps that learn from data should be fitted on training data only. Its examples use feature selection because the failure is easy to see, but the same risk applies to scaling, imputation, and PCA. Common pitfalls: data leakage.

Leakage rarely looks dramatic in code

The dangerous version is often just one tidy line in the wrong place:

selected = selector.fit_transform(features, labels)
train_x, test_x, train_y, test_y = train_test_split(selected, labels)

The split happens after feature selection. The selector has already used the labels from rows that will later be called “test” data. The model may never directly fit on those rows, but the pipeline has still been shaped by them.

The safer ordering is less clever and more honest:

train_x, test_x, train_y, test_y = train_test_split(features, labels)
selector.fit(train_x, train_y)

Then the fitted selector transforms both partitions without learning from the test one. A Pipeline makes this easier to preserve when cross-validation or parameter search enters the picture.

Leakage does not need to be intentional to be damaging. It usually comes from trying to keep code compact. That is exactly why evaluation code deserves more suspicion than its size suggests.

Repeated looking is also a form of training

There is another route to a flattering number: repeatedly trying models, features, seeds, and settings, then reporting the best-looking result from the same evaluation process.

Nothing has to be technically broken. The test set can remain physically separate. But once its score guides enough choices, it is no longer functioning as a final test. It has become a selection instrument.

Cawley and Talbot describe this as over-fitting in model selection and selection bias in performance evaluation. Their point is uncomfortable because it applies even when the score estimator itself is reasonable: variance in the selection criterion creates an opportunity to select the luckiest result. The resulting gap can be comparable to the claimed differences between algorithms. Cawley and Talbot, 2010.

A result chosen after enough searching is not necessarily false. It is incomplete unless the search is reported.

One split is a sample, not a verdict

Even a clean pipeline can look better or worse depending on the partition. A limited test set may happen to contain easier cases. A single random seed can become a quiet source of optimism when it is retained because the outcome looked good.

Cross-validation helps estimate variation by fitting and evaluating across several partitions. It is not magic either: it still needs the entire fitting process to stay inside each training fold, and it is not a license to keep tuning until a preferred result appears. scikit-learn’s evaluation guidance makes both points: train/test separation matters, and model selection can itself overfit an evaluation set. Cross-validation and model selection.

The practical response is to show more than the best score. Report the split strategy, the seeds or folds, the spread of scores, the model-selection process, and any held-out final evaluation. A number becomes more useful when it comes with the conditions that could have moved it.

What a believable result leaves behind

A reader should be able to inspect the evaluation without reverse-engineering it from a screenshot. At minimum, a useful record has:

• the dataset version or fingerprint;
• the exact split or cross-validation strategy;
• the full fitted pipeline, not just the final estimator;
• hyperparameters and the rule used to choose them;
• per-split scores as well as a summary;
• the software environment and random seeds;
• a clear distinction between exploratory tuning and final evaluation.

This is not bureaucracy around a model. It is the evidence required to interpret the score.

The score is the beginning of the question

When a model reports a surprisingly good result, there are two possible reactions. One is to celebrate and move on. The better one is to ask what would make the result disappear: a different split, a safer pipeline, a less forgiving dataset boundary, or a hidden choice made after seeing the score.

That question does not make machine learning less useful. It makes the eventual result harder to fake by accident.

Related implementation

R03: reproducible evaluation for data pipelines is a controlled study of feature-selection leakage, split variation, and exact run regeneration. Its recorded results are separate from the sources and argument in this article.