DATA SYSTEMS · ML EVALUATION · REPRODUCIBILITY
Reproducible evaluation for tabular classification
I built R03 around a question that is easy to miss when a model score looks good: did the evaluation procedure give the model information it should not have had, and could somebody else reproduce the number later?
The Python code, pinned dependencies, run records, charts, and verification script live in the R03 folder.
Stack Python / scikit-learn / pandas / NumPy
The setup
There are two deliberately different datasets here. The first is a bundled tabular reference dataset, used to exercise a normal scaling and logistic regression pipeline. The second has random features and random labels. It has no real signal, which makes it useful for exposing a flawed evaluation path.
For every run, I save the configuration, split seeds, dependency versions, dataset fingerprint, individual metrics, and the summary. The project then re-runs those records and compares the whole generated object rather than only checking one final score.
The leakage comparison
I selected 25 features before splitting the random-label data, then trained and
evaluated a classifier. That path reported mean held-out accuracy of 0.840.
The labels were random, so the score was not evidence of a useful classifier.
It came from letting feature selection inspect all rows before the eventual test
partition was protected.
I then put the same selector inside a scikit-learn pipeline, so it was fitted
only on training rows. Mean accuracy fell to 0.544, close to chance for the
same random-label task.
One score is not the whole result
The safe reference pipeline was also run across 30 predetermined train/test
splits. Accuracy ranged from 0.958 to 0.993, with a mean of 0.978. The
best individual split was 0.015 above that mean. I kept that result in the
record to show why it would be misleading to report only the best run.
Experiments
Reference pipeline
A correctly split scaling and logistic-regression pipeline was run across five fixed seeds.
Feature-selection leakage
Random-label data scored 0.840 after leaky selection and 0.544 with selection inside the training pipeline.
Split variation
Thirty fixed splits produced safe-pipeline accuracy from 0.958 to 0.993.
Recorded-run verification
The saved E01–E03 objects were regenerated exactly from their recorded configuration and environment.
What I take from it
A pipeline is useful here because it puts feature selection and model fitting inside the training step. The more important habit is to protect the test data before making choices, record more than one split, and leave enough detail for the run to be checked again.
Limits
The random-label experiment is intentionally controlled; it does not say how large leakage effects will be in every real dataset. The reference dataset is not a medical model or a deployment claim. Exact regeneration currently depends on the recorded Python environment and versions, so it should be treated as a local reproducibility check, not a guarantee for every platform.