The same two lists, scaled two ways. Look at what the training rows are divided by.
train = [10, 20, 30] test = [40] # Leaky: fit on everything, then split everything = train + test # min 10, max 40 # train -> [0.0, 0.333, 0.666] test -> [1.0] # the training rows were scaled using a maximum of 40 — a number that # only exists in the test set # Correct: fit on train, apply to both # min 10, max 30 # train -> [0.0, 0.5, 1.0] test -> [1.5] # # 1.5 is outside the range, and that is the honest answer: the model has # never seen a value this large.
- The scaling statistics are part of the model
A minimum and a maximum learned from data are parameters exactly like weights are — fitted on the training set, then frozen and applied to everything afterwards. Once you see them that way, computing them from the test set is obviously the same mistake as training on the test set, because it is the same mistake.
- A test value outside 0 to 1 is the correct answer, not a bug
It means the model is being shown something bigger than anything it trained on, which is true and worth knowing. Clipping it into range, or rescaling the test set on its own so that it lands neatly in 0 to 1, hides exactly the fact that would have warned you. Tidy output is not the goal here.
- Production has no test set to borrow from
This is why the score could never be reproduced. At inference time the rows arrive one at a time and there is no batch to compute a minimum over — so whatever statistics the model uses must have been stored at training time. If your preprocessing cannot run on a single row, it was never going to work live.
Look at the first line. Which rows are the minimum and maximum computed from, and which rows is the model allowed to know about?
The statistics must come from the training half only. Everything else stays as it is — both halves are still scaled with the same two numbers, and the test values are no longer guaranteed to land inside 0 to 1.
min(train) and max(train), with the empty-training case refused before them, since min() raises on nothing. Resist the urge to clamp the test values into range afterwards: a value outside it is information, and hiding it is the same instinct that caused the bug.
def prepare(train, test): if not ____: raise ValueError('no training data to fit the scaling on') low, high = min(____), max(____) span = high - low if span == 0: raise ValueError(f’every training value is {low}') return ( [(value - low) / span for value in train], [(value - low) / span for value in ____], )
def prepare(train, test):
if not train:
raise ValueError("no training data to fit the scaling on")
# The training half only. These two numbers are parameters of the
# model, and the test rows are not allowed to influence them.
low, high = min(train), max(train)
span = high - low
if span == 0:
raise ValueError(
f"every training value is {low}, so there is no range to scale"
)
# Both halves, same two numbers. Test values outside the training
# range land outside 0-1, which is the honest answer.
return (
[(value - low) / span for value in train],
[(value - low) / span for value in test],
)
The fix is `min(train)` instead of `min(everything)`, and the honest validation score will be lower than the one in the handover note — because the old number was partly measuring the test set. That is the hardest part of fixing leakage: the correct answer looks like a regression, and somebody has to be told that the previous figure was never real.
The leaky version has no need for it — it always has *some* data to fit on, even when the training half is empty, because it is looking at the test half too. Once the statistics come from the training rows alone, having none of them is a genuine impossibility. New guards appearing is often a sign that a fix has made the function honest about what it needs.
This function scales two lists that it has been handed together, which is fine for an experiment and not how production works: there, the low and high have to be saved alongside the model and applied to single rows months later. If your preprocessing cannot be applied to one row without a batch around it, it is not deployable — and that is the same fact the bug report was describing.
The real answer is not to write this at all: sklearn’s Pipeline exists precisely so this mistake becomes difficult, because fit runs on the training fold and transform runs on both, and cross_val_score re-fits the scaler inside every fold. Scaling before the split is the classic leak; scaling before cross-validation is the same leak, harder to see, and it is why you put the scaler in the pipeline rather than applying it first. The same reasoning covers imputation, feature selection, target encoding and resampling — anything that learns from data belongs inside the fold. Leakage is not specific to scaling; scaling is just where you meet it first.
- Return the low and high alongside the scaled values, then write a transform(value, low, high) that can scale a single row arriving live.
- Add clipping of test values into 0–1 behind a flag, and argue both sides of whether it should default on.
- Do the same thing to the mean and deviation from z-scores, and check you can state which rows each statistic came from.