random.shuffle with a seeded generator, and why the seed is not optional.
import random rows = [1, 2, 3, 4, 5] # A generator of its own, seeded — not the module-level random shuffled = list(rows) random.Random(0).shuffle(shuffled) print(shuffled) # the same order every run, for seed 0 # random.shuffle(shuffled) uses the global generator: reproducible only # if nothing else in the program has drawn from it — which you cannot know. # Splitting the shuffled copy size = round(len(rows) * 0.2) # -> 1 test, train = shuffled[:size], shuffled[size:]
- random.Random(seed) is its own generator
Calling random.shuffle uses one global generator shared by the whole program, so anything else that draws a random number changes your split — including a library you did not write. An instance created with random.Random(seed) has its own state, so the split depends on the seed and on nothing else. That is the difference between reproducible and reproducible-if-you-are-lucky.
- Shuffle a copy, never the caller’s list
random.shuffle works in place, so it has to be handed a copy or it rearranges the rows the caller still holds — which would silently change the meaning of every later operation on them. list(rows) is the copy, and it is also what lets the function accept a tuple.
- Shuffling is not optional, because data arrives sorted
Exports come out ordered by date, by id, by customer. Taking the last 20% of that is not a sample, it is the most recent month — and a model that scores well on it may simply have learned this month. The one case where you deliberately do not shuffle is a time series, where testing on the future is the whole point and shuffling is itself the leak.
Validate the fraction, copy the rows, shuffle the copy with a generator of its own, work out the test size, and slice. The order of those five steps is the whole function.
random.Random(seed).shuffle(shuffled) — an instance, not random.shuffle, so nothing else in the program can affect the result. The test size is round(len(rows) * test_fraction).
Take the test rows from the front and the training rows from the rest, or the other way round — it makes no difference once the list is shuffled, so pick one and be consistent. Two complementary slices guarantee no row is in both halves and none is lost, which is the property the tests care about most.
import random def split(rows, test_fraction=0.2, seed=0): if not 0 <= test_fraction <= ____: raise ____(f’test_fraction must be between 0 and 1, got {test_fraction}') shuffled = ____(rows) random.____(seed).shuffle(shuffled) size = ____(len(rows) * test_fraction) return (shuffled[size:], shuffled[:____])
import random
def split(rows, test_fraction=0.2, seed=0):
if not 0 <= test_fraction <= 1:
raise ValueError(
f"test_fraction must be between 0 and 1, got {test_fraction}"
)
# A copy, because shuffle works in place — and a generator of our own,
# so nothing else in the program can change the split.
shuffled = list(rows)
random.Random(seed).shuffle(shuffled)
size = round(len(rows) * test_fraction)
# Two complementary slices: every row in exactly one half, always.
return (shuffled[size:], shuffled[:size])
shuffled[size:] and shuffled[:size] meet exactly at the same index, so the no-overlap and nothing-lost properties are guaranteed by the shape of the code rather than checked afterwards — the same half-open reasoning as the split-at exercise. Any implementation that samples the two halves independently has to prove it, and sampling with replacement cannot.
Calling random.seed sets the global generator, which means your function has just changed the random behaviour of everything else in the process — including whatever was relying on it. An instance keeps its state to itself. In library code this is the difference between reproducible and rude.
int() truncates, so 0.2 of 8 becomes 1 instead of 2 — a test set twenty per cent smaller than requested, and always in the same direction. On a small dataset that is a measurable loss of evaluation data for no reason anybody chose. Note Python rounds halves to even, so 0.5 of 5 gives 2 rather than 3; at these sizes nobody minds, and it is worth knowing before it surprises you.
sklearn.model_selection.train_test_split does this, plus stratification, plus splitting several arrays in step so that X and y stay aligned — which is the thing most likely to go wrong when you hand-roll it. Use it. Do not use any of this on a time series: shuffling lets the model learn from the future to predict the past, and the correct split there is a cut at a date, with TimeSeriesSplit for cross-validation. And if rows can share an identity — several visits by one patient, several photos of one person — a random row split puts the same subject on both sides and the score is meaningless. GroupShuffleSplit exists for that, and it is the bug the last exercise in this track is about.
- Split two lists in step, so features and labels stay aligned, and work out why doing them in two separate calls with the same seed is not safe.
- Add a validation fraction for a three-way split, and decide whether the fractions are of the whole or of what is left.
- Make it accept a generator of rows and see what that costs you — the shuffle needs the whole thing in memory, so there is nothing to stream.