Same model.Two scores.
Fifty patients, 5,000 random gene readings each, a coin-flip diagnosis apiece: nothing here predicts anything. One classifier, scored two ways. The only difference is when the answers were looked at. An honest score, and what breaks it, is this whole episode.
97%
after one peek at the test answers
50%
with the test answers sealed first
Means of 50 runs; single runs scored 92–100% and 26–74%. Drawn behind: one typical run’s 50 patients × 5,000 random readings, the 20 genes its peek picked in red pencil, the coin-flip diagnoses (+ or −) down the left edge.
The Trade’s wiggly curve,on a leash.
First, a model to examine. These are The Trade’s 30 points and its degree-12 curve: 13 weights, one per wiggle shape (drawn under the bars), fitted by least squares. It threads the points (training error ) and pays on new data (). That is overfitting.
Where thewiggles live.
67% of the curve’s weight sits on the even shapes, w₂, w₄ and on (marked in red). This sine never uses them: that weight is spent chasing the noise. So make every unit of weight cost something.
Charge forbig weights.
loss = average squared miss
+ λ × (w₁² + w₂² + … + w₁₂²)
λ is the price of a unit of squared weight. At λ = 0 it is free: The Trade’s curve. Raise it and the bars shrink, the even ones first (4% of the weight at λ = ). This is ridge regression (L2).
λ is ahyperparameter.
Too much, and the curve goes flat (error on new data at λ = 10). The fit cannot choose λ: on its own points, error only falls as the leash loosens, so it would always pick none. Someone else has to. Drag the leash yourself.
training error · on new data · size of the weights
“On new data” is exact here: we built this world, so we can measure it without opening the sealed exam.
A second leash:lasso.
loss = average squared miss
+ λ × (|w₁| + |w₂| + … + |w₁₂|)
Lasso charges for each weight’s size, not its square (L1). Same curve, lasso’s price: raise λ and the bars drop to exactly zero, one by one, until only w₁ and w₃ are left: the sine’s own shapes. Sparsity. Ridge only ever shrank them. Why?
Why lassomakes zeros.
Keep two weights, w₁ and w₃. Every price λ is a budget t, the most weight the fit may spend: a circle for ridge, a diamond for lasso. Tighten it. The circle is touched on its smooth side; the diamond at a corner, where from t = down w₁ is exactly 0.
training error · on new data · the weights’ sizes add to
Who sets λ?
Someone has to choose λ, and it can’t be the training error. So hold 6 cards back (ringed), fit on the other 24 at every λ, and score each fit on those 6. The best score, , picks λ = : one rehearsal. Its answer rests on just 6 cards.
Five rehearsals.
Let every card sit out once. Shuffle The Trade’s thirty points and deal them like a hand of cards into five piles of six; each pile takes its turn as the held-back six, and the five scores are averaged. A sealed envelope waits on the desk.
Pile 1 sits out.
Fit the degree-12 curve on the other 24 cards at every strength of penalty, from a tight leash to none at all. Score each fit on the 6 cards it never saw: the average of their squared misses.
Every card sits outexactly once.
Same sweep, a different pile each time. Each pile alone is one rehearsal, and they disagree: alone they would pick λ from to . Pile 4 holds the card at the far left edge, and without a leash the curve flies past it.
Average the five.
The chartreuse curve is their mean: cross-validation. The shaded band is one standard error either side, the rehearsals’ own uncertainty about the score.
λ =
The lowest average, , scored on all thirty cards, not six. Refit on all thirty with that leash, and the curve settles over the sine. The faint curves are the five rehearsals’ own fits.
Now openthe envelope.
fresh points, sealed until now: the test set. Exam score ; the rehearsals promised . One rehearsal’s λ would have scored . Open the envelope once, last; open it earlier and it stops being an exam: that is the peek.
The peek.
Now one of those 50 runs, step by step. Fifty patients, 5,000 random gene readings each, every diagnosis face up down the left. Step 1: find the 20 genes that best follow the diagnoses, reading all 50 answers. With 5,000 tries, luck alone finds some: red pencil.
Then the test.
Now cross-validate: deal the patients into five piles of 10. Pile 1’s answers go into the envelope, sealed. A simple classifier learns the red genes from the other 40, then guesses pile 1. Open it to score: of 10 right. All five piles: .
Sealedtoo late.
But in step 1 those ten answers were face up: they helped choose the red genes (marked where their rows cross them). The envelope was sealed after the choice, so the genes already carry its answers. That is the leak, and cross-validation can’t see it.
Done right.
Seal first, then choose. Pile 1’s answers go into the envelope before anything is picked; the 20 genes come from the other 40 answers only (chartreuse, drawn through those 40). Open it to score: of 10 right. All five piles: , a coin flip, as it should be.
Fifty runs,two scores.
Fifty fresh sets of random readings and coin-flip diagnoses: the peek scores or more every time; done right, it scatters around 50%. Seal the test answers before anything is chosen: genes, λ, anything.
Two yearsof days.
One more way to peek, without meaning to. An invented café’s customers, one dot per day: a slow rise, a busy summer, busy Saturdays, and lucky streaks. The forecaster knows only the date, through the leash’s degree-12 curve, and the weekday. The last ten weeks are sealed.
Shufflethe days.
Deal the first 657 days into five piles, as before. Pile 1’s days are ticked in red: of its have their tomorrow in the other piles, so the curve only fills gaps between days it has seen. The rehearsals pick λ = , no leash at all, and promise ± .
Splitby time.
A forecast only ever has the past, so rehearse that way: fit on every day before a block, test on the next 73, move on. Carried past its last day, the curve drifts; block misses by . Five blocks pick λ = and promise ± .
Openthe envelope.
Each fits all 657 days at its own λ and forecasts the sealed ten weeks. Shuffling promised and delivered : with no leash, the curve runs away. Splitting by time promised and delivered : rough, but on the safe side. When the rows are days, split by time.
This café is the median of 61 invented series by how much worse shuffling’s λ did; on … of the other it did more than twice as badly.
Your exam.
Change anything and the whole procedure runs again: the deal, every rehearsal, the average, the pick. The envelope is open now, so its score is shown; choosing λ by it would be the peek all over again.
A penalty, five rehearsals,one envelope.
The leash tames the curve. The rehearsals choose the leash. The envelope, opened once, gives the score. Under all three, one rule: nothing the score is graded on may help build the model. An honest score, then. But a score of what, and wrong in which direction? Next: The Rare Case.