Draw thebest line.
Sixty measured points and one straight line through them. Drag its two handles. The number scores the line: the lower, the better.
4.50your line’s score
Behind it, faintly: 65,536 other lines you could have drawn, inked darker the better they score.
How wrong,point by point.
Each thread is one miss: the gap straight up or down from a point to the line. The line predicts y from x, so the miss is measured in y: r = y − ŷ, where ŷ is the line’s prediction.
The biggest miss is 5.40. 29 points sit above your line and 31 below it.
Not theshortest gap.
The dashed line is the shortest way from the point to the line. Regression ignores it. Each x is given and only y is predicted, so the miss is the vertical one.
Squareevery miss.
Each thread opens into a square. A miss of 2 becomes a square of area 4; a miss of 3, area 9. Big misses count far more: the largest square alone is 11% of the total.
The score is theaverage square.
MSE=1nΣ (yi − ŷi)2
In words: add up the areas of all sixty squares and divide by sixty. That is the mean squared error, 4.50. The lone square at the top left has exactly that area. Drag the line and watch the squares breathe.
A line hastwo knobs.
Slope w tilts it. Intercept b lifts it. The rule ŷ = w·x + b is the model; w and b are its parameters.
Every lineis a point.
The second sheet holds every setting of the two knobs. Your line is the dot: slope 0.30 across, intercept 5.00 up. Drag the dot and the line moves; drag the line and the dot follows.
Now give ita height.
Lift the dot by its line’s score, 4.50. Do that for every point on the sheet and the sheet becomes a surface: the loss surface. We will build it one data point at a time.
One point makesa trough.
Every line through this point misses it by zero: the coral fan on the data sheet is the coral crease on the sheet of knobs. Move the line away and the miss grows, its square faster: a trough. In the average it counts once in sixty, so it settles to a sixtieth of its height.
Two points,two creases.
One line fits both exactly, so their creases cross at zero.
A third point.
Three creases cross in three different places. No line fits all three, so the lowest spot lifts off the floor: 0.02.
Sixty troughsmake a bowl.
Each coral sheet is one point’s trough, settling onto the sum. The sum only rises, and its lowest spot is the best any line can do so far.
The bottom isthe best line.
Slope 0.916, intercept 2.020, score 1.570: the least-squares line. Even it misses, so the bowl never touches the floor. Your line rests higher on the glass, at 4.50. One bottom, no other dips: a convex bowl.
Bend the bowl.
Drag any point on the data sheet far from the rest. Its trough swings, the whole bowl tilts, and the best line follows. Squaring makes outliers loud: the largest square is now 12% of the best line’s score.
Best line: slope 0.916, intercept 2.020, score 1.570
Your line moves too: drag the coral ball on the glass, or the line’s two handles.
Here the bottomhas a formula.
w*=cov(x, y)var(x)
b*= − w*·
Together they give the best slope and the best intercept straight from the data, with no searching. How much x and y move together, over how much x moves on its own. For almost every other model there is no formula like this. So we will have to walk downhill, one step at a time. Next: Downhill.