0.000
average surprise, the loss

Which sideare you on?

Episode 1’s line predicted a number. This one has to pick a side. Two crowds of 120 points, navy and white, mix in the middle, and a new point lands between them. Name its side, and say how sure you are.

Draw a line.

A first guess, not learned yet: white on the far side, navy on the near side. Calling a side instead of a number is classification. The guess gets of 240 right.

How far,which side?

That is all the model will ever know about a point: its distance from the line, and the side. This white point stands units out, on the navy side: the ticks on the dashed mark are one unit each. The arrow points across the line, toward white.

The line’sverdict.

s = w·x + b

The score is the line’s verdict on a point. The sign is the side: minus for navy, plus for white. The size is the distance from the fence, the decision boundary. This arrow w has length 1, so the verdict is the distance itself, , and the dot product w·x + b computes it. Next, a verdict becomes a belief.

p = 11 + e−s

Turn distanceinto belief.

s = → p = → → s, the score p, the belief 0.5

A score can be any number; a belief must sit between 0 and 1. The sigmoid squeezes one into the other: 0 becomes exactly 0.5, far positive creeps toward 1, far negative toward 0. The three verdicts from the floor land on the curve.

The sheetrises.

Lift every spot on the floor to its p: the model’s belief that a point there is white, the predicted probability. Near 0 deep in navy, near 1 deep in white; the post marks 0, 0.5 and 1. This is what the model believes, not yet how wrong it is.

The line isstill a line.

p =11 + e−sthe sigmoid

p1 − p= esrearranged: the odds

s = lnp1 − pthe log-odds

The sheet curves, but it crosses half height along a straight line, exactly over the boundary. Read the curve backwards and the score is the log of the odds, p against 1 − p: the log-odds. Our ball: ln( / ) ≈ . A line fitted to the log-odds: that is logistic regression.

How goodis a guess?

This line was a guess. To fit it, we first need to say how wrong its beliefs are.

How surprised?

surprise = −ln p

Judge the model by its own belief. Each point grows a stalk as tall as the model’s surprise at that point’s true colour. Certain and right: 0. A coin flip: 0.69. Sure and wrong: tall.

This one stings.

A white point among the whites, given p(white) = . Surprise −ln = : the tallest stalk on the floor.

Why minusthe log?

p = 1 · 0 p = 0.5 · 0.69 p = · p, the belief in the true colour surprise · no ceiling stops at 1 Episode 1’s squared miss

Three things Episode 1’s squared miss, the grey curve, cannot do. One: punish a confident mistake without limit; it stops at 1, surprise keeps climbing. Two: add up; chances multiply, surprises add, so 240 stalks can simply be summed. Three: make one bowl with one bottom, so the ball cannot get stuck.

Add them up.

Every bead carries a stalk as tall as its surprise. The average stalk, , is the loss: the cross-entropy. Training pushes the stalks down. any bead to read its own.

Now let it learn.

ww − ηL

Downhill’s rule, on the new loss, step size η = . This loss is one bowl with one bottom, so the ball cannot get stuck.

Every wrongpoint pulls.

L = average of (p − y) · x

y is the true colour, 1 for white, 0 for navy, and p the belief. A wrong white bead pulls the arrow w toward itself, a wrong navy bead pushes it away, each by its error. The spokes are the strongest tugs. The rust arrow is their sum over every bead: the step w takes next.

The line leans.

The line sits at right angles to w, so every step of w swings it. The dashed line is the first guess; the fan is where the boundary stood along the way, fast at first, then slower.

Three knobsturn it.

w1 climbs from to , w2 eases from to , b barely moves. The sheet steepens as the arrow grows.

Most stalksshrink.

After steps the line has turned . The average surprise fell from to ; of 240 stalks shrank.

Some can’tbe pleased.

grew. Points deep inside the other crowd only get more surprising as the line gets better for everyone else.

Right and sureare different.

Accuracy counts sides: of 240 now. Training does not chase it: step had right, step only , with a lower loss. The loss also counts how sure each answer was.

0.9 is nota promise.

points get p(white) between 0.8 and 0.95, on average; of them are white. A probability describes many points like one, not any single one.

So, which side?

The new point from the start. The trained sheet gives it p(white) = : nearly a coin flip, and rightly so, it stands between the crowds. A line that leans still knows when it does not know.

Pull the crowds apart.

The same 240 points, each crowd slid outward to twice the distance. Now a straight line can split them perfectly.

100%. It keeps going.

From step every point is on its own side, yet the slope never reaches zero: each point could always be a little more sure.

Certainty without end.

By step 20,000 the sheet is a wall: every point certain. The loss has crept from to , toward 0, and never arrives. Stopping it is for later episodes: Episode 6 adds a penalty for growing too sure, Episode 15 asks for the widest gap instead.

Your turn.

The line is back on its first guess. Drag any point, or carry the new point around and watch its belief change. Set the step size and the steps, or pull the crowds apart, then press play to watch it train from the guess, step by step.

This was logistic regression: a regression in name, on the log-odds; a classifier in use. A line decides by which side of it you stand on. What if you asked who lives nearest instead? Next: Ask the Neighbours.