0.00
loss · gradient descent · step 0
slope here 0.00
dashed line: lowest loss on the map,

Learning is a walk downhill.

A model cannot see the valley it is searching for. It can only feel the slope under its feet. Scroll, and take the steps yourself.

It cannot seethe valley.

The ground is the loss, the score of Episode 1 under its usual name: every spot is one setting of two knobs, its height how wrong that setting is. The walker knows only the slope exactly where it stands, nothing past the next ridge. Episode 1’s bowl had one bottom. Most landscapes don’t, and this hill is drawn to show why; Episode 1’s own bowl, drawn true, comes later.

So it stepsthe opposite way.

θ
where it stands
L
which way is uphill
η
how far it dares to step

The pale arrow is the downhill direction. Its length is the slope: steep ground, long arrow.

Again.And again.

Every ring is one step. After 0 steps the loss has fallen from to 0.00.

θ θ − η L(θ)

step size η0.008

Too cautious.

step size η0.06

Just right.

step size η0.30

Too bold.

your step size η0.06

Your call.

It stops.

Each step is η times the slope, and down here the slope runs out. So do the steps.

That is notthe bottom.

It is only the bottom it can feel. The real valley is across the ridge, and nothing in the slope says so.

Give itmomentum.

vβ v − ηLthenθθ + v

A heavy ball remembers its last step, v. Each new step keeps of it and adds the slope. Training code usually keeps 0.9; this ball keeps a longer memory so it clears the ridge at a step small enough to watch, .

Uphill, itkeeps going.

The pale arrow is the slope, pulling it back into the dip. The dark arrow is its memory. For steps memory wins: the loss climbs from to , and the ball tips over the ridge.

Loss .

The canyon. Same rule as before, plus a memory.

Back atthe rim.

The bold step, , bounced from wall to wall and never settled. Across the canyon the slope is steep, so a bold step across is far too long; along the canyon the slope is gentle, so the same step along is too short. One step size cannot suit both directions.

Give each directionits own step.

θθ − ηm√s

Two memories. m is the heavy ball’s: the recent slope, carried on. s is new: how big the slope has recently been, kept separately for each direction. Dividing by √s shrinks the step where the slope has been big and keeps it where the slope has been small.

Same bold step.It walks the floor.

From the rim, the bold step would have been long; Adam’s first step is . Across the canyon its step shrinks; along it, it holds. The pale arrow is the slope, the blue one the step it takes. With the same it settles on the floor in steps, at loss .

That isAdam.

Memory, plus a step size per direction, both learned from the walk itself. It is the default in most training code: not because it always finds the lowest valley, but because it rarely needs its step size tuned.

Same hill.Three walkers.

  1. Gradient descent

  2. Momentum

  3. Adam

Speed is not direction. An optimizer is judged on how fast it walks and on where it ends up.

Loss surface seen from above, contours every 0.5. Same start for all three. Step sizes: 0.03, 0.012 (β 0.97) and 0.06.

Episode 1’s bowl,drawn true.

The same sixty points, the same score. Episode 1 drew a unit of slope times longer than a unit of intercept, to fit its page. The walker has no page: the slope multiplies x, which runs from 0 to 10, and the intercept multiplies 1. Square them and average, and the ground curves times more steeply along the slope. Drawn true, the bowl is a ravine, times longer than it is wide.

curve along the slope : along the intercept : 1

Downhill pointsat the wall.

From Episode 1’s first guess, the slope under its feet points across the ravine, not along it. So every step overshoots to the far wall, and back. Any step bigger than and it is thrown out. At its best, , it needs steps to reach the bottom.

Measure xin other units.

Subtract its average, : the tilt goes. Divide by its spread, : the ravine rounds out. Same points, same best line, same lowest score, . Only the knobs mean something new. This is feature scaling: standardise every feature before you walk.

steepest ÷ gentlest curve144

Round,truly.

Now downhill points straight at the bottom. The largest stable step is , times bigger, and the best one, , gets there in step. Real landscapes are not bowls, but the stretch that units put in is the easiest part to take out.

Drop a ballanywhere.

Click the terrain. Pick a walker and a step size, or drop all three at once. Most of them end up in the same few dips. Some find the canyon.

Every model you usewas walked downa hill like this.

Every figure on this page was computed live from the surface you were looking at.

Real networks have billions of directions instead of two. The step is the same.Up there the traps are mostly saddles, not dips, and the slope is read on a few examples at a time: noisy enough to slide off them.

A line can predict a number. What if the answer is a side, yes or no?

NextThe Line That Leans