0

99% accurate.Useless.

A town of 100,000 people. 1,000 of them are ill. A model that calls everyone healthy is right about 99,000 of them, and finds none of the ill.

called healthy
100,000
right
99,000
ill people found
0 of 1,000

A made-up town. Every count on this page is exact.

Every answer landsin one of four rooms.

Two questions sort every person: are they ill, and did the model flag them? That makes four rooms. This grid of four is called a confusion matrix.

Call everyone healthy.

The 1,000 ill walk into the room for the missed: ill, but cleared (a false negative). The other 99,000 are right without being looked at. 99% accurate.

Now flag everyone.

All 1,000 ill are caught, and all 99,000 healthy get a false alarm (a false positive). 1% accurate. Accuracy adds up the caught and the rightly cleared. It never says which way the rest went wrong.

ill
healthy
flagged
0caught
0false alarms
cleared
1,000missed
99,000right

accuracy 99%

Now a good test.

It catches 99% of the ill and clears 95% of the healthy. Anyone it flags tests positive. It sweeps across the town.

Who lights up?

990 of the 1,000 ill. And 4,950 of the 99,000 healthy: 5% of a crowd that big.

Everyone who testedpositive, one queue.

5,940 people walk out of the crowd. Each of them has heard the same word: positive.

Now count them.

The ill walk to the head of the queue. They fill 5 of its 30 lanes: 990 of 5,940. One in six.

16.7%

is the chance that someone who tests positive is ill. The test catches 99% of the ill, and still most alarms are false: the healthy outnumber the ill 99 to 1.

ill, given positive= 990990 + 4,950 =16.7%

The ill who test positive, over everyone who tests positive. In rates: 99% of the 1% who are ill, over that plus 5% of the 99% who are healthy. This is Bayes’ rule. The 1% is the base rate: how common the illness is before any test.

Doctors get this wrong too.

In 1978, sixty Harvard doctors and medical students were asked almost this question, about an illness ten times rarer: 1 in 1,000 ill, a test that flags 5% of the healthy. The chance that a positive is ill? The most common answer was 95%. It is about 2%. Eleven of the sixty got it right.

Casscells, Schoenberger and Graboys, New England Journal of Medicine 299:999 (1978); counts as tabulated by Manrai et al., JAMA Internal Medicine (2014).

Now a real model.

The town’s test said yes or no. A model gives a probability, and you choose where yes begins. 2,812 patients, a Sydney clinic, the 1980s; 223 had an underactive thyroid. The Line That Leans’ model learned from 1,687 of them: age and three blood tests. These 563 it has never seen; 42 are ill.

Where do you cut?

Flag everyone past the line, the decision threshold. At 0.50 it flags 14; 12 are ill. That share of the flagged row, 12 of 14, is its precision: the queue’s one in six. Of the 42 in the ill column it catches 12: its recall (or sensitivity), the town’s 99%. Drag the line.

Price the mistakes.

Say a miss costs as much as 10 false alarms. The cheapest line, found on 562 other patients, is 0.07. It catches 31 of 42; 122 of its 153 alarms are false. Accuracy fell from 94% to 76%, below the 93% of calling everyone healthy. It is the better line.

 

Every threshold at once.

Slide the line from the highest probability down to the lowest. Each stop catches some share of the ill (up: recall) and flags some share of the healthy (across: the false positive rate; the town’s was 5%, what its 95% specificity leaves). That path is the ROC curve.

The area is a tournament.

Pick one ill and one healthy patient at random. How often does the ill one get the higher probability? 17,813 of 21,882 pairs: 0.81, the area under the curve (AUC). A coin toss scores 0.5, the dashed diagonal; a perfect ranking scores 1.

The other curve counts the alarms.

At 0.07 the ROC point looks good: 74% of the ill caught, 23% of the healthy flagged. But the healthy outnumber the ill 12 to 1, so 4 in 5 alarms are false: the queue again. The precision–recall curve shows it; its area is 0.51, where a guess would follow the dashed base-rate line.

 

 

Should you balance the data?

The healthy outnumber the ill 12 to 1, a class imbalance. The usual advice: train on a balanced sample, 128 ill and 128 healthy. Does that help? First, the model you have: the 563 in five lanes by the probability it gave them, lowest on the left, the ill at the front.

The queue hardly moves.

Now the model trained on the 50/50 sample lines the same 563 up. 454 keep their lane. It ranks the ill ahead of the healthy about as well as before: AUC 0.83, against 0.81. Balancing barely changed how well it tells them apart.

It only moved the line.

So what changed? At 0.50 the first model flags 14 people and catches 12 of the 42 ill. The balanced one flags 135 (lit) and catches 31. The first model also catches 31 at 0.07, the line you chose by cost. Balancing the data is a threshold change in disguise.

And its numbers stopped being true.

Add up the probabilities a model gives a lane and you get a head count: how many it expects to be ill. A line of light stands that many heads back. The balanced model expects 194 ill; there are 42. It tells one healthy woman 53%. The first model told her 8%.

Keep the data. Move the line.

The model you had was fine; its line was in the wrong place. At 0.07 it flags 153 (lit) and catches 31 of 42, as many as the balanced one. And its numbers still mean what they say: it expects 43 ill and 42 are. That is calibrated.

 

Your town.

Move the first slider. The test has not changed, only the town.

Of the 5,940 who test positive, 990 are ill: 16.7%

 

It depends which wayI’m wrong.

  1. Accuracy cannot tell a missed case from a false alarm: it ranked calling everyone healthy (99%) above the test (95%).
  2. When the illness is rare, most alarms are false, unless the test almost never flags the healthy.
  3. Ask for the four rooms and the base rate. Choose the line by what each mistake costs.

Every number here is forthe whole crowd at once.

Split the crowd in two and the numbers stop agreeing. Next: Wrong for Whom?