WEDNESDAY, DECEMBER 9, 2020
South Dakota compared to New York state: Doggone it! We forgot to finish yesterday's post about the problem with the Covid statistic known as "positivity rate."
As we noted, Robin Lloyd has offered a critique of the funky statistic at New York magazine. Her essay appeared beneath a straightforward headline:
"The Problem With the Positivity Rate."
That's what the headline said. To our ear, Lloyd seemed to be doing a decent job until she came up with this:
LLOYD (12/7/20): [M]ost researchers avoid relying on any single number such as the positivity rate to understand the status of a community’s outbreak, preferring to examine it alongside other statistics, such as the number of and trend direction for positive coronavirus cases in a community...
For instance, it would be misleading to base policy on South Dakota’s 448 new infections reported on December 1 without also looking at its eye-popping positivity rate of 42.5 percent. Together, these numbers start to paint a picture of a runaway outbreak and insufficient testing. By contrast, New York state on the same day reported over 16 times more new infections (7,413). In the context of the state’s 3.7 percent positivity rate that day, it could suggest a more controlled outbreak and enough testing to inform efforts to control or respond to transmission.
We thought that passage was very strange—but we also thought it teaches us an anthropology lesson. First, though, what's the general problem with the use of the "positivity rate?" And what the heck is it, anyway?
When you record a state's positivity rate, you're recording what percentage of Covid tests come back positive. But that, of course, may be affected by the number of people being tested and by who those people are.
Imagine this:
State A is testing a wide cross-section of its population. By way of contrast, State B is only testing people who arrive at an emergency room displaying very bad symptoms.
All things being equal, State B will record a much higher "positivity rate." That said, there's little to gain from a comparison between two states with such different testing regimens.
We thought Lloyd was doing a decent job explaining that point—but then, she presented that passage. Immediately, the analysts began to scream and cry. They pointed to some obvious problems:
How weird! Lloyd said that, on a certain day, New York state recorded 16 times more new infections (new cases) than South Dakota did. She didn't mention a basic fact:
New York state's population is roughly 22 times the size of South Dakota's.
Under the circumstances, it seemed strange to mention the one fact without mentioning the other. Meanwhile, Lloyd had cited the numbers from one particular day, rather than looking at the average numbers of new cases over the course of a given week.
Was New York's number of new cases 16 times the size of South Dakota's on that one particular day? Lloyd didn't say where her numbers came from, but yes, it basically was, according to New York Times numbers.
But over the course of the 7-day week which ended on December 6, the number of New York state's new cases was only 10.6 times as large as South Dakota's. It seemed weird to juxtapose numbers from those two states based upon one single day, and without making any attempt to cite the large difference between the two states' populations.
Whatever Lloyd was trying to do in that passage, it seemed to us that none of its made any obvious sense. This is why we say that:
President Trump's lunatic rantings on the topic to the side, the number of new cases a jurisdiction records will depend, in part, on the amount of testing that jurisdiction is conducting, and also on who's getting tested. Consider what Lloyd says about those numbers from South Dakota:
Is a 42.5% positivity rate really a sign of a "runaway outbreak?" Not necessarily, no. What if you're only testing people when they show up at the hospital with apparent symptoms?
In that case, wouldn't you expect to record a high positivity rate? The positivity rate, by itself, wouldn't necessarily mean that you had a large outbreak.
How about South Dakota's 448 new infections that day? Would a number like that, day after day, indicate a "runaway outbreak?"
Not necessarily, no. In the case of South Dakota, it might, due to the state's small population. In a much larger jurisdiction, that number of new infections would look very different.
In short, Lloyd's analyses didn't seem to make much sense. We were especially struck by the comparison in the number of new cases, without any mention of the fact that New York State has a much larger population than South Dakota. But no part of her presentations seemed to us to make any real sense.
Alas! Within the world of mainstream journalism, Covid statistics are still very hard. The worst part of the story is this:
By now, we're nine months in!
The most direct comparison: The number of Covid cases will be affected by the amount of testing. Almost surely, the number of Covid deaths, while imperfect, is a more reliable measure.
Lloyd wasn't exactly trying to compare the outbreaks in those two states. But just for the record, here are the two states' current numbers for Covid deaths, based on Financial Times numbers:
Covid deaths per day, per million population, as of December 7 (7-day averages):
South Dakota: 26.3
United States, nationwide: 6.4New York state: 3.6
Which state has the larger outbreak? Using Covid deaths, and adjusting for population, it seems fairly easy to tell.
Presumably, "Covid deaths" remains a more reliable measure than "Covid cases." But none of these numbers means a darn thing if you don't take the size of population into account.
Covid statistics remain very hard. According to experts, this is an anthropological fact about our upper-end press corps and about the jumbled state of our national discourse.
