CASE STUDY: Drum assesses the UVa study!

THURSDAY, JANUARY 5, 2023

It's "a dog's breakfast," he says: We're prepared to swear it! We haven't been trolling Kevin Drum in recent weeks, at least not consciously.

As far as we know, we haven't even been practically trolling Drum.

We say that because Drum has now offered a detailed post about that recent UVa study—the study we've been examining in our own ongoing "Case Study."

Drum isn't in love with the UVa study! As he starts, he offers this overview, headline included:

Do doctors believe Black people suffer less pain? A review of one influential study.

For the past week or so Bob Somerby has been writing about whether doctors believe that Black patients are more tolerant of pain than white patients. In particular, he's been writing about a frequently cited study from 2015 which you can read here if you're so inclined.

Now, Bob is practically trolling me here. This kind of thing is right in my wheelhouse: read the study, explain the ins and outs, and discuss the results. But there's a problem: I've probably read this study half a dozen times over the past few years and I read it again last night. And I've never written about it because I've never been able to make sense of it.

For the record, we haven't knowingly been trolling Kevin. That said, we're glad that he's examined the UVa study, and we're struck by his assessments.

One instant correction! In fact, we haven't been writing about "whether doctors believe that Black patients are more tolerant of pain."

In fact, the UVa study doesn't involve a survey of doctors. It's built around a survey of medical students—young people who are still involved in their medical training.

Most of the participants in the UVa study were still in their first or second year of medical school. In our view, a survey of actual doctors would have been much more interesting, but the cynic in us tells us this:

Students are easily led to the slaughter, while actual doctors are much harder to survey. At any rate, we'll start by noting several points about what Kevin has already said in the passage we've posted. 

We note that Kevin says that he has read the UVa study "half a dozen times over the past few years." We'll assume that he'd done this because of something else he says:

He describes the UVa study as "frequently cited," and also as "influential."

We'd agree with those assessments. As best we recall, we first became aware of the UVa study when Michele Norris noted one of its (supposed) findings in a column in the Washington Post in December 2020. As we've noted again and again, Norris said that the UVa study had found this:

NORRIS (12/9/20): We are not just tussling with historical wrongs. A recent study of White medical students found that half believed that Black patients had a higher tolerance for pain and were more likely to prescribe inadequate medical treatment as a result.

"A recent study of White medical students found that half believed that Black patients had a higher tolerance for pain?"

On its face, that struck us as the kind of claim we shouldn't assume to be accurate. And sure enough! When we went to the UVa study and fact-checked the claim, we found that Norris' assertion seemed to be crazily wrong.

As best we recall, that was the time when we first became aware of this UVa study. That said, in our experience, what Kevin said is right:

The UVa study actually is frequently cited by high-end journalists, and in that sense it's influential. That said, also this:

When journalists cite the UVa study, they almost always cite some variant of the alleged finding Norris described. They almost always describe what those (white) medical students allegedly said they believed about black patients' tolerance for pain. 

Norris' claim struck us as highly suspect—and as it turned out, her claim was grossly inaccurate. Also this, and this is important:

Norris' claim, which is grossly wrong, concerns an issue of race. And because of our nation's brutal racial history, such issues are deeply, profoundly important. 

Sadly, race is our struggling nation's most important ur-topic. The brutal behavior of our benighted ancestors has saddled us all with this fact.

For that reason, you'd almost think that newspapers like the Washington Post would want to be especially careful when discussing such an important topic. In reality, if you still believe something like that, you've been living on the dark side of Neptune in the past dozen years as our routinely incompetent but self-certain blue team continues developing a tribal culture which major top experts now describes as "the racialization of everything."

Alas! Our deeply unimpressive blue tribe likes, loves and simply adores claims like the one Norris attributed to the UVa study. We love to make sweeping claims about the racism of everyone else. 

By now, our tribal tribunes make such claims in the way other people breathe. In doing so, we keep creating an ugly, and profoundly unintelligent, self-satisfied tribal culture.

Kevin goes on to say quite a few unflattering things about this frequently-cited UVa study—the influential study he's "never been able to make sense of." For starters, he says this:

DRUM (1/4/23): The problem with the [UVa] study is that after presenting the results of the survey it immediately dives into a long and messy bunch of weird measurements and unclear statistics. 

According to Drum, the UVa study quickly "dives into a long and messy bunch of weird measurements and unclear statistics." Rather plainly, he doesn't mean that as a compliment.

Kevin is more conversant with academic statistical measures than we are at this site. He seems to think that the UVa study is rather weak in this area.

Having said that, we'll also say this: Even when such academic studies are built on meticulous statistical work, very few elite mainstream journalists will know how to evaluate their statistical methods and findings. 

Quite routinely, our academics make little attempt to present their findings in journalist-friendly ways. This can constitute a problem even when their statistical methods are exemplary—and according to Drum, the UVa study may have fallen considerably short of that standard.

Kevin goes on to offer other findings about the UVa study. At one point, he discusses the part of the study's research design which has tended to strike us as perhaps a bit of a scam:

DRUM: Answers are given on a scale of 1-6. [The medical students] are allowed to mark an answer as "possibly," "probably," or "definitely" true or false. With one exception, which I'll get to, virtually every single person who marked a false statement as true said it was only "possibly true." Among all the false statements, there were 229 marks of "possibly true" and only 9 marks of "probably true." There was not a single mark of "definitely true."

Sad! As we've noted, the UVa study didn't allow respondents to say that they simply didn't know if a given statement was true. If they simply said that a statement was "possibly true," respondents were charged with believing—indeed, with "endorsing"—the statement in question!

That strikes us as a very strange research procedure. It also strikes us as a procedure which will be able to locate a whole lot of racists among those medical students.

Drum seems to have proceeded further than we have into the bowels of the UVa study. At various points, he makes statements about its findings which we ourselves don't understand.

We plan to research those statements. But in the passage we've just posted, he says that very few of the medical students actually said that a false statement was "probably" or definitely" true. 

Given no way to say that they didn't know, these students merely said that certain false statements were possibly true. On that basis, they were charged with believing and endorsing the false statement, and they were implicitly condemned as racists, the judgment our blue tribe adores.

At the end of the day, how does Drum rate the UVa study? Kevin Drum is more conversant with academic work of this type than we are. That doesn't mean that his assessments are automatically correct, but this is the overall judgment he reaches:

DRUM:  Overall, this is a dog's breakfast of a study. The authors end up focusing on whether [medical students] who harbor more false beliefs also tend to rate pain lower in Black patients compared to better-informed [medical students]. It turns out they don't, but they do rate pain in white patients higher. However, the amount is smallish; it makes little difference in treatment; and the statistics presented seem cherry-picked and gnawed at a little too carefully. I'm not really sure I put much stock in the authors' conclusions.

I'd recommend that no one cite this study—and if you do, at least cite it correctly...

"The statistics presented seem cherry-picked?"  The very design of this UVa study has always seemed to be a bit scam-adjacent to us. 

We've tried to leave room for the possibility that this tiny hint of suspicion is wrong. That said, the study almost seems designed to produce the kind of judgment our blue tribe increasingly loves—to produce a claim in which These (White) Medical Students Today are the latest gaggle of racists, what with their "shocking" and "disturbing" ideas from "the 19th century."

We've shown you some of the basics in Kevin Drum's assessment. Having done so, we'll remind you of this:

We had one principal interest concerning this influential study. We've been interested in the way this study has been presented by major journalists—more specifically, in the way major journalists like Norris have managed to draw an inaccurate though tribally pleasing claim from this widely cited bit of research.

No, Virginia! Those snarling (white) medical students didn't say anything like what Norris said they did. Also, Norris didn't fact-check her sweeping claim, and her editors didn't fact-check it either.

To his day, Norris' pleasing but inaccurate claim stands uncorrected by the rapidly devolving Washington Post. Rather plainly, our tribe loves claims like the one Norris launched, but these pleasing, routinely inaccurate claims are almost surely self-defeating, politically and culturally.

The woods are lovely, dark and deep, but our own blue tribe is often a bit of a hugely self-satisfied mess. 

We love to make the racial claims which makes us feel supremely moral and pure. In truth, we aren't especially moral or pure, and we ought to stop toying with claims about race. Our conduct is often lazy and dumb, and it's also self-defeating.

Race is our tribe's most treasured topic. (Our alleged concern with issues of misogyny seems to have come and gone in a flash.) We show ourselves to be deeply flawed when we toy in these cavalier ways with the most important topic our struggling nation confronts.

Don't cite this study at all, Drum says. If you do, please cite it correctly.

He may be aiming that last bit of advice at us! Before we're done, we plan to consider some of the statements he makes in his detailed post.

Tomorrow: Let us count the ways