CASE STUDY: Drum assesses the UVa study!

THURSDAY, JANUARY 5, 2023

It's "a dog's breakfast," he says: We're prepared to swear it! We haven't been trolling Kevin Drum in recent weeks, at least not consciously.

As far as we know, we haven't even been practically trolling Drum.

We say that because Drum has now offered a detailed post about that recent UVa study—the study we've been examining in our own ongoing "Case Study."

Drum isn't in love with the UVa study! As he starts, he offers this overview, headline included:

Do doctors believe Black people suffer less pain? A review of one influential study.

For the past week or so Bob Somerby has been writing about whether doctors believe that Black patients are more tolerant of pain than white patients. In particular, he's been writing about a frequently cited study from 2015 which you can read here if you're so inclined.

Now, Bob is practically trolling me here. This kind of thing is right in my wheelhouse: read the study, explain the ins and outs, and discuss the results. But there's a problem: I've probably read this study half a dozen times over the past few years and I read it again last night. And I've never written about it because I've never been able to make sense of it.

For the record, we haven't knowingly been trolling Kevin. That said, we're glad that he's examined the UVa study, and we're struck by his assessments.

One instant correction! In fact, we haven't been writing about "whether doctors believe that Black patients are more tolerant of pain."

In fact, the UVa study doesn't involve a survey of doctors. It's built around a survey of medical students—young people who are still involved in their medical training.

Most of the participants in the UVa study were still in their first or second year of medical school. In our view, a survey of actual doctors would have been much more interesting, but the cynic in us tells us this:

Students are easily led to the slaughter, while actual doctors are much harder to survey. At any rate, we'll start by noting several points about what Kevin has already said in the passage we've posted. 

We note that Kevin says that he has read the UVa study "half a dozen times over the past few years." We'll assume that he'd done this because of something else he says:

He describes the UVa study as "frequently cited," and also as "influential."

We'd agree with those assessments. As best we recall, we first became aware of the UVa study when Michele Norris noted one of its (supposed) findings in a column in the Washington Post in December 2020. As we've noted again and again, Norris said that the UVa study had found this:

NORRIS (12/9/20): We are not just tussling with historical wrongs. A recent study of White medical students found that half believed that Black patients had a higher tolerance for pain and were more likely to prescribe inadequate medical treatment as a result.

"A recent study of White medical students found that half believed that Black patients had a higher tolerance for pain?"

On its face, that struck us as the kind of claim we shouldn't assume to be accurate. And sure enough! When we went to the UVa study and fact-checked the claim, we found that Norris' assertion seemed to be crazily wrong.

As best we recall, that was the time when we first became aware of this UVa study. That said, in our experience, what Kevin said is right:

The UVa study actually is frequently cited by high-end journalists, and in that sense it's influential. That said, also this:

When journalists cite the UVa study, they almost always cite some variant of the alleged finding Norris described. They almost always describe what those (white) medical students allegedly said they believed about black patients' tolerance for pain. 

Norris' claim struck us as highly suspect—and as it turned out, her claim was grossly inaccurate. Also this, and this is important:

Norris' claim, which is grossly wrong, concerns an issue of race. And because of our nation's brutal racial history, such issues are deeply, profoundly important. 

Sadly, race is our struggling nation's most important ur-topic. The brutal behavior of our benighted ancestors has saddled us all with this fact.

For that reason, you'd almost think that newspapers like the Washington Post would want to be especially careful when discussing such an important topic. In reality, if you still believe something like that, you've been living on the dark side of Neptune in the past dozen years as our routinely incompetent but self-certain blue team continues developing a tribal culture which major top experts now describes as "the racialization of everything."

Alas! Our deeply unimpressive blue tribe likes, loves and simply adores claims like the one Norris attributed to the UVa study. We love to make sweeping claims about the racism of everyone else. 

By now, our tribal tribunes make such claims in the way other people breathe. In doing so, we keep creating an ugly, and profoundly unintelligent, self-satisfied tribal culture.

Kevin goes on to say quite a few unflattering things about this frequently-cited UVa study—the influential study he's "never been able to make sense of." For starters, he says this:

DRUM (1/4/23): The problem with the [UVa] study is that after presenting the results of the survey it immediately dives into a long and messy bunch of weird measurements and unclear statistics. 

According to Drum, the UVa study quickly "dives into a long and messy bunch of weird measurements and unclear statistics." Rather plainly, he doesn't mean that as a compliment.

Kevin is more conversant with academic statistical measures than we are at this site. He seems to think that the UVa study is rather weak in this area.

Having said that, we'll also say this: Even when such academic studies are built on meticulous statistical work, very few elite mainstream journalists will know how to evaluate their statistical methods and findings. 

Quite routinely, our academics make little attempt to present their findings in journalist-friendly ways. This can constitute a problem even when their statistical methods are exemplary—and according to Drum, the UVa study may have fallen considerably short of that standard.

Kevin goes on to offer other findings about the UVa study. At one point, he discusses the part of the study's research design which has tended to strike us as perhaps a bit of a scam:

DRUM: Answers are given on a scale of 1-6. [The medical students] are allowed to mark an answer as "possibly," "probably," or "definitely" true or false. With one exception, which I'll get to, virtually every single person who marked a false statement as true said it was only "possibly true." Among all the false statements, there were 229 marks of "possibly true" and only 9 marks of "probably true." There was not a single mark of "definitely true."

Sad! As we've noted, the UVa study didn't allow respondents to say that they simply didn't know if a given statement was true. If they simply said that a statement was "possibly true," respondents were charged with believing—indeed, with "endorsing"—the statement in question!

That strikes us as a very strange research procedure. It also strikes us as a procedure which will be able to locate a whole lot of racists among those medical students.

Drum seems to have proceeded further than we have into the bowels of the UVa study. At various points, he makes statements about its findings which we ourselves don't understand.

We plan to research those statements. But in the passage we've just posted, he says that very few of the medical students actually said that a false statement was "probably" or definitely" true. 

Given no way to say that they didn't know, these students merely said that certain false statements were possibly true. On that basis, they were charged with believing and endorsing the false statement, and they were implicitly condemned as racists, the judgment our blue tribe adores.

At the end of the day, how does Drum rate the UVa study? Kevin Drum is more conversant with academic work of this type than we are. That doesn't mean that his assessments are automatically correct, but this is the overall judgment he reaches:

DRUM:  Overall, this is a dog's breakfast of a study. The authors end up focusing on whether [medical students] who harbor more false beliefs also tend to rate pain lower in Black patients compared to better-informed [medical students]. It turns out they don't, but they do rate pain in white patients higher. However, the amount is smallish; it makes little difference in treatment; and the statistics presented seem cherry-picked and gnawed at a little too carefully. I'm not really sure I put much stock in the authors' conclusions.

I'd recommend that no one cite this study—and if you do, at least cite it correctly...

"The statistics presented seem cherry-picked?"  The very design of this UVa study has always seemed to be a bit scam-adjacent to us. 

We've tried to leave room for the possibility that this tiny hint of suspicion is wrong. That said, the study almost seems designed to produce the kind of judgment our blue tribe increasingly lovesto produce a claim in which These (White) Medical Students Today are the latest gaggle of racists, what with their "shocking" and "disturbing" ideas from "the 19th century."

We've shown you some of the basics in Kevin Drum's assessment. Having done so, we'll remind you of this:

We had one principal interest concerning this influential study. We've been interested in the way this study has been presented by major journalists—more specifically, in the way major journalists like Norris have managed to draw an inaccurate though tribally pleasing claim from this widely cited bit of research.

No, Virginia! Those snarling (white) medical students didn't say anything like what Norris said they did. Also, Norris didn't fact-check her sweeping claim, and her editors didn't fact-check it either.

To his day, Norris' pleasing but inaccurate claim stands uncorrected by the rapidly devolving Washington Post. Rather plainly, our tribe loves claims like the one Norris launched, but these pleasing, routinely inaccurate claims are almost surely self-defeating, politically and culturally.

The woods are lovely, dark and deep, but our own blue tribe is often a bit of a hugely self-satisfied mess. 

We love to make the racial claims which makes us feel supremely moral and pure. In truth, we aren't especially moral or pure, and we ought to stop toying with claims about race. Our conduct is often lazy and dumb, and it's also self-defeating.

Race is our tribe's most treasured topic. (Our alleged concern with issues of misogyny seems to have come and gone in a flash.) We show ourselves to be deeply flawed when we toy in these cavalier ways with the most important topic our struggling nation confronts.

Don't cite this study at all, Drum says. If you do, please cite it correctly.

He may be aiming that last bit of advice at us! Before we're done, we plan to consider some of the statements he makes in his detailed post.

Tomorrow: Let us count the ways


45 comments:


  1. tl;dr
    Hey, dear Bob, it's a fucking study; the word synonymous with bullshit.

    Someone paid to produce this it (someone pays to produce any study). People/institutions that paid knew what results they wanted. And the people who produced this (or any other) study produced exactly what their customers wanted. It's that simple, dear Bob.

    ...and that's all there is to it, dear Bob...

    ReplyDelete
  2. "Kevin is more conversant with academic statistical measures than we are at this site. He seems to think that the UVa study is rather weak in this area.

    Having said that, we'll also say this: Even when such academic studies are built on meticulous statistical work, very few elite mainstream journalists will know how to evaluate their statistical methods and findings. "

    Drum clearly did not understand the statistical analysis. Neither Somerby nor Drum has the background to do a serious critique of this work.

    In his own Howler, Drum suggests that researchers can just never cite this study. No serious researcher would do that. It is bad scholarship -- the cherrypicking Drum himself seems to recognize is wrong.

    I am done with this topic. Unlike either Drum or Somerby, I have conducted pain research and I understand the statistics. I am not going wade into the weeds, nor am I going to try to explain stats over the internet to people without the background to understand them. It is a waste of time.

    Research is intended to answer questions, not defend or contest favorite points of view, as Somerby tries to do. Dismissing one study doesn't address the large literature that already exists supporting the fact of undertreatment of minority pain. Somerby ignores the main part of the study, which shows a link between misinformation and pain decision-making. Drum follows Somerby's lead and dismisses the main part of the study too, misinterpreting the finding and ignoring the statistics (too complicated, he says). There is no spirit of inquiry in their critique, only a desire to dismiss the findings. That isn't anything like science. But then, neither of them is a scientist.

    So, I am not going to waste time going down whatever rabbit holes they think undermine this research. They do not even have the premise of the study right, its main research question, so nothing that follows can make any sense. This is what happens when amateurs with an over-inflated sense of their own knowledge, confuse matters and then commenters with no knowledge whatsoever come along and confuse matters even more. Chaos ensues and nothing productive will come from this except a few morons will believe that undertreatment of pain is not a thing, and Somerby can defend against the idea that he himself might be categorized as a racist based on his unwillingness to confront racial issues honestly.

    Fortunately, physicians take their jobs more seriously than these two jokers, and are working to make sure that all of their patients are treated as effectively as possible.

    ReplyDelete
    Replies
    1. You're stupid if you believe the bigots on the Right are going to accept this.

      Fixed it for you, 11:36.

      Delete
    2. 11:36: You have cooties. And you eat worms.

      Delete
    3. Your comment is possibly true.

      Delete
    4. But it's possibly false.

      Delete
    5. Thank you seems interesting. Not sure I was able to get much from it so far - mainly you're saying they are wrong and it can't be explained to layperson(s)?

      Delete
  3. I fully expect Somerby to bring out the research that helped decide Brown v Board of Education and claim that it is flawed and separate but equal really doesn't damage the self esteem of black children. And then Kevin Drum can chime in and tell us that the Clarks' work should never be cited. And then Somerby can go back to defending book banning and the other assaults on education.

    Classroom teaching is one part of education. Access to books is another part of education. Participation in research and learning research techniques and methodology is a third part of education. Undergrads and med students were subjects in this study. Conducting research is the third main function of college faculty (together with teaching and service to the school and community). Mocking the research that comes out of such work is part of the attack on professors and universities that Somerby has been engaging in for years now. It is part of the right wing attack on free inquiry and idea, and it is what authoritarians and fascists do in order to avoid criticism and control institutions when they take over power.

    This study about pain is a twofer -- Somerby gets to undermine professors and their work plus he gets to hit the idea that racial misunderstandings lead to mistreatment of black patients.

    Kevin Drum has made a huge mistake today. In a sane world, it would damage his reputation. I have been assuming that he doesn't put the same effort into his blog as in the past when he was not battling cancer, but he has just phoned this one in. The mystery is why Somerby doesn't attack Drum, given that he is a journalist, and why Drum continues to keep Somerby on his blogroll given that Somerby has gone over to the dark side.

    ReplyDelete
  4. The choice of statistics is based on the research question(s), the study design, and the nature of the data. Drum's complaint about the statistics is ridiculous. Someone without the background to understand a statistical analysis would consult a more knowledgeable colleague or a statistician, not complain that the stats are too complicated.

    Research studies in academic journals (like PNAS, Proceedings of the National Academy of Sciences) are not intended to be read by the general public. They are aimed at trained experts in the domain being investigated. They become part of the scientific literature in that field. The National Academy of Sciences is an advisory body to the President (and government) and consists of top experts in a variety of scientific disciplines. It is a top journal. Academics know how to interpret the seriousness of the journals that studies appear in, not simply their content.

    ReplyDelete
    Replies
    1. Actually PNAS is not as rigorous as refereed journals, like JAMA.

      Delete
    2. David, both are peer reviewed.

      Delete
  5. anon 12:16 - as TDH makes clear, his main focus is on Norris's characterization of the results of the study - that it showed 50% or the med students and residents believed that blacks had a higher tolerance of pain. TDH says that this take of what the study determined has been put forth by a number of liberal pundits and commentators - the study is commonly accepted as making that claim. TDH says that Norris' characterization of the study is bogus. Are you saying that Norris got it right?

    ReplyDelete
    Replies
    1. This comment has been removed by the author.

      Delete
    2. Somerby sure has expended a lot of effort trying to debunk/mock/ridicule the study for you to say his main focus is Norris.

      Delete
    3. MH, the fact that the study is flawed reinforces the criticism of Norris, so it is relevant.

      Delete
    4. Ac, um, logic? If he merely wanted to debunk Norris, he would simply show how she misrepresented the study. But he doesn’t do that. He has spent a lot of time debunking the study. For Christ’s sake, he called it “scam-adjacent.” It’s clear (to make a perfectly obvious point … well, obvious to anyone who doesn’t have his head up Somerby’s ass) that he wants to attack both Norris and the study (quick, can you name one us liberals’ favorite researchers who ran the study? I mean, it’s practically all my blue friends ever talk about) as examples of liberal deceit.

      Delete
    5. All studies have flaws. The purpose of peer review is to determine whether there is still value in publishing the study after the flaws have been identified. Fixable flaws cause the paper to be returned to the author for more data analysis, clarification of writing, etc. Fatal flaws cause the paper to be rejected. Many journals rejected papers with fixable flaws, under the assumption that the authors with revise and resubmit and then take their chances with another peer review. The idea that there are any papers without flaws is unrealistic and not the standard for publication.

      Somerby and Drum don't understand this, and neither do you AC/MA, judging by your comments. Somerby additionally seems to believe that if he pokes enough holes in a study (including specious ones that are not actually problems), then he can deny the results outright and need not accept anything the data shows or the paper says. It seems to be all-or-nothing for him, and science doesn't work that way, nor does any area of human endeavor.

      Delete

    6. "Fatal flaws cause the paper to be rejected"

      Bullshit, dear dembot. The reviewers are in the same business, and they need to play ball if they want to make money.

      Here:
      https://en.wikipedia.org/wiki/Grievance_studies_affair

      Delete
    7. Yes, postmodernism in literary criticism sucks. What does that have to do with science?

      Delete
  6. Drum says this:

    “The problem with the study is that after presenting the results of the survey it immediately dives into a long and messy bunch of weird measurements and unclear statistics. “

    Yes. Friend, can you really say you understand something our “Statisticians” call “standard deviation?”

    I mean, how can deviation be standard?

    And what is the “mean?” Are they trying to call the white medical students mean??

    Your lizard will disagree.

    ReplyDelete
  7. Drum says this in his post:

    “The only noticeable effect is that S&Rs [white medical students and residents] who hold a lot of false beliefs tend to have higher assessments of pain in white people.”

    …which means they have lower assessments of pain for black people. (This comes to us from something called “captain obvious”).

    So, ultimately, Drum affirms the result of the study he has just attempted to debunk!

    Priceless!

    ReplyDelete
    Replies
    1. Affirms the 50% claim?

      Delete
    2. 2:04: you are retarded.

      Delete
    3. Yes, because the study says that 50% of the S&R’s endorse false beliefs, and that among that 50%, the number and strength of the endorsement correlates to a lower pain rating for blacks. It is a slope, when you graph it. Drum edits this by focusing on the top end. He omits the gradient, but nonetheless claims even his edited presentation shows significant results.

      Delete
    4. So Drum does not affirm the 50% claim.

      Delete
    5. 3:11: Ah. There it is. Can’t state what he is even referring to, you little game-playing weasel.

      Delete
    6. Oh. The Norris 50% claim at the center of the discussion.

      Delete
    7. Her claim is not relevant to my original comment. My point is that Drum is essentially endorsing the results of the study. As the study puts it: “The simple slope analyses indicated that participants who endorsed more false beliefs rated the black target as feeling less pain than the white target.”

      Delete
    8. Oh. Ie. your original comment is not relevant to the claim at the center of the discussion. Cool!

      Delete
    9. There are two discussions, both initiated by Somerby. One, about Norris. Another, about the study itself. I understand that you have trouble keeping the two separate, but do try.

      Delete
    10. Will do. Thanks. Drum is endorsing the results of the study he called a dog's lunch, simple with weak results that no one should bother with or cite. Awesome!

      Delete
    11. Drum has no idea what he is talking about. Adjectives are cheap. They sell them 10 cents a dozen, or 5 cents if they are negative.

      For example, I think Kevin Drum is a dog's lunch, simple-minded with no actual analysis, someone no one should bother with or cite (one only cites actual research, in this case no one should quote him).

      mh's point was that Drum doesn't even realize he is describing the same results as the study itself. What a fool Drum is!

      Delete
  8. It’s Groundhog Day, the movie.

    In the House, “McCarthy appears to lose speaker vote for ___th time.”

    And here at the Daily “that UVa study is a scam” blog, “that UVa study is a scam.”

    ReplyDelete
    Replies
    1. Thankfully, MH, there are other news sources than TDH where you can catch on the news stories he misses.

      Delete
    2. Oh, I think you misunderstood me, AC. I am mocking Somerby, not asking him to report on anything else. If he wants to discuss the UVa study every day until he croaks, that’s fine by me. It just makes him look obsessed and ridiculous. That’s all I’m saying.

      Delete
  9. Kevin McCarthy can end this joke anytime he wants. He needs to give the Republican holdouts what they want: an 8 x 10 glossy photo of Hunter Biden's penis, they can love as their own.

    ReplyDelete
    Replies
    1. He'll need to "shove it down their throats", like Obama did with the ACA, if he wants to excite them enough to vote for him.

      Delete
    2. I had to try to decide whether he needs to shove the photo or the actual penis down their throats.

      Delete
    3. The Republicans could end this joke by voting for Hakeem Jeffries.

      Delete
    4. NY Times: Why won’t the Democrats put country ahead of party and vote for McCarthy?

      Delete
    5. If there's one thing the American people are demanding, it's a House Speaker whose members of his own party proudly declare is not trustworthy.

      The republican do-nothing party - you can't make it up.

      Delete