Repeatedly, Brooks won’t tell us how much!

TUESDAY, DECEMBER 11, 2012

Always look for this problem: In this morning’s New York Times, David Brooks does his annual column about social science research.

This column showcases a problem which often has us tearing our hair. The problem appears in Brooks’ first example of top social science research (italics by Brooks):

BROOKS (12/11/12): Organic foods may make you less generous. In a study published in Social Psychology and Personality Science, Kendall J. Eskine had people look at organic foods, comfort foods or a group of control foods. Those who viewed organic foods subsequently volunteered less time to help a needy stranger and they judged moral transgressions more harshly.

Interesting! Those who viewed organic foods subsequently volunteered less time to help a needy stranger! That said, how much less time did they volunteer?

Uh-oh! Brooks doesn’t say!

We don’t mean to single out Brooks. Columnists routinely engage in this practice; routinely, columnists will cite an allegedly fascinating study without telling us how large the observed distinction was. This may let them pimp a favorite theme without telling readers that the observed distinction was in fact quite small.

This morning, Brooks isn’t pushing a policy idea; he is just having some annual fun. But again and again, he fails to tell us how large the measured effect was.

This is his second example:

BROOKS: Men are dumber around women. Thijs Verwijmeren, Vera Rommeswinkel and Johan C. Karremans gave men cognitive tests after they had interacted with a woman via computer. In the study, published in the Journal of Experimental Social Psychology, the male cognitive performance declined after the interaction, or even after the men merely anticipated an interaction with a woman.

We’re told that “the male cognitive performance declined”—but we aren’t told how much it declined! (In our view, if performance declines for more than four hours, men should seek medical attention.)

In his third example, Brooks presents the kind of finding many columnists will present to drive a point about gender equity issues. By now, you can incomparably spot the problem:

BROOKS: Women inhibit their own performance. In a study published in Self and Identity, Shen Zhang, Toni Schmader and William M. Hall gave women a series of math tests. On some tests they signed their real name, on others they signed a fictitious name. The women scored better on the fictitious name tests, when their own reputation was not at risk.

Women scored better on the fictitious tests! But did they score better by much?

Brooks presents a dozen studies in which some distinction was observed. In none of these cases are we told how big the distinction was. We often tear our hair about this very familiar practice. We do so when columnists try to advance a policy preference by citing a study this way.

Brooks closes his column as follows. Slyly, the analysts smiled about the highlighted points:

BROOKS: It’s always worth emphasizing that no one study is dispositive. Many, many studies do not replicate. Still, these sorts of studies do remind us that we are influenced by a thousand breezes permeating the unconscious layers of our minds. They remind us of the power of social context. They’re also nice conversation starters. If you find this sort of thing interesting, you really should check out Kevin Lewis’s blog at National Affairs. He provides links to hundreds of academic studies a year, from which these selections have been drawn.

No one study is dispositive? In fact, few studies are useful at all unless we’re given a rough idea how big their measured effect was. In studies like these, size matters.

Then too, as Brooks later notes, these studies can be good conversation starters! Just a guess:

Women are more likely to warm to a man in a bar. If he has a good study to show her.

This correction is no damn good: Uh-oh! Just like that, Brooks has a correction to make:

BROOKS: An earlier version of this column misstated the findings of a study in the journal Economics Letters about corporate success. The authors found that C.E.O.’s were disproportionately less likely—not disproportionately likely—to have been born in June and July.

CEOs are less likely to have been born in June and July? Either way, it didn't much matter. You see, Brooks didn't tell us how much more or less likely these picked-apart CEOs are.