THE PROFESSORIATE FAILS US AGAIN: No one checked Reinhart and Rogoff’s work!

TUESDAY, APRIL 30, 2013

Part 2—The tale of the graduate student: We can think of two obvious questions about the latest giant bungle by the nation’s high-ranking professors.

Here’s the first question, and others have asked it: How did Carmen Reinhart and Kenneth Rogoff manage to bungle so badly?

People do make mistakes, of course, especially Harvard professors. But some of the errors by Reinhart and Rogoff do seem just a bit strange.

If you’re averaging data from twenty countries, how do you manage to leave out five countries? (Answer: You make an Excel coding error.) Once you’ve made your coding error, how do you fail to notice that your results don’t seem quite right?

We have learned to expect the worst from our highest-ranking authorities. Even so, some of the bungling in this case does seem somewhat strange.

That said, our second question is harder to answer—and we haven’t seen other folk ask it. This is our second question:

Why did three years go by before someone checked Reinhart and Rogoff’s work? And why was this important task left to three graduate students?

Here’s why that question is puzzling:

According to Paul Krugman and others (see below), many people were skeptical of Reinhart and Rogoff’s results from the first. And uh-oh! The study by the two professors turned out to be quite influential.

Powerful interests used their results to support austerity policies. According to Krugman, their study “instantly became famous; it was, and is, surely the most influential economic analysis of recent years.”

Under the circumstances, wouldn’t you think that somebody other than graduate students would have double-checked Reinhart and Rogoff’s results—results which didn’t quite seem to make sense? How did three long years go by before someone checked their work?

If you’re willing to wonder about that question, the BBC’s report on this matter is especially worthwhile. Writing for the BBC, Ruth Alexander interviewed Thomas Herndon, the University of Massachusetts graduate student who took the lead in double-checking Reinhart and Rogoff’s work.

This interview helps us see the way our society’s intellectual elites tend to function. Because alas! Herndon’s account of his adventure does seem sadly familiar.

Alas! Like a latter-day Joseph Campbell, Alexander seems to sketch the journey of the modern intellectual hero in her report on Herndon. As Herndon tells his story, Alexander records four stages on his journey:
His initial sense that something is wrong with some authority's findings: Herndon finds himself doubting Reinhart and Rogoff’s results.
His attempt to confirm that authority's findings: Herndon tries to replicate the professors’ findings but is unable to do it.
His assumption that his own lowly work must be wrong: Because Reinhart and Rogoff are famous professors, Herndon assumes that he, the graduate student, must be doing something wrong.
His eventual slaying of the dragon: Herndon asks the professors for their data, discovers that they have erred.
The most interesting part of Alexander’s report involves the third stage of this sadly familiar journey. Herndon describes a prevailing assumption—our highest-ranking intellectual elites simply can’t be wrong:
ALEXANDER (4/19/13): [W]hile US politicians were arguing over whether to inject more stimulus into the economy, the euro was creaking under the strain of forced austerity, and a new coalition government in the UK was promising to raise taxes and cut spending to get the economy under control, Thomas Herndon's homework assignment wasn't going well.

No matter how he tried, he just couldn't replicate Reinhart and Rogoff's results.

“My heart sank,” he says. "I thought I had likely made a gross error. Because I'm a student the odds were I'd made the mistake, not the well-known Harvard professors."

His professors were also sure he must be doing something wrong.

"I remember I had a meeting with my professor, Michael Ash, where he basically said, 'Come on, Tom, this isn't too hard—you just gotta go sort this out.’”
In this, the dark night of the grad student’s soul, everyone assumed that he, the lowly student, must be getting it wrong!

To his credit, Professor Ash finally urged young Herndon to ask Reinhart and Rogoff for their data. To their credit, Reinhart and Rogoff complied with this request.

When Herndon got the professors’ data, he spotted their mistakes. This brings us back to our puzzling question:

Why was it left to three graduate students to check the professors’ work? Since people had doubts about their study, why didn’t other professors ask to see their data?

Below, you see Krugman’s capsule account of these events. Why was the slaying of this dragon left to three graduate students?
KRUGMAN (4/19/13): [T]he truth is that Reinhart-Rogoff faced substantial criticism from the start, and the controversy grew over time. As soon as the paper was released, many economists pointed out that a negative correlation between debt and economic performance need not mean that high debt causes low growth. It could just as easily be the other way around, with poor economic performance leading to high debt. Indeed, that's obviously the case for Japan, which went deep into debt only after its growth collapsed in the early 1990s.

Over time, another problem emerged: Other researchers, using seemingly comparable data on debt and growth, couldn't replicate the Reinhart-Rogoff results. They typically found some correlation between high debt and slow growth—but nothing that looked like a tipping point at 90 percent or, indeed, any particular level of debt.

Finally, Ms. Reinhart and Mr. Rogoff allowed researchers at the University of Massachusetts to look at their original spreadsheet—and the mystery of the irreproducible results was solved. First, they omitted some data; second, they used unusual and highly questionable statistical procedures; and finally, yes, they made an Excel coding error. Correct these oddities and errors, and you get what other researchers have found: some correlation between high debt and slow growth, with no indication of which is causing which, but no sign at all of that 90 percent ''threshold.''
In Krugman’s account, the graduate students are described as “researchers at the University of Massachusetts.” This softens the blow as we ask once again:

Why didn’t other professors check Reinhart and Rogoff’s work?

There may be a good answer to that, but this pattern seems all too familiar. On the one hand, we think of the famous child who was able to spot his emperor’s lack of new clothes.

We also think of our own experience with a fraudulent test score-reporting practice affecting the entire state of Virginia. Tomorrow, in part 3 of this report, we will revisit that episode.

Three long and punishing years went by before anyone checked the professors’ work! In this badly fallen age, this is very much the way our intellectual elites tend to function.

Tomorrow: Nobody bothered to check. After we checked, no one tattled!

18 comments:

  1. The best explanation is that the paper in question was never formally published in a journal, so no one peer reviewed their work. When you publish in a journal you must "show your work" so to say, to the journal's editors and referees. Still, having said that, someone should have requested the data from the two authors and checked their work, since it became so famous and influential. This is another black mark on the social science of economics, and its practitioners.

    ReplyDelete
    Replies
    1. I am a professor and I have twice requested data from other researchers. Both times my request was ignored -- no response whatsoever. It is part of our ethics to provide data when requested by a legitimate researcher. I find it dismaying that this aspect of ethics is ignored but there is also nothing to be gained and a great deal to be lost by letting others examine your data. If you do find an error in someone else's work, you make an enemy.

      Delete
    2. I find your experience very disturbing. It needs to be a requirement for researchers to make their data open to those who wish to see it, since so many important public policy decisions are made based on scientific research. There is a movement in the medical sciences, spearheaded by Dr. Ben Goldacre, to open up and publish all data from all medical research, published or not. You can see the campaign at http://alltrials.net . Hopefully, other scientists will follow suit and demand this be standard practice.

      Delete
  2. As I understand it, Reinhart-Rogoff never shared their data before. According to Dean Baker, from 2010, "Mr Rogoff and Ms. Reinhart have declined to adhere to standard ethics within the economics profession and have refused to share the data on which they base their conclusion with other researchers."

    So, it seems that people did attempt to replicate the data at the time, but were unable to do so. What seems curious to me is that, if sharing data is standard ethics for academic economists, why the Reinhart-Rogoff didn't do so back then, and why they chose to do so now.

    http://www.cepr.net/index.php/blogs/beat-the-press/not-following-professional-ethics-matters-also

    ReplyDelete
    Replies
    1. I was not aware that they refused to share their data, which is even more troubling.

      Delete
    2. More information can be seen here:

      http://en.wikipedia.org/wiki/Growth_in_a_Time_of_Debt

      Delete
    3. Robert Samuelson, a politically moderate economist, responds to Krugman and Dean Baker here.

      According to Samuelson, after fixing the error, one of its main conclusions still stands: High debt and low economic growth often go together.

      In any event, Samuelson says neither austerity nor big, new stimulus is going to occur:

      With some exceptions, most advanced countries, including the United States, seem caught in a similar trap. Their debt/GDP ratios are high and rising, so it’s hard to embrace massive deficit-financed stimulus programs. But austerity programs of spending cuts and tax increases may dampen growth and raise debt/GDP ratios. There is no obvious exit from this dilemma except a burst of spontaneous growth, which is conspicuous by its absence.

      Delete
    4. High debt and low economic growth often go together.

      But which causes which, if there is any causation at all?

      Dean Baker's reply:

      http://www.cepr.net/index.php/blogs/beat-the-press/robert-samuelson-tries-to-salvage-reinhart-rogoff-and-austerity

      Delete
    5. Are you the same David who drones about facts and opinions in another comment elsewhere on this site today??

      I only ask because what you see fit to draw attention to in quoting Samuelson is strictly opinion. But you know that, right?

      And of course you're happy to pretend Samuelson somehow contradicts Krugman and Baker in saying "high debt and low economic growth often go together."

      But of course that *doesn't* contradict Krugman or Baker, both of who are aware of this monumentally trivial observation, which only a fool could claim rises to the level of a "main conclusion."

      Unlike you, Krugman and Baker have noted that R&R's work does literally nothing to establish that debt *causes* lower growth, the false conclusion which R&R were quite happy to see drawn from their work up until now, never bothering to correct the many, many pundits who (ab)used their fundamentally flawed research to claim that exceeding 90% GDP debt would inexorably lower growth.

      You are useless, David. Will you ever shut up, or must you forever go on pretending you have something to contribute?

      Delete
    6. Samuelson is not an economist, and he really isn't moderate.

      David, you do understand, we assume, that the debt-to-GDP ratio automatically increases in a recession as a matter of simple arithmetic? That is cause-and-effect, and it's not a matter of opinion. A ratio has a numerator and a denominator. The numerator in the ratio increases because tax revenues go down and safety net expenditures increase automatically, while the denominator declines because GDP is depressed. Presto: the ratio gets higher.

      Now that's not opinion, it's fact. What is raw opinion is saying a high debt ratio slows growth. Empirical examples defy any such causal effect -- Japan, the U.S. in the late 1940s, Great Britain throughout most distinguished parts of its history, Germany in the 1930s and today -- and there isn't even anything more than the weakest speculative theory to support causation going in that direction.

      Delete
  3. Please, it was one graduate student and two professors.

    ReplyDelete
  4. Why didn't other professors check their work? A reason that applies generally is that re-doing someone else's work tends not lead to a publishable paper. Academic journals generally want something new to publish. The pressure to publish pushes academics not to go the the major effort of checking someone else's paper.

    Some years ago I was on the Committee on Review of Papers for my professional society. I recall working and re-working one paper, with numerous letters to the author. At the end, he had a publishable paper. With the same amount of effort, I could have written a paper of my own, rather than be an anonymous reviewer. In short, reviewing the full details of someone else's paper is a lot of hard work and it's there's no reward.

    ReplyDelete
    Replies
    1. "Some years ago I was on the Committee..."

      Cue collective [SNORE]

      No, just kidding!! Your contributions are extraordinarily valuable, of course. We all benefit from the experience you deign to share with us daily, David.

      Please, do go on!

      Delete
    2. DAinCA, You say you were a member of a committee of your professional society. There's a professional society of incorrigible ignoramuses who post nonsense on political blogs?

      I learn something new every day.

      Delete
  5. Michael Ash's father is the distinguished mathematician J Marshall Ash.

    ReplyDelete
  6. Researchers do take the time to confirm results when a paper make highly unusual or revolutionary claims. It is part of the skepticism of science.

    R&R's paper was published for the first time in a special section of the journal reserved for Unrefereed work. They never bothered submiting the paper again to refereed publication.

    New Deal 2.0 covered this failure extensively. R&R refused many requests to share data. Harvard should be pressured to discipline them.

    ReplyDelete
  7. When one's work is shredded by peers, there is only one response possible.

    "Fools! I'll show them all!"

    ReplyDelete
  8. I don't understand the question why other professors didn't check their work. The fact that they refused to share their data has been part of the story from the beginning. Lots of professors criticized their findings from the beginning, too.

    The better question: why did prominent journalists and policy makers (and some ideological professors) in effect run interference for R&R, giving them such overriding if false credibility that R&R could afford such professional arrogance?

    ReplyDelete