Limning Minnesota: Ripley bungled these topics too!

FRIDAY, NOVEMBER 1, 2013

In the U.S., the improvement was general: Was Minnesota number two in the teaching of math around the year 2007?

That’s what Amanda Ripley seems to say in her egregious but ballyhooed book, The Smartest Kids in the World. But alas! This and several other claims about Minnesota are either wrong or grossly misleading.

Once again, we present Ripley’s basic text on this topic:
RIPLEY (page 72): Of the three American students I followed, Eric was the only one who did not loathe math. Coincidence or not, Eric’s home state of Minnesota was one of only two states that came close to achieving world-class math performance. Roughly speaking, Minnesota ranked below just a dozen other countries (including Canada, Korea and Finland) in math proficiency; only Massachusetts did better in the United States.

When Eric arrived in Korea, he had a solid math background. There were lots of reasons for this: One might have been that his timing was good. Had he been born earlier, things might have turned out different.

In 1995, Minnesota fourth graders placed below average for the United States on an international math test. Despite being a mostly white, middle-class state, Minnesota was not doing well in math. When Eric started kindergarten two years later, however, the state had smarter and more focused math standards. When he was eleven, Minnesota updated those standards again, with an eye toward international benchmarks. By the time he went to high school, his peers were scoring well above average for the United States and much of the world. In 2007, Minnesota elementary students rocked a major international math test, performing at about the same level as kids in Japan.

What was Minnesota doing that other states were not? The answer was not mystical. Minnesota had started with a relatively strong education system. Then they’d made a few pragmatic changes, the kind of common sense repairs you would make if you believed math was really, truly important—and that all kids were capable of learning it.
In that first paragraph, Ripley seems to say that Minnesota trailed only Massachusetts in the teaching of math in or around the year 2007. Almost surely, that claim is untrue. For the data, see yesterday’s post.

Ripley seems to make two other claims in the passage we’ve posted. Let’s present those apparent claims. Each claim is extremely hard to defend:

Minnesota was below the national average in 1995: Ripley seems to say that Minnesota was performing below the national average in math in 1995, at least on the Grade 4 level.

“Minnesota was not doing well in math,” she gloomily writes. She bases this claim on only one measure—the 1995 TIMSS math test, Grade 4 only.

Between 1995 and 2007, Minnesota showed unusual progress: Ripley seems to say that Minnesota made unusual progress in math between 1995 and 2007.

Again, a careful reader may note that Ripley is only discussing “elementary” students. In context, this makes little sense, since she is discussing the preparation of a high school student for his math study abroad.

But the state which “was not doing well in math” is now said to have “rocked a major international math test.” “What was Minnesota doing that other states were not?” Ripley immediately asks. In the next few pages, she discusses the way Minnesota had improved its math teaching.

So how about it? Are those claims true? Let’s examine them one at a time.

Was Minnesota below the national average in 1995?

Was Minnesota below the national average in math in 1995? Almost surely, it wasn’t. At this point, Ripley is discussing the state’s “aggregate” scores—its scores for the student population as a whole.

Was Minnesota below the national average? Here are the scores the state recorded on the National Assessment of Educational Progress (the NAEP) in 1996. (The NAEP wasn’t given in 1995.)

On the NAEP scale, ten points is often compared to one academic year. For all NAEP scores, start here:
Grade 4 math, 1996 NAEP, all students:
Minnesota 232
United States 222

Grade 8 math, 1996 NAEP, all students:
Minnesota 284
United States 271
In 1996, the state was outscoring the national average in both grades by a substantial margin. In Grade 4, it was the second highest scoring state. In Grade 8, it was third best.

This same pattern obtained in 1992, the previous year in which the NAEP math test was given. Despite Ripley’s gloomy pronouncement, it’s very unlikely that Minnesota was ever performing below the national average in math.

How did Ripley go wrong on this point? Let us count the ways:

First, she relied on one measure alone, the 1995 TIMSS. She chose to ignore the NAEP’s more voluminous data. (This general pattern of picking and choosing data is found all through her book.)

Beyond that, she referred to Grade 4 only. Minnesota’s eighth-graders scored well on the 1995 TIMSS. Ripley didn’t mention that fact.

Finally, she put her thumb on the scale with that “below average” claim. As noted earlier, Minnesota’s fourth graders were only two points below the national average on the TIMSS, on a test where the standard deviation is 100 points!

Plainly, that’s a statistical tie. It’s absurd, and grossly misleading, to say that Minnesota’s fourth graders “placed below average for the United States” on that particular test.

Minnesota “was not doing well in math” in 1995? Ripley put her thumb on the scale three different ways, creating a gloomy story about Minnesota’s performance.

Why didn't Minnesota's fourth graders outscore the nation on the 1995 TIMSS? If we had to guess, we would guess that the TIMSS may have selected a slightly bad sample of Minnesota’s fourth graders that year. Sampling error does happen! But there’s no excuse for Ripley’s failure to consult and present the full range of data—unless she was trying to construct a thrilling before/after story.

Did Minnesota show an unusual amount of progress from 1995 to 2007?

In a word, no! Again, let’s consult the NAEP data, along with those from the TIMSS.

Please understand: Minnesota did show progress in math during the years in question. But Minnesota’s progress was matched by that of the United States as a whole.

There was nothing unusual about the amount of progress Minnesota recorded during this era. With regard to this matter, Ripley constructed a gloomy tale which disappeared the gains of the other states.

Sticking with aggregate scores, here are the relevant scores on the Grade 4 NAEP. As you can see, Minnesota’s progress on the NAEP was matched by that of the nation:
Grade 4 math, 1996 NAEP, all students:
Minnesota 232
United States 222

Grade 4 math, 2007 NAEP, all students:
Minnesota 247
United States 239
Minnesota gained 15 points during this period, but the nation gained 17! A similar pattern obtained at Grade 8:
Grade 8 math, 1996 NAEP, all students:
Minnesota 284
United States 271

Grade 8 math, 2007 NAEP, all students:
Minnesota 292
United States 280
Minnesota gained eight points. The United States gained nine!

If you disaggregate the scores, the same general pattern obtains. There is one important difference, of course. In 2007, Minnesota’s minority students scored right around the national average for their group. Minnesota’s performance was very ordinary with its minority kids.

Let’s review the things Ripley said about Minnesota's performance in math, circa 1995 through 2007:

She said the state was number two in 2007, behind only Massachusetts. As we showed yesterday, that’s only true because Minnesota had a higher percentage of white students than other successful states. Minnesota was outscored among all demographic groups by Massachusetts and by several other states. Beyond that, the performance of Minnesota's minority kids was extremely average.

Ripley said Minnesota “was not doing well in math” in 1995. (She was referring to the state’s aggregate scores.) Almost surely, that claim isn’t true. It wasn’t even true on the Grade 4 TIMSS, her only source for the claim. (On the Grade 4 TIMSS, Minnesota kids matched the national average.)

Finally, Ripley seemed to suggest that Minnesota showed an unusual amount of improvement in math from 1995 through 2007. That doesn’t seem to be true. Minnesota’s math scores did go up, even after disaggregation, but national math scores were going up too. Ripley conducted her latest scam by failing to mention that fact.

Ripley’s work would be a scandal if we lived in a nation with a real journalistic and academic culture. But as is obvious by now, we don’t live in any such nation.

In our nation, elite social clubs present themselves to the world as “think tanks” or “foundations.” They accept large sums from corporate elites, then produce studies and books which follow preferred elite narratives.

Ripley’s book is an ungodly mess, but everyone knows they mustn’t say so. “Education writers” rushed into line to heap praise on her terrible work. On cable, hacks like Maddow hand you crap and throw black kids under the limo.

MSNBC doesn’t care about low-income kids! It wastes your time on oodles of silly piddle. Freed from any threat of correction, Ripley produced a hopelessly bungled, gloomy book which followed elite requirements.

Next: Sadly, much more remains to be said

6 comments:

  1. On the 1995 TIMSS Minnesota's fourth graders got a 516. The USA as a whole got 518. (Average is 500, so they were above average).

    On the 2007 TIMSS tests Minnesota's fourth grade got a 554. A great improvement -- 34 points -- from its 1995 score (which in all honesty were above average to begin with). In 2007, teachers in Minnesota reported that they now devoted 60 per cent more time to teaching math (test prep?) in fourth grade. These figures were trumpeted by U of Michigan Professor William H. Schmidt (of U Mich), as vindicating his revised Minnesota curriculum.

    However, on the 8th grade 1995 TIMSS test Minnesota got a 518 (same as rest of country, which was somewhat above average) and on the 2007 tests the eighth grade got 532. So Professor Hanushek's new curriculum, if that's what it was, produced a 14 point improvement but did not work miracles. Just below Japan? I don't think so. Japan got a 570. (The US eighth-grade average was 508.)

    This pamphlet comes from Dr. Schmidt at U Mich., who also consults on the Common Core (I believe). http://www.scimathmn.org/timss/timss2007/2007timss_report.pdf

    However, others have pointed out that Schmidt himself engages cherry picking, since he ignores all other scores in between, not to say other tests besides the TIMSS which makes him look so good.

    In general in the USA math scores tend to go down as pupils age because (probably) as one progresses in math it becomes becomes more like reading and languages, subjects for which the US has not figured out how to raise its test scores. (More foreign language study, anyone? Music? Arts. Poetry? That might do it.) - E

    ReplyDelete
  2. Diane Ravich was on the Daily Show a couple of days ago, making the same point TDH so often does: that it is a lie that U.S. students do so badly compared to other countries. For some reason TDH doesn't like her, I'm not sure why.

    ReplyDelete
    Replies
    1. From what I can tell, Somerby thinks Ravich is a self-aggrandizing phony.

      Delete
    2. He has been offended by the way she manipulates data to suit her own preferred message. You need to understand that Somerby wants people to be honest about facts, not agree with a particular point of view, especially when they are journalists or academics who are supposed to be devoted to a pursuit of knowledge. Did you miss his post about truth?

      Delete
    3. Anonymous November 1, 2013 at 7:58 PM:

      Leave now.

      Delete
  3. In an article in Forbes, Diane Ravitch has repeated the story of Minnesota's 34 point jump, crediting Professor Schmidt's new curriculum with the change.

    She has long favored changes in the math curriculum (along with a majority of math teachers) and is a former colleague of Dr. Schmidt. However, she is not a supporter of union busting or high stakes testing -- or developmentally inappropriate teaching methods, as are the soi-disant "reformers."

    ReplyDelete