Limning Minnesota: Ripley bungled these topics too!

FRIDAY, NOVEMBER 1, 2013

In the U.S., the improvement was general: Was Minnesota number two in the teaching of math around the year 2007?

That’s what Amanda Ripley seems to say in her egregious but ballyhooed book, The Smartest Kids in the World. But alas! This and several other claims about Minnesota are either wrong or grossly misleading.

Once again, we present Ripley’s basic text on this topic:

RIPLEY (page 72): Of the three American students I followed, Eric was the only one who did not loathe math. Coincidence or not, Eric’s home state of Minnesota was one of only two states that came close to achieving world-class math performance. Roughly speaking, Minnesota ranked below just a dozen other countries (including Canada, Korea and Finland) in math proficiency; only Massachusetts did better in the United States.

When Eric arrived in Korea, he had a solid math background. There were lots of reasons for this: One might have been that his timing was good. Had he been born earlier, things might have turned out different.

In 1995, Minnesota fourth graders placed below average for the United States on an international math test. Despite being a mostly white, middle-class state, Minnesota was not doing well in math. When Eric started kindergarten two years later, however, the state had smarter and more focused math standards. When he was eleven, Minnesota updated those standards again, with an eye toward international benchmarks. By the time he went to high school, his peers were scoring well above average for the United States and much of the world. In 2007, Minnesota elementary students rocked a major international math test, performing at about the same level as kids in Japan.

What was Minnesota doing that other states were not? The answer was not mystical. Minnesota had started with a relatively strong education system. Then they’d made a few pragmatic changes, the kind of common sense repairs you would make if you believed math was really, truly important—and that all kids were capable of learning it.

In that first paragraph, Ripley seems to say that Minnesota trailed only Massachusetts in the teaching of math in or around the year 2007. Almost surely, that claim is untrue. For the data, see yesterday’s post.

Ripley seems to make two other claims in the passage we’ve posted. Let’s present those apparent claims. Each claim is extremely hard to defend:

Minnesota was below the national average in 1995: Ripley seems to say that Minnesota was performing below the national average in math in 1995, at least on the Grade 4 level.

“Minnesota was not doing well in math,” she gloomily writes. She bases this claim on only one measure—the 1995 TIMSS math test, Grade 4 only.

Between 1995 and 2007, Minnesota showed unusual progress: Ripley seems to say that Minnesota made unusual progress in math between 1995 and 2007.

Again, a careful reader may note that Ripley is only discussing “elementary” students. In context, this makes little sense, since she is discussing the preparation of a high school student for his math study abroad.

But the state which “was not doing well in math” is now said to have “rocked a major international math test.” “What was Minnesota doing that other states were not?” Ripley immediately asks. In the next few pages, she discusses the way Minnesota had improved its math teaching.

So how about it? Are those claims true? Let’s examine them one at a time.

Was Minnesota below the national average in 1995?

Was Minnesota below the national average in math in 1995? Almost surely, it wasn’t. At this point, Ripley is discussing the state’s “aggregate” scores—its scores for the student population as a whole.

Was Minnesota below the national average? Here are the scores the state recorded on the National Assessment of Educational Progress (the NAEP) in 1996. (The NAEP wasn’t given in 1995.)

On the NAEP scale, ten points is often compared to one academic year. For all NAEP scores, start here:

Grade 4 math, 1996 NAEP, all students:
Minnesota 232
United States 222

Grade 8 math, 1996 NAEP, all students:
Minnesota 284
United States 271

In 1996, the state was outscoring the national average in both grades by a substantial margin. In Grade 4, it was the second highest scoring state. In Grade 8, it was third best.

This same pattern obtained in 1992, the previous year in which the NAEP math test was given. Despite Ripley’s gloomy pronouncement, it’s very unlikely that Minnesota was ever performing below the national average in math.

How did Ripley go wrong on this point? Let us count the ways:

First, she relied on one measure alone, the 1995 TIMSS. She chose to ignore the NAEP’s more voluminous data. (This general pattern of picking and choosing data is found all through her book.)

Beyond that, she referred to Grade 4 only. Minnesota’s eighth-graders scored well on the 1995 TIMSS. Ripley didn’t mention that fact.

Finally, she put her thumb on the scale with that “below average” claim. As noted earlier, Minnesota’s fourth graders were only two points below the national average on the TIMSS, on a test where the standard deviation is 100 points!

Plainly, that’s a statistical tie. It’s absurd, and grossly misleading, to say that Minnesota’s fourth graders “placed below average for the United States” on that particular test.

Minnesota “was not doing well in math” in 1995? Ripley put her thumb on the scale three different ways, creating a gloomy story about Minnesota’s performance.

Why didn't Minnesota's fourth graders outscore the nation on the 1995 TIMSS? If we had to guess, we would guess that the TIMSS may have selected a slightly bad sample of Minnesota’s fourth graders that year. Sampling error does happen! But there’s no excuse for Ripley’s failure to consult and present the full range of data—unless she was trying to construct a thrilling before/after story.

Did Minnesota show an unusual amount of progress from 1995 to 2007?

In a word, no! Again, let’s consult the NAEP data, along with those from the TIMSS.

Please understand: Minnesota did show progress in math during the years in question. But Minnesota’s progress was matched by that of the United States as a whole.

There was nothing unusual about the amount of progress Minnesota recorded during this era. With regard to this matter, Ripley constructed a gloomy tale which disappeared the gains of the other states.

Sticking with aggregate scores, here are the relevant scores on the Grade 4 NAEP. As you can see, Minnesota’s progress on the NAEP was matched by that of the nation:

Grade 4 math, 1996 NAEP, all students:
Minnesota 232
United States 222

Grade 4 math, 2007 NAEP, all students:
Minnesota 247
United States 239

Minnesota gained 15 points during this period, but the nation gained 17! A similar pattern obtained at Grade 8:

Grade 8 math, 1996 NAEP, all students:
Minnesota 284
United States 271

Grade 8 math, 2007 NAEP, all students:
Minnesota 292
United States 280

Minnesota gained eight points. The United States gained nine!

If you disaggregate the scores, the same general pattern obtains. There is one important difference, of course. In 2007, Minnesota’s minority students scored right around the national average for their group. Minnesota’s performance was very ordinary with its minority kids.

Let’s review the things Ripley said about Minnesota's performance in math, circa 1995 through 2007:

She said the state was number two in 2007, behind only Massachusetts. As we showed yesterday, that’s only true because Minnesota had a higher percentage of white students than other successful states. Minnesota was outscored among all demographic groups by Massachusetts and by several other states. Beyond that, the performance of Minnesota's minority kids was extremely average.

Ripley said Minnesota “was not doing well in math” in 1995. (She was referring to the state’s aggregate scores.) Almost surely, that claim isn’t true. It wasn’t even true on the Grade 4 TIMSS, her only source for the claim. (On the Grade 4 TIMSS, Minnesota kids matched the national average.)

Finally, Ripley seemed to suggest that Minnesota showed an unusual amount of improvement in math from 1995 through 2007. That doesn’t seem to be true. Minnesota’s math scores did go up, even after disaggregation, but national math scores were going up too. Ripley conducted her latest scam by failing to mention that fact.

Ripley’s work would be a scandal if we lived in a nation with a real journalistic and academic culture. But as is obvious by now, we don’t live in any such nation.

In our nation, elite social clubs present themselves to the world as “think tanks” or “foundations.” They accept large sums from corporate elites, then produce studies and books which follow preferred elite narratives.

Ripley’s book is an ungodly mess, but everyone knows they mustn’t say so. “Education writers” rushed into line to heap praise on her terrible work. On cable, hacks like Maddow hand you crap and throw black kids under the limo.

MSNBC doesn’t care about low-income kids! It wastes your time on oodles of silly piddle. Freed from any threat of correction, Ripley produced a hopelessly bungled, gloomy book which followed elite requirements.

Next: Sadly, much more remains to be said