Limning Minnesota: Rocking the TIMSS!

WEDNESDAY, OCTOBER 30, 2013

Ripley’s first round of embellishments: Did public school students in Minnesota “rock a major international test” in 2007?

That’s what Amanda Ripley says in her widely praised book, The Smartest Kids in the World. Just to refresh you, here’s the passage in which she makes that assessment:

RIPLEY (page 72): Of the three American students I followed, Eric was the only one who did not loathe math. Coincidence or not, Eric’s home state of Minnesota was one of only two states that came close to achieving world-class math performance. Roughly speaking, Minnesota ranked below just a dozen other countries (including Canada, Korea and Finland) in math proficiency; only Massachusetts did better in the United States.

When Eric arrived in Korea, he had a solid math background. There were lots of reasons for this: One might have been that his timing was good. Had he been born earlier, things might have turned out different.

In 1995, Minnesota fourth graders placed below average for the United States on an international math test. Despite being a mostly white, middle-class state, Minnesota was not doing well in math. When Eric started kindergarten two years later, however, the state had smarter and more focused math standards. When he was eleven, Minnesota updated those standards again, with an eye toward international benchmarks. By the time he went to high school, his peers were scoring well above average for the United States and much of the world. In 2007, Minnesota elementary students rocked a major international math test, performing at about the same level as kids in Japan.

What was Minnesota doing that other states were not? The answer was not mystical. Minnesota had started with a relatively strong education system. Then they’d made a few pragmatic changes, the kind of common sense repairs you would make if you believed math was really, truly important—and that all kids were capable of learning it.

In that passage, Ripley says Minnesota “was one of only two states that came close to achieving world-class math performance” at the time in question. She stresses the improvement Minnesota made between 1995 and 2007, when she says it rocked that major international test.

The test in question is the TIMSS, the Trends in International Mathematics and Science Study. Did Minnesota really rock that test?

We wouldn’t necessarily say so. Let’s start by noting one major qualification:

The TIMSS tests students in Grade 4 and Grade 8. If you read carefully, you will see that Ripley only claims that Minnesota rocked the TIMSS at the Grade 4 level.

Did the state actually do that? On the TIMSS scale, 500 is set as the average score, with a standard deviation of 100.

These are the Grade 4 scores in 2007. Not all the world’s countries took part. We are including every country which scored above 510:

Grade 4 math, 2007 TIMSS:
Hong Kong 607
Singapore 599
Taiwan 576
Japan 568
Minnesota 554
Kazakhstan 549
Russia 544
England 541
Latvia 537
Netherlands 535
Lithuania 530
United States 529
Germany 525
Denmark 523
Australia 516
[...]
International average 500

Did Minnesota rock that Grade 4 test? It’s pretty much as you like it! The state scored behind all the Asian tigers, ahead of all the countries in Europe which took part.

(Finland didn’t show up.)

In Grade 8, Minnesota scored 532. The state still trailed all the Asian nations, in this case by larger margins. (Korea took part in Grade 8, scoring 597.) It still outscored all participating European countries.

We wouldn’t say that Minnesota rocked that Grade 4 test. Still, the state outscored the United States and all European nations. This brings us to Ripley’s second claim, concerning Minnesota’s alleged large improvement over the course of twelve years.

In her book, Ripley goes on for several pages, describing the efforts which explain why Minnesota improved so much from 1995 to 2007. But how much did the state improve in its math performance?

In some ways, not that much! Or at least, no more than many states did.

Below, you see Minnesota’s scores (and those of the United States as a whole) on the TIMSS in the years in question. Minnesota’s score improved by 38 points at Grade 4, but by only 14 points at Grade 8:

Minnesota, Grade 4 TIMSS, math:
1995: 516 (U.S. 518)
2007: 554 (U.S. 529)

Minnesota, Grade 8 TIMSS, math:
1995: 518 (U.S. 492)
2007: 532 (U.S. 508)

In our view, Ripley’s portrait falters a bit as we look at these numbers.

According to Ripley, “Minnesota was not doing well in math” in 1995. But in both grades tested on the TIMSS, Minnesota outscored the international average that year, which is always set at 500.

At Grade 8, Minnesota also outscored the national average, by a healthy 26 points.

Twelve years later, Minnesota’s eighth graders produced a higher score on the TIMSS—but so did the U.S. as a whole. (This means that quite a few other states had also shown improvement.) If we rely on TIMSS scores alone, Minnesota’s fourth-graders gained ground on the nation during this period, but the state’s eighth-graders did not.

Why then does Ripley imply, in the fourth paragraph quoted above, that Minnesota showed much more improvement than other states during this twelve-year period? She writes:

“What was Minnesota doing that other states were not?” We’re not sure! But at the eighth-grade level, the fifty states gained an average of 16 points in math while Minnesota was gaining 14!

Ripley also writes this about Eric: “By the time he went to high school, his peers were scoring well above average for the United States and much of the world.” In fact, the state’s eighth-graders were always well above average for the U.S., by a consistent amount. Minnesota’s relative standing hadn’t changed during the twelve years in question.

As a general matter, we don’t think Ripley’s portrait is justified by those TIMSS scores. We don’t see Minnesota doing all that poorly in 1995. Over the next twelve years, we don’t see Minnesota making the type of unparalleled progress that Ripley’s portrait suggests.

Then too, there’s one last point. Did Minnesota’s fourth-graders really “place below average for the United States” in 1995?

Technically yes, they did! The United States scored 518 that year; Minnesota scored 516. But on the TIMSS scale, that is an utterly trivial difference. It’s silly to say that Minnesota “placed below average” for the U.S., except to set up a good story.

That two-point difference is utterly trivial. If you doubt that, look at the way Ripley treats the twelve-point difference between Japan and Minnesota in 2007.

In 2007, Japan outscored Minnesota by 12 points. According to Ripley, this meant that Minnesota was “performing at about the same level as kids in Japan.”

She took a different approach in 1995. In that year, the U.S. outscored Minnesota by only two points. This meant that Minnesota “placed below average for the United States.”

Alas! All through this widely praised book, thumbs appear on many scales as Ripley constructs her stories. In this case, we see no clear sign that Minnesota improved much more than other states did, a key part of Ripley’s portrait. We even find ourselves daring to wonder if the TIMSS might perhaps have gotten a bad sample of Minnesota’s fourth graders in 1995!

Luckily, other data exist; as usual, Ripley makes no attempt to consult them. Why bother? She gets a good story from the TIMSS, Grade 4 level only.

Additional data do exist. They help us answer the questions Ripley poses:

Is Minnesota really our number two state in the teaching of math? Did Minnesota really show more improvement down through the years than the other states did?

It’s stunningly silly to make such claims without consulting the full range of data. Tomorrow, we’ll start to look at NAEP scores for the period in question. When we do, we’ll see Ripley’s embellished claims continue to break down.

Here's the good news: Many states seemed to improve in math during that twelve-year period! As is almost always the case, the actual story seems a bit brighter than the tale Ripley tells.

One final, extremely basic point: To the extent that Minnesota can be ranked number two in math, it’s only because the state has lots of white kids. Throughout her book, Ripley seems completely oblivious to such basic factors.

So do the tools who reviewed her book. You live in a post-journalistic world, a fact the reviews make clear.