Raising Minnesota: Ripley responds!

THURSDAY, OCTOBER 24, 2013

Why did she dump the TIMSS: Amanda Ripley is a good “human interest” writer.

In her widely praised book, The Smartest Kids in the World, she creates interesting portraits of three American exchange students. Her portrait of South Korea’s “pressure cooker” educational culture is especially striking.

Ripley is good at human interest. But she’s terrible at the nuts and bolts of analyzing test scores. When she compares the scores of various states, she doesn’t even “disaggregate”—that is, she doesn’t compare different groups of kids in one state to comparable groups in the others.

For that reason, she ends up thinking that states which have the most white kids are the states which have the best schools. That is a very basic, fundamental error.

We’ll look at that problem in our next post, when we compare Minnesota’s performance in math to that of the other states. Next week, we’ll examine Ripley’s failure to consider the particular needs and circumstances of many low-income kids.

Today, let’s examine a more basic question. Why didn’t Ripley use data from the TIMSS and the PIRLS in her book?

The world’s developed nations take part in three international testing programs—the TIMSS, the PIRLS and the PISA. Presumably, all three programs are considered valuable.

But Ripley completely ignores the TIMSS and the PIRLS in her international comparisons. Because American kids have scored least well on the PISA, this tends to tilt her story in a gloomy direction.

On Tuesday, we told you what a cynic would say about this choice. A cynic would say that Ripley dumped the TIMSS and the PIRLS to produce the gloomy tale about U.S. schools that ranking “reformers” love.

Is that why Ripley ignored the TIMSS? We can’t answer that. Ripley doesn’t denounce American teachers, students and schools in the way the most aggressive “reformers” do. On the other hand, by citing the PISA alone, she produces a gloomier narrative, as preferred by current elites.

(She also overstates routinely. This starts with the largest howler in the history of books. It sits right on page 2.)

Why did Ripley deep-six the TIMSS and the PIRLS? On Tuesday, Kevin Drum linked to our post; he noted the way this choice tilts Ripley’s book toward the gloomy. In response, Ripley sent an e-mail to Drum, explaining her total reliance on the PISA.

Ripley explains this decision on pages 19-23 of her book. In our view, her explanation is embarrassingly unsophisticated. To simplify matters, let’s look at her e-mail to Drum, the text of which can be seen by all.

This is the way she begins. Warning! Gorilla dust!

RIPLEY EMAIL (10/23/13): First of all, I am excited whenever anyone is interested in a debate about TIMSS versus PISA! I geeked out on this debate for many months while working on the book, and it was a pretty lonely existence—so I am happy to have an excuse to talk about it.

I agree it is critical to be skeptical of PISA—or any test or metric. All have their flaws. Which is why I spent a long time learning about all the various international tests—studying sample questions, reading about their strengths and weaknesses and analyzing their results over time.

In fact, I used many different data points to decide which countries to feature in the book, including high school graduation rates, college attainment rates, spending per pupil, rankings of national competitiveness and other economic indicators, as well as test data from TIMSS, PIRLS and NAEP.

As it turns out, international test data is strongly correlated from one test to another. (The correlation between TIMSS 2007 and PISA 2007 was 0.93.) But as you note, there are some differences between PISA and TIMSS findings. In the end, I made a very conscious decision to prioritize PISA findings for two main reasons:

Try to ignore the pleasant asides and self-deprecation with which elites will sometimes endear themselves to other elites. Also ignore that third paragraph, which is completely irrelevant.

Ignore the wonky statement about correlation, which is completely useless in this context. As you ignore all that dust, please note one key point:

In paragraph 2, Ripley says that all tests of this type “have their flaws.” She says we should “be skeptical about the PISA—or any test or metric.”

We would tend to agree. There is no such thing as a perfect test, especially where sampling is involved.

But if all testing programs have their flaws, it seems even stranger that Ripley builds her entire book around the PISA alone, while ignoring the TIMSS and the PIRLS. This is especially true based on what we noted on Tuesday—at one point in her book, Ripley uses TIMSS results to establish Minnesota’s elite status in math.

Ripley never names the TIMSS, not even in her endnotes. But she does put TIMSS results to that use, and she refers to the unnamed TIMSS as “a major international test.”

If TIMSS results can be used that way in the case of Minnesota, why are TIMSS results AWOL when she compares the United States to Finland? Why would you disregard two-thirds of the available data?

She had three sets of data to consider. Why did she only consider one? As Ripley’s email continues, she offers these two reasons:

RIPLEY: In the end, I made a very conscious decision to prioritize PISA findings for two main reasons:

1. PISA is a test administered to 15-year-olds, which means it catches kids closer to the end of their compulsory schooling. TIMSS is given to 4th and 8th graders, which is useful, too, but I was most interested in the cumulative effects of countries' education systems, rather than the midpoint.

2. Unlike TIMSS, PISA was designed to test students' abilities to apply knowledge to solve real-world problems and think for themselves. (TIMSS is a test of school curriculum.) I was most interested in those higher-order thinking skills, since they are increasingly valuable in the modern economy. To see if the hype on PISA was true, I took the test myself, and I found it to be a remarkably sophisticated test.

We’re slicing the lunch meat amazingly thin in that first explanation. Eighth-graders and 15-year-olds are rather close in age. By the way:

In her book, Ripley uses fourth grade TIMSS scores to establish Minnesota’s elite status. If you recall, we told you to read the highlighted passage with care:

RIPLEY (page 73): In 1995, Minnesota fourth graders placed below average for the United States on an international math test. Despite being a mostly white, middle-class state, Minnesota was not doing well in math. When Eric started kindergarten two years later, however, the state had smarter and more focused math standards. When he was eleven, Minnesota updated those standards again, with an eye toward international benchmarks. By the time he went to high school, his peers were scoring well above average for the United States and much of the world. In 2007, Minnesota elementary students rocked a major international math test, performing at about the same level as kids in Japan.

Minnesota’s elementary students rocked the world on that major international test! She is referring to the state's fourth grade scores on the 2007 TIMSS.

As we noted, Minnesota’s eighth graders didn’t do as well on the TIMSS as the state’s fourth graders. That’s why Ripley qualified her claim, in a way most readers wouldn’t have noticed.

But please note: In that passage from her book, Ripley is using fourth grade scores from the TIMSS to establish Minnesota’s status. This week, in her e-mail to Drum, she says she didn’t use eighth grade scores because eighth graders are too young!

Such contradictions abound in Ripley’s work. It seems to us that Ripley cuts corners in pursuit of more thrilling stories. She may also be cutting corners, and embellishing, so that she doesn’t have to break with the preferred elite story line, in which miraculous Finland is off on a cloud and the U.S. is fumbling along.

Beyond that, Ripley simply doesn’t seem to be sophisticated with test scores; she doesn’t seem to understand the basic blocking and tackling involved. This brings us to her second point to Drum, where she says she took the PISA and “found it to be a remarkably sophisticated test.”

In our view, that’s a semi-embarrassing statement. The corresponding passage in her book is even worse. Here’s why:

As Ripley tells Drum, the TIMSS is designed as “a test of school curriculum.” It tests how much math a student knows. (Presumably, knowledge of math is important if we want kids to be able to learn more math as they continue in school.)

By way of contrast, the PISA advertises itself as a test of critical thinking. Its questions are more “arty” than those on the TIMSS.

We assume both tests have been judged to be valid and useful, or all those nations wouldn’t be taking them. The question remains: why did Ripley focus entirely on the PISA while throwing away the TIMSS?

In her book, she describes the process by which she came to accept the cult of the PISA. At one point, she decides to take the test for herself so she can better understand it.

Pouring on the human interest, she says she gave herself a pop quiz on the day of the test. Do you believe this story?

RIPLEY (page 20): I got there early, probably the only person in history excited to take a standardized test. The researchers who administered PISA in the United States had an office on K Street in downtown D.C., near the White House, wedged between the ;law firms and the lobbyists.

In the elevator, it occurred to me that I hadn’t actually taken a test in fifteen years. This could be embarrassing. I gave myself a quick pop quiz. What was the quadratic formula? What was the value of pi? Nothing came to mind. The elevator doors opened.

Here’s your test: Do you believe a single word of that second paragraph?

Ripley is a highly successful career achiever. Do you believe that she didn’t think about these matters until she found herself in the elevator that day? Do you believe she stood there trying to remember the value of pi?

Everything is possible! That said, we don’t believe that, any more than we believe that Ripley decided to write her book because her mind was blown by the chart that appears on page 3. (She misdescribes the chart on page 2, grossly overstating what it shows about Finland’s miraculous rise.)

Ripley is good with human interest. At times, this may produce results which aren't entirely truthful. Back to her adventure with the PISA:

For the next several pages in her book, she describes her experiences taking and self-scoring the test. She describes some items on the test. She describes the claims made by the test’s developers.

Finally, she draws a conclusion. The highlighted statement seems deeply clueless to us:

RIPLEY (page 23): After I left the building, my sense of relief faded. My score, I realized, did not bode well for teenagers in my own country. This test wasn’t easy, but it wasn’t that hard either. On one question that I’d gotten right, only 18 percent of American fifteen-year-olds were with me. There were other questions like that, which many or most of the Finns and the Koreans were getting right, just as I was, but most young Americans were getting wrong.

PISA demanded fluency in problem solving and the ability to communicate; in other words, the basic skills I needed to do my job and take care of my family in a world choked with information and subject to sudden economic change. What did it mean for a country if most of its teenagers did not do well on this test? Not all of our kids had to be engineers or lawyers, but didn’t all of them know how to think?

I still didn’t believe PISA measured everything, but I was now convinced that it measured critical thinking. The American Association of University Professors had called critical thinking “the hallmark of American education—an education designed to create thinking citizens for a free society.” If critical thinking was the hallmark, why didn’t it show itself by age fifteen?

Good God! Can a person determine, in that manner, if a test “measures critical thinking?” That would require a difficult statistical determination, even after you came to some agreement on what “critical thinking” is.

Does Ripley really believe that a non-professional can assess a major testing program in the manner she describes? That a person can sensibly become “convinced that it measures critical thinking” in this slapdash manner?

That passage seems very foolish to us. It helps stamp Ripley as a rank amateur in this basic part of her book.

We assume the TIMSS, the PIRLS and the PISA are all useful tests. Even if Ripley thinks the PISA is great, that doesn’t explain why she wouldn’t want to consider the TIMSS as well.

In her account, the TIMSS can tell us how well our eighth grade students know “the school curriculum” in math. In a book of 230 pages, why wouldn’t she want to include the fact that American eighth graders matched Finland in math on the 2011 TIMSS?

Why doesn’t she mention that fact in her book events?

Minnesota’s TIMSS scores from 2007 showed the state’s elite status. But American scores from 2011 aren’t worth mentioning. And remember—Minnesota’s scores were for the fourth grade. American eighth grade scores are no good. Eighth graders are too young!

Here’s the next chunk of Ripley’s email. We will make a quick point:

RIPLEY (continuing directly): You are right that American kids do better on TIMSS, especially in reading. And you are right that many people exaggerate our failings relative to other countries. It drives me nuts. Which is why I went to great lengths throughout the book to avoid such hyperbole.

I feel weird quoting myself, but just in case you don't believe me, this is from p. 4: "The vast majority of countries did not manage to educate all their kids to high levels, not even all of their better-off kids. Compared to most countries, the United States was typical, not much better nor much worse…Our elementary students did fine on international tests, thank you very much, especially in reading. The problems arose in math and science, and they became most obvious when our kids grew into teenagers...."

It’s true. Ripley doesn’t bash American students and teachers the way some reformers do. We don’t always agree with her judgments in this general area. But she doesn’t lambaste our ratty teachers with their infernal unions.

She does quote herself a bit selectively. On page 4 of her book, she also says this:

RIPLEY (page 4): The vast majority of countries did not manage to educate all their kids to high levels, not even all of their better-off kids. Compared to most countries, the United States was typical, not much better nor much worse. But, in a small number of countries, really just a handful of eclectic nations, something incredible was happening. Virtually all kids were learning critical thinking skills in math science and reading They weren’t just memorizing facts; they were learning how to solve problems and adapt. That is to say, they were training to survive in the modern economy.

How to explain it? American kids were better off, on average, than the typical child in Japan, New Zealand or South Korea, yet they knew far less math than those children.

Do American kids know far less math than children in New Zealand? If you look at the PISA, yes! But if you look at the TIMSS, you get a different result.

How did those countries score on the most recent TIMSS? In Grade 4 math, the United States outscored New Zealand by a substantial margin:

Grade 4 math, 2011 TIMSS, selected countries:
Finland 545
United States 541
New Zealand 486

These are the eighth-grade scores:

Grade 8 math, 2011 TIMSS, selected countries:
Finland 514
United States 509
New Zealand 487

The U.S. also outscored New Zealand at both grade levels in science. Those correlations between PISA and TIMSS don’t look real strong here!

The United States should have better schools. We should teach more math, and we should teach the joy of exploration. We should push our middle-class students more, and everyone else who can keep up. We should examine the needs of American kids from low-literacy backgrounds from their earliest preschool years on.

That said, Ripley is a Time magazine alumna. She is good at telling, selling and embellishing human interest stories.

Her book is often strong in its human interest aspects. But in basic ways we’ll continue to explore, it’s also an ungodly mess. American students deserve much better than this kind of scattershot work.

Next post: Misconstruing Minnesota