WHERE THE PISA GAPS ARE: Amanda Ripley wrote a book!

TUESDAY, OCTOBER 11, 2016

Previous report in this series

Part 1—Half the facts disappeared: In August 2013, a former Time magazine writer published a widely acclaimed, much discussed book about the world's public schools.

Amanda Ripley's ballyhooed book was called The Smartest Kids in the World/And How They Got That Way. One point was clear in Ripley's highly readable presentation:

As it turned out, the smartest kids in the world weren't found in the U.S.! Instead, Ripley focused on three nations whose students, she said, had scored remarkably well on international tests of reading, science and math.

Predictably, some of the smartest kids in the world were found in miraculous Finland. The small, middle-class European country had been a darling of American education writers over the prior decade. In Ripley's book, Finland became one of three countries with "the smartest kids in the world."

Ripley also found the world's smartest kids in South Korea. Again, this was no surprise. South Korea is one of three large Asian nations whose students consistently outscore the rest of the world on the the two international testing programs in which the world's developed nations take part. We refer to the Trends in International Math and Science Studies (the TIMSS) and the Program in International Student Assessment (the PISA).

(The TIMSS has a companion international reading test, the PIRLS.)

Finally, some of the smartest kids in the world were said to be living in Poland. This was perhaps a bit surprising, because this is the way Poland's kids had performed on the latest TIMSS math and science tests, whose results had been released in December 2012:

Average scores, Grade 4 math, 2011 TIMSS
United States: 541
Poland: 481

Average scores, Grade 4 science, 2011 TIMSS
United States: 544
Poland: 505

(Poland didn't participate in the TIMSS on the Grade 8 level.)

Given the way the TIMSS scale works, those were substantial score differentials. Now, Ripley seemed to be describing Poland's students as among the smartest kids in the world, with our pitiful U.S. kids sadly lagging behind.

That said, a funny thing happened to people who read Ripley's widely-praised book. Unless they were education specialists, they had no way of knowing what follows. But as they read Ripley's highly readable book, they learned about results from only one of the developed world's two major testing programs.

Ripley discussed results from the PISA. Results from the TIMSS (and the PIRLS) were, in effect, completely disappeared.

Except in a single endnote, the TIMSS was never mentioned by name at any point in Ripley's book. (The TIMSS was only glancingly named in that lone endnote. The acronym appeared, but not the name. The endnote didn't explain what the acronym meant, or what the TIMSS even is.)

Ripley did cite results from the TIMSS at one point in her book (see page 73). She referred to the TIMSS, without citing its name, as "a major international math test." But she did so only to praise the state of Minnesota for its students' high scores on this unnamed "major international test." She went on to attribute Minnesota's success on this major test to certain types of "education reform."

Ripley never mentioned the high scores achieved by American kids in general on that "major international test." In the course of her entire book, she never said its name.

Readers of Ripley's book were never told that there are two major international testing programs. They learned about results from the PISA. Except for that one targeted use, results from the TIMSS were ignored.

A funny thing happened to readers of Ripley's book. They read a book from which exactly half the international data had in effect been banned. Ripley never told her readers why she had sifted the data this way. Indeed, those readers had no way to know that half the international data were being shielded from view.

On its face, Ripley's presentation of data was strangely selective. On its face, it would be strange to sift the available data this way, without any attempt at explanation, in a 1200-word Sunday newspaper essay.

On its face, this selective sifting of data was only that much more peculiar in a 238-page book.

That said, Ripley's selective presentation had become rather common by the time her book appeared. Especially among advocates of certain types of "education reform," it was now common to do what Ripley had done—to cite results from the PISA, while skipping results from the TIMSS (and the PIRLS).

The United States, and other developed nations, participated in both testing programs. Increasingly, though, only results from the PISA were mentioned. Why in the world was that?

Two possible explanations exist. First, it's commonly said that the PISA focuses on "critical thinking" while the TIMSS focuses on "knowledge of the curriculum." On that basis, a person might judge that the PISA is the more valuable program.

That might be a perfectly valid judgment, though it isn't universally shared. That said, Ripley never explicitly made that case in her widely-praised book. Instead, the TIMSS was simply disappeared, except when its results could be used to praise "education reform."

Is that why Ripley dumped the TIMSS? Did she disappear the TIMSS because she feels the PISA is the only valid measure? Does she believe that the PISA is a substantially better measure?

We can't answer those questions. But there's another possible reason why advocates of "reform" stress the PISA while largely ignoring the TIMSS:

American students have scored less well on the PISA!

When compared to their peers in the rest of the world, American students have scored less well on the PISA. They've scored better on the TIMSS. As such, PISA scores can more fruitfully be used to portray American schools as an embarrassing mess—to argue the case for certain types of "education reform."

Is that why Ripley skipped the TIMSS? We can't answer that. What we can do—what we'll do this week—is offer an overview of the way American students have scored on the PISA.

We'll look at "aggregate" average scores on the PISA—at the average scores achieved by American students as a whole. We'll also "disaggregate" those average scores to look at the performance of major groups within our student population.

How have white American kids performed as compared to the rest of the world? How about black kids and Hispanic kids? How about Asian American students? How have these different groups of kids performed as compared to the rest of the world?

We'll explore an additional question. How have America's immigrant kids performed on the PISA? In one of the most deceptive passages we've ever seen in such a major book, Ripley gives the clear impression that Finland was achieving its usual miracles with its (very small number of) immigrant kids.

We'll show you the data which Ripley withheld. Those data will come directly from the source Ripley cited in one of the most deceptive passages we've ever seen in such a book.

How have American students performed on the PISA? It's an important question. Without forgetting results from the TIMSS, we'll explore the answers (plural) all week.

In many ways—not all—the answers may seem depressing. Within the American student population, very substantial "achievement gaps" exist on both the TIMSS and the PISA.

Those very large gaps may speak to our brutal racial history. They may speak to current educational and social practices. But however daunting those gaps may seem, they help define our ongoing educational challenges. They lead us to next week's final set of reports in this five-week series, a set of reports in which we'll examine this topic:

"Where the Challenges Are."

A final word. Most of our educational challenges are found in our classrooms and homes. But some of our educational challenges are found in the lazy, failing-grade work performed by the American press corps.

Ripley's widely ballyhooed book was highly readable. It many respects, it was highly informative.

It also involved journalistic sleights of hand which should have embarrassed a serious people. But alas! When it comes to the way they discuss our schools, it's hard to say that America's press corps can be described that way. As a group, American "education experts" aren't a whole lot more impressive.

Next week, we'll examine where the challenges are. All this week, we'll be taking a look at the PISA gaps.

Tomorrow: The aggregate scores