AMANDA RIPLEY’S BELIEVE IT BY LAW: You might almost think you’re getting conned!

THURSDAY, SEPTEMBER 26, 2013

Part 4—“Miracle,” Ripley said: Amanda Ripley’s The Smartest Kids in the World is an interesting, highly readable book.

That said, an instructive moment occurs on page 126. It’s part of a two-page sub-section bearing this headline: “The Polish miracle.”

In fairness, it’s important to know what Ripley means when she refers to this particular “miracle.” She doesn’t say that Poland has miraculously become one of the world’s highest-scoring countries, although every reviewer in the country seems to think that’s what she said.

(Toch and White in the Washington Monthly: “Enter journalist Amanda Ripley [who] follows three American exchange students to high schools in three of the top-scoring PISA countries: Finland, South Korea, and Poland.” Quite plainly, Poland is not “one of the top-scoring PISA countries.” Nor does Ripley ever quite say that.)

According to Ripley, Poland’s miracle is different—it’s a miracle of rapid improvement. This is the instructive passage which helps readers accept Ripley’s uplifting, novelized portrait:

RIPLEY (page 127): Like the United States, Poland was a big country whose people distrusted the centralized government. Yet something remarkable had happened in Poland. From 2000 to 2006, the average reading score of Polish 15-year-olds shot up by 29 points on the PISA exam. It was as if Polish kids had somehow packed almost three-fourths of a school year of extra learning into their brains. In less than a decade, they had gone from below average for the developed world to above. Over the same period, U.S. scores had remained flat.

“Like the United States, Poland was a big country whose people distrusted the centralized government?”

For the record, Poland is “a big country” of 38 million people. In Ripley’s account, it is a “homogeneous place with few immigrants or racial minorities.”

By way of contrast, the United States is “a big country” of 315 million people! With that rather facile comparison, Ripley helps us gasp at the way the one big country improved while its near-twin didn’t.

That’s a fairly silly construction. But that isn’t the passage we have in mind.

In the paragraph we have quoted, we were struck by what Ripley said about the “miracle” of Poland’s improvement on the PISA, the only international testing program whose scores she discusses in her book.

Wow! “From 2000 to 2006, the average reading score of Polish 15-year-olds shot up by 29 points on the PISA exam,” Ripley writes. She says that represents “almost three-fourths of a school year of extra learning,” encouraging us to gasp at “the Polish miracle.”

In fact, that was a large score gain—from a very low starting point, it should be said. But we couldn’t help noting Ripley’s rather peculiar time frame.

Ripley told us how much Poland improved from 2000 to 2006. But how odd! The most recent PISA test results come from 2009. Why didn’t Ripley give us the full enjoyment which would inevitably result from making a full nine-year comparison?

Perhaps you can guess at the answer! These are Poland’s average scores in reading over that nine-year period:

Average score, Poland, PISA reading test
2000: 479
2003: 497
2006: 508
2009: 500

Oops! The six-year gain was 29 points—but the nine-year gain was 21! With a “miracle” to sell, Ripley disappeared the drop in scores on the most recent test.

(American students also averaged 500 on the 2009 reading test.)

What happened to “the Polish miracle” after 2006? Why did Poland’s average score take that eight-point drop? We have no way to answer that question, but data sometimes jump around that way. Some possible explanations:

The average score of 508 in 2006 may have been misleadingly high, the result of some sort of sampling imperfection. On the other hand, the average score from 2009 may be misleadingly low, for the same sort of reason. (Data from the 2012 PISA haven’t been released yet.)

Alas! There’s no such thing as perfect data, even from the best testing programs. That’s why it’s generally better to look at data from three testing programs where this can be done, instead of restricting ourselves to one source of data, as Ripley does in her somewhat suspiciously slanted book.

Why has Poland’s average reading score jumped around so much? According to Ripley, that first big jump, in 2003, largely resulted from a change in basic procedure. After the 2000 testing, Poland delayed “tracking” students into vocational schools until they were 16. All students stayed in “academic” programs until after they’d taken the PISA.

Does that explain that first big jump in scores, as Ripley seems to think? Let’s assume it does. Subsequent to that one large jump, Poland recorded exactly three points in improvement from 2003 to 2009, years which followed this basic procedural change.

In that same six-year period, the American average score in reading went up by five points, from 495 to 500. But because of the time frame Ripley selected, we are told that Poland is performing a “miracle” of ascent.

Can we talk? Neither the United States nor Poland is producing any “miracles” at the present time. In fairness, miraculous Finland isn’t producing any “miracles” either.

That said, education writers have always loved the con the “miracle” metaphor permits. A bit later, still discussing Poland’s ascent, Ripley seems to play us again.

In this passage, Ripley describes the “shocking” gains Poland achieved on the PISA between 2000 and 2003. This is the passage we weren’t able to quote in full in yesterday’s post:

RIPLEY (page 135): [I]n 2000, Polish fifteen-year-olds took the PISA...

No one in Poland had expected to lead the world, but the results were disheartening all the same. Polish fifteen-year-olds ranked twenty-first in reading and twentieth in math, below the United States and below the average for the developed world. Once again, Poland had found itself on the outside looking in. If the vocational students were evaluated separately, the inequities were startling. Over two-thirds scored in the rock-bottom lowest literacy level.

Three years later, a new group of Polish fifteen-year olds took PISA. They had spent their elementary years in the old system but were by then attending the new gymnasia schools. Unlike their predecessors, they had not yet been tracked. They were the experimental group.

The results were shocking—again. Poland, the punch line for so many jokes around the world, ranked thirteenth in reading and eighteenth in math, just above the United States in both subjects. In the space of three years, Poland had caught up with the developed world. (Ripley’s italics)

In this passage, we’re being sold a thrilling story. But look how little respect Ripley seems to have for American readers!

As readers, we’re supposed to find it “shocking” when Poland ascends from twentieth place in the world in math all the way up to eighteenth! In fact, the PISA Data Explorer seems to show that Poland ranked 21st in the world in math in both those years. (Click here, then continue clicking.) But why would a reader find it “shocking” to see a country’s ranking change by two places over a span of three years?

In fact, Poland did make a large gain in average score in math from 2000 to 2003—a gain of some twenty points. Again, Ripley largely attributes the gains in 2003 to the country’s one-time change in “tracking” procedure.

That was a large one-time gain. But from 2003 to 2009, Poland’s average score in math went up by only five points. The average score for the United States went up by four points during that same time period.

Poland may well be doing good things in its schools as it emerges from decades under the Soviet thumb. That said, there is no obvious “miracle” of ascent transpiring there.

On the other hand, test scores have been rising rather rapidly in this country on the National Assessment of Educational Progress (NAEP) once you “disaggregate” scores, as everyone knows you must. But that improvement wasn’t reported, assessed or explained in Ripley’s suspiciously slanted book.

Ripley completely skips the NAEP, which seems to show so much U.S. improvement. She also skips the TIMSS and the PIRLS, international tests which seem to paint a gloomier portrait of Poland’s ongoing performance.

Ripley asks us to gasp at Poland’s score gain in reading from 2000 to 2006. She doesn’t mention the drop in Poland’s score in 2009.

You might almost think you’re getting conned. But why would Ripley do that?

Tomorrow: Ripley, past and present