AMANDA RIPLEY’S BELIEVE IT BY LAW: You might almost think you’re getting conned!

THURSDAY, SEPTEMBER 26, 2013

Part 4—“Miracle,” Ripley said: Amanda Ripley’s The Smartest Kids in the World is an interesting, highly readable book.

That said, an instructive moment occurs on page 126. It’s part of a two-page sub-section bearing this headline: “The Polish miracle.”

In fairness, it’s important to know what Ripley means when she refers to this particular “miracle.” She doesn’t say that Poland has miraculously become one of the world’s highest-scoring countries, although every reviewer in the country seems to think that’s what she said.

(Toch and White in the Washington Monthly: “Enter journalist Amanda Ripley [who] follows three American exchange students to high schools in three of the top-scoring PISA countries: Finland, South Korea, and Poland.” Quite plainly, Poland is not “one of the top-scoring PISA countries.” Nor does Ripley ever quite say that.)

According to Ripley, Poland’s miracle is different—it’s a miracle of rapid improvement. This is the instructive passage which helps readers accept Ripley’s uplifting, novelized portrait:
RIPLEY (page 127): Like the United States, Poland was a big country whose people distrusted the centralized government. Yet something remarkable had happened in Poland. From 2000 to 2006, the average reading score of Polish 15-year-olds shot up by 29 points on the PISA exam. It was as if Polish kids had somehow packed almost three-fourths of a school year of extra learning into their brains. In less than a decade, they had gone from below average for the developed world to above. Over the same period, U.S. scores had remained flat.
“Like the United States, Poland was a big country whose people distrusted the centralized government?”

For the record, Poland is “a big country” of 38 million people. In Ripley’s account, it is a “homogeneous place with few immigrants or racial minorities.”

By way of contrast, the United States is “a big country” of 315 million people! With that rather facile comparison, Ripley helps us gasp at the way the one big country improved while its near-twin didn’t.

That’s a fairly silly construction. But that isn’t the passage we have in mind.

In the paragraph we have quoted, we were struck by what Ripley said about the “miracle” of Poland’s improvement on the PISA, the only international testing program whose scores she discusses in her book.

Wow! “From 2000 to 2006, the average reading score of Polish 15-year-olds shot up by 29 points on the PISA exam,” Ripley writes. She says that represents “almost three-fourths of a school year of extra learning,” encouraging us to gasp at “the Polish miracle.”

In fact, that was a large score gain—from a very low starting point, it should be said. But we couldn’t help noting Ripley’s rather peculiar time frame.

Ripley told us how much Poland improved from 2000 to 2006. But how odd! The most recent PISA test results come from 2009. Why didn’t Ripley give us the full enjoyment which would inevitably result from making a full nine-year comparison?

Perhaps you can guess at the answer! These are Poland’s average scores in reading over that nine-year period:
Average score, Poland, PISA reading test
2000: 479
2003: 497
2006: 508
2009: 500
Oops! The six-year gain was 29 points—but the nine-year gain was 21! With a “miracle” to sell, Ripley disappeared the drop in scores on the most recent test.

(American students also averaged 500 on the 2009 reading test.)

What happened to “the Polish miracle” after 2006? Why did Poland’s average score take that eight-point drop? We have no way to answer that question, but data sometimes jump around that way. Some possible explanations:

The average score of 508 in 2006 may have been misleadingly high, the result of some sort of sampling imperfection. On the other hand, the average score from 2009 may be misleadingly low, for the same sort of reason. (Data from the 2012 PISA haven’t been released yet.)

Alas! There’s no such thing as perfect data, even from the best testing programs. That’s why it’s generally better to look at data from three testing programs where this can be done, instead of restricting ourselves to one source of data, as Ripley does in her somewhat suspiciously slanted book.

Why has Poland’s average reading score jumped around so much? According to Ripley, that first big jump, in 2003, largely resulted from a change in basic procedure. After the 2000 testing, Poland delayed “tracking” students into vocational schools until they were 16. All students stayed in “academic” programs until after they’d taken the PISA.

Does that explain that first big jump in scores, as Ripley seems to think? Let’s assume it does. Subsequent to that one large jump, Poland recorded exactly three points in improvement from 2003 to 2009, years which followed this basic procedural change.

In that same six-year period, the American average score in reading went up by five points, from 495 to 500. But because of the time frame Ripley selected, we are told that Poland is performing a “miracle” of ascent.

Can we talk? Neither the United States nor Poland is producing any “miracles” at the present time. In fairness, miraculous Finland isn’t producing any “miracles” either.

That said, education writers have always loved the con the “miracle” metaphor permits. A bit later, still discussing Poland’s ascent, Ripley seems to play us again.

In this passage, Ripley describes the “shocking” gains Poland achieved on the PISA between 2000 and 2003. This is the passage we weren’t able to quote in full in yesterday’s post:
RIPLEY (page 135): [I]n 2000, Polish fifteen-year-olds took the PISA...

No one in Poland had expected to lead the world, but the results were disheartening all the same. Polish fifteen-year-olds ranked twenty-first in reading and twentieth in math, below the United States and below the average for the developed world. Once again, Poland had found itself on the outside looking in. If the vocational students were evaluated separately, the inequities were startling. Over two-thirds scored in the rock-bottom lowest literacy level.

Three years later, a new group of Polish fifteen-year olds took PISA. They had spent their elementary years in the old system but were by then attending the new gymnasia schools. Unlike their predecessors, they had not yet been tracked. They were the experimental group.

The results were shocking—again. Poland, the punch line for so many jokes around the world, ranked thirteenth in reading and eighteenth in math, just above the United States in both subjects. In the space of three years, Poland had caught up with the developed world. (Ripley’s italics)
In this passage, we’re being sold a thrilling story. But look how little respect Ripley seems to have for American readers!

As readers, we’re supposed to find it “shocking” when Poland ascends from twentieth place in the world in math all the way up to eighteenth! In fact, the PISA Data Explorer seems to show that Poland ranked 21st in the world in math in both those years. (Click here, then continue clicking.) But why would a reader find it “shocking” to see a country’s ranking change by two places over a span of three years?

In fact, Poland did make a large gain in average score in math from 2000 to 2003—a gain of some twenty points. Again, Ripley largely attributes the gains in 2003 to the country’s one-time change in “tracking” procedure.

That was a large one-time gain. But from 2003 to 2009, Poland’s average score in math went up by only five points. The average score for the United States went up by four points during that same time period.

Poland may well be doing good things in its schools as it emerges from decades under the Soviet thumb. That said, there is no obvious “miracle” of ascent transpiring there.

On the other hand, test scores have been rising rather rapidly in this country on the National Assessment of Educational Progress (NAEP) once you “disaggregate” scores, as everyone knows you must. But that improvement wasn’t reported, assessed or explained in Ripley’s suspiciously slanted book.

Ripley completely skips the NAEP, which seems to show so much U.S. improvement. She also skips the TIMSS and the PIRLS, international tests which seem to paint a gloomier portrait of Poland’s ongoing performance.

Ripley asks us to gasp at Poland’s score gain in reading from 2000 to 2006. She doesn’t mention the drop in Poland’s score in 2009.

You might almost think you’re getting conned. But why would Ripley do that?

Tomorrow: Ripley, past and present

10 comments:

  1. Why? Her book was funded by Bill Gates and publicized by the Gates echo chamber, which has a vested interest in making America look bad so they can "reform" it by getting rid of unions and paying workers a pittance. --Ellen

    ReplyDelete
    Replies
    1. Bob knows that. His point is really that ...[CeceliaMC?]

      Delete
    2. Ripely needs to be barefoot, pregnant, and in the kitchen.

      Oh, and a WASP.

      Delete
    3. What does Bill Gates have against unions?

      Delete
    4. I think Ellen and her friends replying here are wrong about Ripley's book and unions.

      This book gives me hope that we can create education systems of equity and rigor--if we heed the lessons from top performing countries and focus more on preparing teachers than on punishing them.

      Delete
  2. Whatever the imperfections of TDH, I have to say that no other blog has had as great an effect on the way I think about media coverage of education.

    Case in point: on the way home I was listening to the "smart network" NPR and they were reporting on the "depressing" news about SAT scores. Apparently half of test takers are not college ready. Wow, that sounds bad! But has there been any change in the population taking the test in recent years? Have more low-performing students been encouraged to take the test? As TDH would put it, "we have no way of knowing," at least as far as NPR is concerned. And the proposed solution? Wait for it.... Higher standards! If our students are under-performing it must be because they are sandbagging. If we crack the whip they'll certainly rise to the challenge.

    The script never dies.

    ReplyDelete


  3. OMB (The Good , The Bad, and the Foolish)

    Good Bob: "Can we talk? Neither the United States nor Poland is producing any “miracles” at the present time."

    Bad Bob: "On the other hand, test scores have been rising rather rapidly in this country on the National Assessment of Educational Progress (NAEP) once you “disaggregate” scores, as everyone knows you must.

    Foolish BOB: "But that improvement wasn’t reported, assessed or explained in Ripley’s suspiciously slanted book..... You might almost think you’re getting conned."

    NAEP’s long-term trend assessment has tracked performance in reading, mathematics, science, and writing. What do the data show? With some fluctuations, scores have changed little in over a generation.

    Evaluating the impact of these changes depended more on the eye of the beholder than on systematic analysis. The 2004 trend report generated a parade of upbeat, promotional press releases. Even though most scores had changed little and gains were modest, the U.S. Department of Education and leading educational organizations trumpeted the success of school reform.

    Such claims were sometimes true in a strictly numerical sense, but overall they painted a misleading picture. Even at younger ages, reading scores were only a few points higher in 2004 than in 1971 and, as the graphs showed, stable scores would have been a fairer description. The case in math was a stronger one, but the jumps at the younger ages were anomalous. The focus on younger students disregarded high school achievement, which was certainly not at all-time highs. Over 30 years, high school students’ scores had changed little. In 2004, their reading score was exactly the same as it had been in 1971, while their math score was the same as in 1992 and only 3 points higher than it was in 1973.

    "BUT WAIT, KZ", you cry, "But what about the disaggregation BOB demands?"

    Let's take reading. White high school seniors scored 297 in 1998. They scored 296 in 2009, the most recent year of available data. For black seniors there was no change in score (269) at all during the same period. Hispanics seniors, like whites, lost a point, from 275 to 274.

    We'd love to tell you BOB was right about math, but they changed the scoring system in the middle of the decadein the NAEP Data Explorer
    data we were exploring. Scores were climbing in the first half of the 90s for each group, then falling for each group in the latter half, then they changed the scoring system and scores started back up in math. Did the scoring get it get harder?, BOB might ask, We just don't know, BOB might respond. And we'll go with the latter.

    BOB is right about Amanda Ripley's book and the lack of educational imracles here and abroad, but wrong about the "risng rather rapidly" US test scores. The record is decidedly mixed, particularly when you look at our students when they are leaving the system as seniors.

    But BOB is once again the Prince of Poo when he dares to attack someone else for leaving things our of their work. I challenge anyone to go back and find a time when BOB discussed the GOLD STANDARD of tests as he loves to call NAEP with regard to results of testing high school seniors. (Hint, I can link you to it if you bait me.) He does it in a section of a post where he attacks a Harvard Professor which he had the audacity to call "The Data Which Disappeared." Sure enough, in that post, as with this, test results from high school seniors disappeared. In fact, BOB attacked the professor for using them.
    I would love BOB to chime in here and explain why they are of no use.


    KZ (Ruling Doom on the Curve)

    ReplyDelete
  4. If I remember correctly, he didn't say the "are of no use." He said they are problematic if you have a decreasing drop out rate because more of your low-performing students are in the mix. I could be wrong, but I don't have the time or energy to go combing through the archives.

    ReplyDelete
    Replies
    1. You are exactly right, cacambo. He said they were problematic for that reason in one of those types of throwaway sentences he attacks when others write them.
      When I call it a throwawa sentence it is because he used it then as his excuse for DISAPPEARING the fact.

      You don't have the time or energy to comb through the archives, and I don't want to waste the time or energy on this now dead thread to explain the foolish bias in BOB's excuse.

      KZ

      Delete