VISIONS OF PISA: Ravitch buys into the cult!

WEDNESDAY, DECEMBER 11, 2013

Part 3—A deeply flawed trumpeter: Routinely, reading Diane Ravitch is an extremely large challenge.

Yesterday afternoon, then again this morning, we disappeared to favorite nooks to reread parts of her new book, Reign of Error.

Along with Amanda Ripley’s much more ballyhooed groaner, Reign of Error is one of our two current “big education books.” But as a writer, Ravitch tends to be extremely jumbled. For a world-class example, see below.

That said, Ravitch also tends to be excessively partisan, whichever side of whatever fight she happens to be on:

At the start of the last decade, Ravitch was a leading voice in support of standard “education reform”—in support of standards, testing and accountability. Today, she is the leading liberal voice against the side she was recently on.

We certainly don’t doubt Ravitch’s good intentions. Much of what she writes in her new book is at least theoretically useful.

But good intentions can produce bad results, especially when those good intentions are very strongly felt. This brings us back to Ravitch’s reactions to the new PISA scores.

Yesterday, in Part 2 of this series, we saw Ravitch weirdly denying that American scores were “flat” on the 2012 PISA tests. When she gets her dander up, Ravitch tends to say such things.

Today, let’s consider the four basic lessons she says she drew from the new PISA scores. She stated these lessons at the end of a 1400-word post—a post in which she almost seems to accept the reign of the new cult of the PISA.

What lessons did Ravitch draw from the new PISA scores? We’ll quote in full from the end of her post (all emphases are ours):

RAVITCH (12/3/13): From my vantage point as a historian, here is my takeaway from the PISA scores:

Lesson 1: If they mean anything at all, the PISA scores show the failure of the past dozen years of public policy in the United States. The billions invested in testing, test prep, and accountability have not raised test scores or our nation’s relative standing on the league tables. No Child Left Behind and Race to the Top are manifest failures at accomplishing their singular goal of higher test scores.

Lesson 2: The PISA scores burst the bubble of the alleged “Florida miracle” touted by Jeb Bush. Florida was one of three states–Massachusetts, Connecticut, and Florida–that participated in the PISA testing. Massachusetts did very well, typically scoring above the OECD average and the US average, as you might expect of the nation’s highest performing state on NAEP. Connecticut also did well. But Florida did not do well at all. It turns out that the highly touted “Florida model” of testing, accountability, and choice was not competitive, if you are inclined to take the scores seriously. In math, Florida performed below the OECD average and below the U.S. average. In science, Florida performed below the OECD average and at the U.S. average. In reading, Massachusetts and Connecticut performed above both the OECD and U.S. average, but Florida performed at average for both.

Lesson 3: Improving the quality of life for the nearly one-quarter of students who live in poverty would improve their academic performance.

Lesson 4: We measure only what can be measured. We measure whether students can pick the right answer to a test question. But what we cannot measure matters more. The scores tell us nothing about students’ imagination, their drive, their ability to ask good questions, their insight, their inventiveness, their creativity. If we continue the policies of the Bush and Obama administrations in education, we will not only NOT get higher scores (the Asian nations are so much better at this than we are), but we will crush the very qualities that have given our nation its edge as a cultivator of new talent and new ideas for many years.

Let others have the higher test scores. I prefer to bet on the creative, can-do spirit of the American people, on its character, persistence, ambition, hard work, and big dreams, none of which are ever measured or can be measured by standardized tests like PISA.

For all her virtues, Ravitch is a very aggressive partisan. Consider the second lesson she draws—the one in which she declare victory over Jeb Bush and “the Florida miracle.”

It’s true! Of the three states which participated in the PISA as independent entities, Florida produced the worst aggregate scores. Below, you see some of the relevant scores from the PISA math exam.

We’ll throw in a few other jurisdictions to widen the range of comparisons:

Average scores, 2012 PISA, math
Taiwan: 560
Korea: 554
Finland: 519
Massachusetts: 514
Connecticut: 506
United Kingdom: 494
OECD average: 494
United States: 481
Florida: 467

For all scores, click here, then scroll to page 12.

Florida scored quite poorly in math, below even the U.S. average. It scored well below Massachusetts and Connecticut.

Did Massachusetts “do very well?” That’s pretty much as you like it. Ravitch likes it the way she states it, with glorious Massachusetts leaving Jeb Bush for dead.

It’s also true that Massachusetts trailed Taiwan and Korea by perhaps a bit more than one school year. (Just for the record: Massachusetts has engaged in standards and testing too.)

That said, Ravitch understands, but chose to ignore, one basic set of reasons for the aggregate scores produced by those three states:

Florida’s student population is much poorer than that in Massachusetts. Beyond that, Massachusetts has a much higher percentage of white and Asian-American kids. Those groups remain our nation’s highest scorers, although the gaps have been getting smaller.

Demographically, how do Florida and Massachusetts compare? On the 2011 NAEP, 55 percent of Florida’s eighth-graders qualified for free or reduced price lunch. In Massachusetts, the figure was 33 percent. (This is a measure of lower income. It is not a measure of poverty.)

In Massachusetts, 77 percent of eighth-graders were white or Asian-American. In Florida, the corresponding figure was 48 percent.

(In Connecticut, the numbers resembled Massachusetts: 33 percent qualified for free or reduced price lunch, 70 percent were white or Asian-American. For all these demographic data, click here, scroll to page 85.

One year later, samples of those eighth-grade student populations were taking the 2012 PISA. As everyone knows, those differences in demographics help explain the differences in those states’ PISA scores.

Ravitch knows that it doesn’t make sense to compare those aggregate scores and say nothing else. She may even know this:

On the 2013 NAEP, Florida’s black students outscored their counterparts in Connecticut in Grade 8 math. So did Florida’s Hispanic students.

Readers can’t begin to imagine such facts from those aggregate PISA scores, or from reading what Ravitch wrote. But Ravitch tends to be a somewhat unbalanced partisan, whichever side she is currently on.

Not long ago, Ravitch was overstating on behalf of “reform.” Now, she tends to overstate in the other direction.

For our money, the passage called “Lesson 2” is vintage Ravitch. Beyond that, it isn’t especially helpful, except from a partisan standpoint.

But in some ways, Ravitch’s “Lesson 1” is even more striking.

Gack! In Lesson 1, Ravitch almost seems to buy into the primacy of the new cult of the PISA. Here’s why we say that:

At the start of her lengthy post, Ravitch seems to deny that American scores on the PISA were stagnant or flat. But by the time she hits Lesson 1, she seems to be saying something quite different.

In Lesson 1, Ravitch seems to say that our lousy PISA scores “show the failure of the past dozen years of public policy in the United States.” According to Ravitch, our investment in testing, test prep, and accountability “have not raised test scores or our nation’s relative standing on the league tables.”

According to Ravitch’s new position, our test scores have stayed the same for the past dozen years! But that gloomy pronouncement is only true if we do what cult leaders want—if we only consider aggregate scores, and we only consider the PISA.

Elsewhere, American test scores have been rising. This has been happening on tests which may be more reliable than the slightly unconventional and strongly crusading PISA.

In that deeply unfortunate passage, Ravitch seems to say that American test scores have been frozen in place for the past dozen years. Her desire to damn the side she hates seems to make her adopt the frameworks of a new, unfortunate cult.

In the past week, Ravitch’s reactions have been widely recited in liberal circles. What can it mean when we in the liberal world accept such a deeply flawed trumpeter as our leading voice?

Tomorrow: Disappearing the NAEP (and the TIMSS)

A paragraph for the ages: Tomorrow, we may consider the following paragraph from Ravitch’s book. We can’t be certain, but it may be the most confusing paragraph ever composed:

RAVITCH (page 47): The film Waiting for Superman misinterpreted the NAEP achievement levels. David Guggenheim, the film’s director and narrator, used the NAEP achievement levels to argue that American students were woefully undereducated. The film claimed that 70 percent of eighth-grade students could not read at grade level. That would be dreadful if it were true, but it is not. NAEP does not report grade levels (grade level describes a midpoint on the grading scale where half are above and half are below). Guggenheim assumed that students who were not “proficient” on the NAEP were “below grade level.” That is wrong. Actually, 76 percent on NAEP are basic or above, and 24 percent are below basic. It would be good to reduce the proportion who are “below basic,” but it is 24 percent, not the 70 percent Guggenheim claimed.

That paragraph is a marvel. We’ve never been able to read it without feeling forced to take out pencil and paper. At that point, we attempt to diagram the claims it contains.

It’s hard to believe that a person can create so much confusion merely by stating three percentages, while throwing in one bogus definition. (Since when does the term “grade level” automatically “describe a midpoint on the grading scale where half are above and half are below?”)

Reading Ravitch is a challenge. What can it mean when we in the liberal world accept such a puzzling trumpeter as our leading voice?