VISIONS OF PISA: Ravitch buys into the cult!

WEDNESDAY, DECEMBER 11, 2013

Part 3—A deeply flawed trumpeter: Routinely, reading Diane Ravitch is an extremely large challenge.

Yesterday afternoon, then again this morning, we disappeared to favorite nooks to reread parts of her new book, Reign of Error.

Along with Amanda Ripley’s much more ballyhooed groaner, Reign of Error is one of our two current “big education books.” But as a writer, Ravitch tends to be extremely jumbled. For a world-class example, see below.

That said, Ravitch also tends to be excessively partisan, whichever side of whatever fight she happens to be on:

At the start of the last decade, Ravitch was a leading voice in support of standard “education reform”—in support of standards, testing and accountability. Today, she is the leading liberal voice against the side she was recently on.

We certainly don’t doubt Ravitch’s good intentions. Much of what she writes in her new book is at least theoretically useful.

But good intentions can produce bad results, especially when those good intentions are very strongly felt. This brings us back to Ravitch’s reactions to the new PISA scores.

Yesterday, in Part 2 of this series, we saw Ravitch weirdly denying that American scores were “flat” on the 2012 PISA tests. When she gets her dander up, Ravitch tends to say such things.

Today, let’s consider the four basic lessons she says she drew from the new PISA scores. She stated these lessons at the end of a 1400-word post—a post in which she almost seems to accept the reign of the new cult of the PISA.

What lessons did Ravitch draw from the new PISA scores? We’ll quote in full from the end of her post (all emphases are ours):
RAVITCH (12/3/13): From my vantage point as a historian, here is my takeaway from the PISA scores:

Lesson 1: If they mean anything at all, the PISA scores show the failure of the past dozen years of public policy in the United States. The billions invested in testing, test prep, and accountability have not raised test scores or our nation’s relative standing on the league tables. No Child Left Behind and Race to the Top are manifest failures at accomplishing their singular goal of higher test scores.

Lesson 2: The PISA scores burst the bubble of the alleged “Florida miracle” touted by Jeb Bush. Florida was one of three states–Massachusetts, Connecticut, and Florida–that participated in the PISA testing. Massachusetts did very well, typically scoring above the OECD average and the US average, as you might expect of the nation’s highest performing state on NAEP. Connecticut also did well. But Florida did not do well at all. It turns out that the highly touted “Florida model” of testing, accountability, and choice was not competitive, if you are inclined to take the scores seriously. In math, Florida performed below the OECD average and below the U.S. average. In science, Florida performed below the OECD average and at the U.S. average. In reading, Massachusetts and Connecticut performed above both the OECD and U.S. average, but Florida performed at average for both.

Lesson 3: Improving the quality of life for the nearly one-quarter of students who live in poverty would improve their academic performance.

Lesson 4: We measure only what can be measured. We measure whether students can pick the right answer to a test question. But what we cannot measure matters more. The scores tell us nothing about students’ imagination, their drive, their ability to ask good questions, their insight, their inventiveness, their creativity. If we continue the policies of the Bush and Obama administrations in education, we will not only NOT get higher scores (the Asian nations are so much better at this than we are), but we will crush the very qualities that have given our nation its edge as a cultivator of new talent and new ideas for many years.

Let others have the higher test scores. I prefer to bet on the creative, can-do spirit of the American people, on its character, persistence, ambition, hard work, and big dreams, none of which are ever measured or can be measured by standardized tests like PISA.
For all her virtues, Ravitch is a very aggressive partisan. Consider the second lesson she draws—the one in which she declare victory over Jeb Bush and “the Florida miracle.”

It’s true! Of the three states which participated in the PISA as independent entities, Florida produced the worst aggregate scores. Below, you see some of the relevant scores from the PISA math exam.

We’ll throw in a few other jurisdictions to widen the range of comparisons:
Average scores, 2012 PISA, math
Taiwan: 560
Korea: 554
Finland: 519
Massachusetts: 514
Connecticut: 506
United Kingdom: 494
OECD average: 494
United States: 481

Florida: 467
For all scores, click here, then scroll to page 12.

Florida scored quite poorly in math, below even the U.S. average. It scored well below Massachusetts and Connecticut.

Did Massachusetts “do very well?” That’s pretty much as you like it. Ravitch likes it the way she states it, with glorious Massachusetts leaving Jeb Bush for dead.

It’s also true that Massachusetts trailed Taiwan and Korea by perhaps a bit more than one school year. (Just for the record: Massachusetts has engaged in standards and testing too.)

That said, Ravitch understands, but chose to ignore, one basic set of reasons for the aggregate scores produced by those three states:

Florida’s student population is much poorer than that in Massachusetts. Beyond that, Massachusetts has a much higher percentage of white and Asian-American kids. Those groups remain our nation’s highest scorers, although the gaps have been getting smaller.

Demographically, how do Florida and Massachusetts compare? On the 2011 NAEP, 55 percent of Florida’s eighth-graders qualified for free or reduced price lunch. In Massachusetts, the figure was 33 percent. (This is a measure of lower income. It is not a measure of poverty.)

In Massachusetts, 77 percent of eighth-graders were white or Asian-American. In Florida, the corresponding figure was 48 percent.

(In Connecticut, the numbers resembled Massachusetts: 33 percent qualified for free or reduced price lunch, 70 percent were white or Asian-American. For all these demographic data, click here, scroll to page 85.

One year later, samples of those eighth-grade student populations were taking the 2012 PISA. As everyone knows, those differences in demographics help explain the differences in those states’ PISA scores.

Ravitch knows that it doesn’t make sense to compare those aggregate scores and say nothing else. She may even know this:

On the 2013 NAEP, Florida’s black students outscored their counterparts in Connecticut in Grade 8 math. So did Florida’s Hispanic students.

Readers can’t begin to imagine such facts from those aggregate PISA scores, or from reading what Ravitch wrote. But Ravitch tends to be a somewhat unbalanced partisan, whichever side she is currently on.

Not long ago, Ravitch was overstating on behalf of “reform.” Now, she tends to overstate in the other direction.

For our money, the passage called “Lesson 2” is vintage Ravitch. Beyond that, it isn’t especially helpful, except from a partisan standpoint.

But in some ways, Ravitch’s “Lesson 1” is even more striking.

Gack! In Lesson 1, Ravitch almost seems to buy into the primacy of the new cult of the PISA. Here’s why we say that:

At the start of her lengthy post, Ravitch seems to deny that American scores on the PISA were stagnant or flat. But by the time she hits Lesson 1, she seems to be saying something quite different.

In Lesson 1, Ravitch seems to say that our lousy PISA scores “show the failure of the past dozen years of public policy in the United States.” According to Ravitch, our investment in testing, test prep, and accountability “have not raised test scores or our nation’s relative standing on the league tables.”

According to Ravitch’s new position, our test scores have stayed the same for the past dozen years! But that gloomy pronouncement is only true if we do what cult leaders want—if we only consider aggregate scores, and we only consider the PISA.

Elsewhere, American test scores have been rising. This has been happening on tests which may be more reliable than the slightly unconventional and strongly crusading PISA.

In that deeply unfortunate passage, Ravitch seems to say that American test scores have been frozen in place for the past dozen years. Her desire to damn the side she hates seems to make her adopt the frameworks of a new, unfortunate cult.

In the past week, Ravitch’s reactions have been widely recited in liberal circles. What can it mean when we in the liberal world accept such a deeply flawed trumpeter as our leading voice?

Tomorrow: Disappearing the NAEP (and the TIMSS)

A paragraph for the ages: Tomorrow, we may consider the following paragraph from Ravitch’s book. We can’t be certain, but it may be the most confusing paragraph ever composed:
RAVITCH (page 47): The film Waiting for Superman misinterpreted the NAEP achievement levels. David Guggenheim, the film’s director and narrator, used the NAEP achievement levels to argue that American students were woefully undereducated. The film claimed that 70 percent of eighth-grade students could not read at grade level. That would be dreadful if it were true, but it is not. NAEP does not report grade levels (grade level describes a midpoint on the grading scale where half are above and half are below). Guggenheim assumed that students who were not “proficient” on the NAEP were “below grade level.” That is wrong. Actually, 76 percent on NAEP are basic or above, and 24 percent are below basic. It would be good to reduce the proportion who are “below basic,” but it is 24 percent, not the 70 percent Guggenheim claimed.
That paragraph is a marvel. We’ve never been able to read it without feeling forced to take out pencil and paper. At that point, we attempt to diagram the claims it contains.

It’s hard to believe that a person can create so much confusion merely by stating three percentages, while throwing in one bogus definition. (Since when does the term “grade level” automatically “describe a midpoint on the grading scale where half are above and half are below?”)

Reading Ravitch is a challenge. What can it mean when we in the liberal world accept such a puzzling trumpeter as our leading voice?

17 comments:

  1. What it means is that she is doing it (thank God for that) and you and I are not!

    ReplyDelete
  2. I have to say that I also find Somerby hard to read but am glad he is providing this much needed analysis.

    Bringing the salient points (even when they are -- when you think about it -- rather obvious) forward in an easy-to-absorb way is not at all easy, that it is doubtless why people like Hanushek have found it so easy to bamboozle politicians and policy makers.

    ReplyDelete
    Replies
    1. Where are the trolls when you really need them?

      Delete
  3. "What can it mean when we in the liberal world accept such a puzzling trumpeter as our leading voice?"

    It means we need new rules for meeting of Liberal World so we can require at least a majority vote before trumpeters are selected for anything.


    Either that are we are still as paralyzed as the damned Irish.

    ReplyDelete
    Replies
    1. OMB

      What can it mean when we in the liberal world accept such a puzzling trumpeter as our leading voice?"

      Do you believe that Frankl, then a teenager, “jumped out of his chair” as he posed that question to his teacher?

      Answer either question correctly and the larger meaning of Martin Bashir will be revealed, you can attend a PISA cult meeting and strongly crusade with the excessively partisan Diane Ravitch.

      Hint: If you believe Frankl jumpers out of his chair you are more likely to believe BOB diagrams Ravitch claims with the
      pencil he is forced to take out when rereading her marvelous paragraphs.

      Delete
    2. OMB, how on Zarkon could I forget to sign?

      KZ

      Delete
    3. No need to sign, KZ. Your silliness gives you away.

      Delete
  4. I suppose one gets some information on the effectiveness of Jeb Bush's educational reforms by looking at Florida's scores relative to other places. But, it would be more meaningful to compare Florida's results before the reforms to Florida's results after the reforms.

    ReplyDelete
  5. <<(Since when does the term “grade level” automatically “describe a midpoint on the grading scale where half are above and half are below?”)>>>

    Ah, yes! Ha! Ha! Ha! Most humorous. Since when, indeed? Foolish woman!

    I'm reminded of the scene in which intergalactic super-genius Klaatu smirks so patronizingly at the higher math on Prof. Zorba's blackboard. Such naive children these earthlings are!

    Or there's Diane Keaton nonplussing Woody Allen in "Love and Death" with an explanation of being and nothingness so beautifully rendered in multilayers of quasi-philosophical claptrap only a true pseudo-intellectual could follow.

    Psst! If Ravitch is so off the mark, could someone please explain just what the agreed definition of "grade level" actually is? Meanwhile, I'll just pretend to understand and laugh along. Ho! Ho! Ho! How very funny! Superb jest! Hee! Hee!


    ReplyDelete
    Replies
    1. Grade level means ten points on the NAEP! Roughly. Very Roughly. You don't need to be Klaatu to figure that out.

      Where the heck have you been, Jeeves? Under a stump?

      KZ

      Delete
    2. I agree with Jeeves that Somerby is off base here. But instead of the kind of useless snark served up by our self-satisfied troll extraordinaire, KZ, I'll give you a useful bit of internet-gleaned insight which I hope will move the discussion forward (rather than keeping it mired in the feces flinging of KZ and his ilk).

      It seems that Ravitch is merely using a different, but equally valid definition of "grade level." If I'm right about this, then Somerby owes her an apology. (Although I have to say, I did get a little lost when she smuggled in the category "basic" without explaining it. Is basic supposed to be "grade level"?)


      From http://www.espsolutionsgroup.com/espweb/assets/files/ESP_Performing_on_Grade_Level_ORG.pdf:

      STANDARDS-BASED PERSPECTIVE: Grade level is defined as the skills and knowledge established as required for a grade level. The boundary for being on grade level is often referred to as the lowest score that classifies a student as proficient.

      [My note: the use of the word "proficient" attests to how easily things can get confusing: obviously this is not the same definition of "proficient" that the NAEP uses]

      NORMATIVE PERSPECTIVE: Grade level is defined
      as the performance level of the typical student in a grade level. Typical, in a normative sense, is the median or 50th percentile student; however, a lower percentile may be used
      to include all students who might have scored at the 50th
      percentile if retested. In other words, on grade level would include those scoring at or above 50 and all others within some unit of SEM (standard error of measurement) or SD (standard deviation) of 50

      Delete
  6. Describing Ravitch as "deeply flawed" is quite hyperbolic (and completely unjustified). Deeply flawed are the edu-entrepreneurs, politicians, religious fundamentalists and fanatics, racists and segregationists, and tax-evading corporations and their toadies who are destroying the middle class betraying our future and our democracy for selfish short-term gain. Somerby has identified the wrong enemy.

    ReplyDelete
  7. Florida is one of the states that has had the corporate fire-your-way to higher test scores method in place for the longest time, and yet it has failed to raise test scores.

    It seems to me that this information is very relevant because fire-your-way to higher test scores is supposed to work best on just such a low income population as Florida has. Massachusetts on the other hand has testing, but it also spends much more per pupil, the remedy that Hanushek and the other testing fundamentalists deplore as useless. Massachusetts would seem to prove that raising funding for schools rather than firing teachers is more effective as a way to produce higher scores.

    Note: whatever we may think of PISA, this is the metric that the educational corporatists have been using. And by their own metrics, their fundamentalist remedies have not worked (as common sense would have told them in the first place, had they tried it).

    It is not "joining the cult of PISA" to point out that even using PISA the corporate test-and-fire Charter school (for-thee-but-not-for-me and mine) method of improving the schools doesn't work.

    ReplyDelete
  8. Probably you should have to take out paper and pencil to understand the scores. Or actually you should have to compare real numbers such as median, standard deviation, percentiles, etc. These have a precise and unchangeable meaning as opposed to "grade level", "proficient" or "basic" which apparently have to be explicitly defined in every article. The attempt to put these things in words without using statistical terminology and without giving complete numbers (for example variance is very rarely reported) make a lot of these articles and books nearly unreadable.

    ReplyDelete
  9. NAEP scores have been rising very slowly. Neither they or the PISA are consistent with any significant improvement since the start of NCLB. Has there every been any evidence of any kind that the approach of the NCLB (and others advocated on both political sides) does what is claimed?

    ReplyDelete
    Replies
    1. By significant improvement I mean changes in methods have raised the slow overall improvement that all kinds of tests, including IQ tests, tend to show over time.

      Delete
  10. Some critics have pointed out that the slow rise in the rate of improvement over time stalled when No Child Left Behind was instituted. Others, such as Linda Darling Hamilton, a real professor (not one by courtesy, such as Hanushek) at Stanford, believes that the value added method is fundamentally flawed as a way of measuring teacher performance and that, in fact, what it measures is the quality of pupils, which for any given teacher varies from year to year. Supporters of "Value Added teacher assessment", such as Michell Rhee, maintain that pupils who have three years in a row of a good teacher will eliminate the "achievement gap" caused by SES between black and white students. This is their theory (Hanushek himself does not quite say this, he says it would take five years).

    According to Diane Ravitch, a study by Gordon, Kane and Staiger of the liberal Center for American Progress: "followed teachers in the first, second, and third years. It concluded that students assigned to a teacher in the bottom quartile of all teachers (ranked according to their students’ gains lost on average 5 percentile points. Thus, the difference between being assigned to a low- or high-rated teacher was 10 percentile points. Noting that the black-white achievement gap is estimated to be 34 percentile points, they reached this startling conclusion: “Therefore, if the effects were to accumulate, having a top-quartile teacher rather than a bottom-quartile teacher four years in a row would be enough to close the black-white test score gap.”
    quoted in http://voices.washingtonpost.com/answer-sheet/teachers/stopping-the-three-great-teach.html

    And Arne Duncan is on record as having said, “Three great teachers in a row, and the average child will be a year and a half to two grade levels ahead. Three bad teachers in a row, and that average child might be so far behind they might never catch up” (Ravitch, “Reign of Error”, p. 101).

    Critics, however, say that this is all theory. As I understand it, in the real world such a rise in scores has never happened, and, moreover, whatever rises have happened have been limited to the first three grades and then only in elementary arithmetic.

    This makes sense, IMO, since language skills probably become increasingly important as one advances in math.

    ReplyDelete