Part 3—Misled by Diane Ravitch: Yesterday, we criticized the New York Times editorial board.
We described its members as “functionally illiterate,” at least in their work on public schools and international test scores.
Today, we’ll apply the same term to the Times’ news reporting about New York City’s schools.
At issue is this very significant news report, a report the famous paper has buried deep inside this morning’s local section. Here’s what gives:
Yesterday, new test scores were released by the National Assessment of Educational Progress (the NAEP), our sole reliable domestic testing program.
These test scores track the progress of a group of city school systems, including New York City’s. Judging from this morning’s report, New York City’s progress seems to be quite substantial over the past decade.
But readers of today’s report won’t get any inkling of that apparent fact. The Times’ two reporters, Al Baker and Motoko Rich, seem to have no idea how to assess or evaluate these new scores.
For reasons which are blindingly obvious, we would say that this is the work of functional illiterates:
BAKER AND RICH (12/19/13): More than a snapshot of achievement, the scores released Wednesday illuminate overall increases the city’s fourth and eighth graders have made in math and reading since 2003, the year after Mayor Michael R. Bloomberg took office.For reasons which should be blindingly obvious, we’d call that the work of functional illiterates. That’s especially true when you see the boxed sub-headline appended by some editor:
For New York City’s fourth graders, the average reading score rose to 216 out of 500 this year, up 10 points from 2003. Nationally, the average fourth-grade reading score rose by four points, to 221. On math tests, the city’s fourth-grade average score rose to 236, up 10 points from 2003; the national score rose by seven points, to 241.
In the past decade, the city has chipped away at an achievement gap with the national average, even as cities with similar proportions of children from low-income families have risen from far lower bases of performance...
“Small but steady improvements in the Bloomberg era.”
Really? Do the reported score gains suggest small improvements during the last ten years?
Plainly, we’d say the answer is no. But there is no way to make an assessment from the text of this bungled report, in which Baker and Rich make no attempt to place the highlighted statistics in any kind of context.
According to Baker and Rich, New York City schools gained ten points in both reading and math during the Bloomberg decade. But would that be a lot or a little?
They make no attempt to say.
On some psychometric scales, like that of the SAT, a gain of ten points represents a trivial rounding error. On other scales, ten points might be a big deal.
In fact, ten points on the NAEP scale is often compared to one academic year. It would represent phenomenal progress if Gotham’s public school students have actually gained a full year in both reading and math over the past ten years.
That said, the New York Times is staffed by a gang of functional illiterates. In the full text of this buried report, its reporters never try to assess the size or significance of “ten points” worth of progress on the NAEP scale.
Gotham kids gained ten points in math! On its own, that statistic is meaningless, an import from an untranslated language.
But in this morning’s news report, the New York Times shows no sign of understanding that fact. It makes no attempt to tell readers how significant such a score gain might be.
Alas! This is the way our public schools are reported and discussed all through the wasteland we still describe as a “press corps.” This brings us back to our question from Tuesday:
Who lost M. Night Shyamalan?
Shyamalan is the Oscar-nominated director of the film, The Sixth Sense. He’s also a very lucid writer, one who has turned himself into a published, book-writing expert about the public schools.
Within our failing intellectual culture, no claim is too dumb to get widely repeated if it concerns public schools. That’s why Ali Velshi trumpeted this manifest nonsense, the fruit of a recent interview:
SHYAMALAN (12/11/13): You know how everyone says America is behind in education, compared to all the countries? Technically, right now, we're a little bit behind Poland and a little bit ahead of Liechtenstein, right? So that's where we land in the list, right? So that's actually not the truth.Are those the facts? If we look at international test scores, is it true that “our white kids are getting taught the best public-school education on the planet?”
The truth is actually bizarrely black and white, literally, which is, if you pulled out the inner-city schools—just pull out the inner-city, low-income schools, just pull that group out of the United States, put them to the side—and just took every other public school in the United States, we lead the world in public-school education by a lot.
And what's interesting is, we always think about Finland, right? Well, Finland, obviously, is mainly white kids, right? They teach their white kids really well. But guess what, we teach our white kids even better. We beat everyone. Our white kids are getting taught the best public-school education on the planet. Those are the facts.
If we don’t count our inner city schools, do “we lead the world in public-school education by a lot?”
In fact, those claims are crazily wrong. How did Shyamalan come to believe the things he has said, in various ways, in a series of interviews in which no journalist ever noted that his claims were wildly wrong?
Who lost M. Night Shyamalan, who seems completely sincere? We can get a clear picture of how he went wrong from a passage in his book.
The passage in question appears early on, right there on page 8. This passage starts to explain the way Shyamalan got fooled:
SHYAMALAN (page 8): The most important international comparison of educational performance is the Programme for International Student Assessment, or PISA, a test given every three years to fifteen-year-olds all over the world. Last time out, the U.S. average score was 500, just behind Poland and ahead of Liechtenstein.Ironically, Shyamalan starts by accepting the cult of the PISA, which mainstream “journalists” typically use to attack the quality of American schools. (American students rank higher on the TIMSS, a second major international test in which most developed nations take part.)
From the numbers Shyamalan provides, it’s clear that he is talking about only one part of the 2009 PISA—the reading test. Because reading is the subject on which U.S. students score best, his tilts his case a bit.
(The PISA also tests students in science and math. U.S. students rank worst in math.)
With that background established, here is his fuller statement from Shyamalan’s book. In this fuller passage, we see how he was misled:
SHYAMALAN (page 8): The most important international comparison of educational performance is the Programme for International Student Assessment, or PISA, a test given every three years to fifteen-year-olds all over the world. Last time out, the U.S. average score was 500, just behind Poland and ahead of Liechtenstein.Gack! Shyamalan is still talking about PISA reading scores. But as he does, he applies a widely-cited statistic in a deeply misleading way.
If American scores were limited to those from schools in districts in which the poverty rate was less than 10 percent—Finland’s poverty rate is less than 4 percent—the United States would lead the world and it wouldn’t be close: 551 on the latest PISA test, compared to Finland’s 536 or South Korea’s 539. In fact, if all you did was exclude the American schools that have student bodies that are more than three-quarters poor, U.S. schools would still score 513, just behind Australia, but ahead of the Netherlands, Belgium, Sweden, Germany, Iceland—well, you get the picture.
Right up through this post last week, Diane Ravitch has spent the last number of years encouraging American to make such bogus analyses. Like many others, Shyamalan got misled by several of the bogus claims Ravitch won’t stop pushing.
Alas! Shyamalan is misapplying a set of real statistics. Below, you see the data in question. Click here, scroll to page 14:
Average scores of U.S. 15-year-old students on combined reading literacy scale, by percentage of students in public school eligible for free or reduced-price lunch: 2009You can see the average score of 551 which Shyamalan cited. Where does his average score of 513 come from?
Less than 10 percent: 551
10 to 24.9 percent: 527
25 to 49.9 percent: 502
50 to 74.9 percent: 471
75 percent or more: 446
United States, all students and schools: 500
To appearances, he simply averaged the average scores of the top four groups of schools. We’ll assume that’s a wildly bogus procedure, though it doesn’t affect what comes next.
Aside from accepting the cult of the PISA, where does Shyamalan go wrong? He goes wrong in two basic ways:
First, he seems to think that the percentages on that chart represent percentages of students living in poverty. That is flatly false, though Ravitch is still promoting this false idea in her new book, Reign of Error.
Here’s his second error:
Shyamalan cites three average scores on this PISA reading test. Korea averaged 539; Finland averaged 536.
In the United States, students in schools where fewer than ten percent get a lunch subsidy averaged 551. That’s higher than Korea’s average, though Shyamalan overstates the significance of the difference. (Twelve points is not a giant big deal on the PISA scale, where 100 points equals one standard deviation.)
Here’s what Shyamalan doesn’t seem to understand about those statistics:
Roughly half of all American students receive free or reduced price lunch. (This isn’t a measure of poverty.) For that reason, very few schools qualify for the category in which fewer than ten percent of students receive a subsidized lunch.
These schools tend to be found in our wealthiest districts and neighborhoods. As such, they house a small, unrepresentative sample of the student population.
It’s interesting that students at these schools produce average scores which surpass or rival those of Korea, even in math. But this is a very advantaged slice of the U.S. student population. Those students don’t in any way represent the overall student population.
They don’t represent the white population. They don’t represent the non-poverty population.
They don’t represent the overall population with inner city schools left out. They represent a narrow, upper-end slice of the student population.
Does that average score of 551 in reading mean that “our white kids” “beat everyone” “by a lot?” It doesn’t mean anything like that.
Does it mean that we beat everyone else by a lot if you don’t count our inner city schools? It doesn’t mean that either!
Simply put, Shyamalan’s statements to Velshi were wildly inaccurate. His formulation from his book isn’t a whole lot better.
In fact, it’s possible to see how white students in the U.S. score on PISA tests. In our view, they score surprisingly well, especially as compared to the inaccurate claims one hears from our elite doom-and-gloom propagandists.
(Bill Keller, New York Times, August 19: We have experienced “decades of embarrassing decline in K-12 education.” Disgraceful. Crazily wrong.)
It’s easy to see how our white students score. Here you see the relevant scores on the most recent PISA tests:
Average scores, 2012 PISA, math:Even on the PISA, white students in the U.S. score quite close to miraculous Finland, whose brilliant practices we’re constantly told to emulate, even by the very shaky Ravitch.
Korea, all students: 554
Finland, all students: 519
United States, all students: 481
United States, white students only: 506
Average scores, 2012 PISA, reading:
Korea, all students: 536
Finland, all students: 524
United States, all students: 498
United States, white students only: 519
That said, do “we beat everyone” “by a lot?” Judged by these international measures, are “our white kids getting taught the best public-school education on the planet?”
Judged by these international measures, those statements are wildly inaccurate. On the PISA, the measure Shyamalan selected, our white students trail Korea in math by a rather large amount—by roughly one half of a standard deviation.
In reading, they come fairly close, though Shyamalan mistakenly says that a 17-point gap actually wouldn’t be close. But they don't beat Korea at all, let alone by a lot.
What if we only measured our white kids who aren’t living in poverty? We’ll guess that such kids would be outscoring Finland. But no such data exist.
What if we simply left out our “inner city” schools, the framework Shyamalan described to Velshi? Almost surely, our scores would be lower than the scores shown above for white students. Those scores would be nowhere near the best in the world.
Who lost M. Night Shyamalan, a lucid writer who seems to be completely well intentioned?
On the one hand, he accepted the PISA as the sole international standard. That’s what the cult of the PISA wants us to do. We know of no reason to do it.
Beyond that, Shyamalan was misled by two bogus ideas being circulated by Ravitch, who is very shaky as an analyst and a rather unfaithful servant:
First, the measure Shyamalan was using isn’t a measure of poverty. (The misperception that it is starts us down a wrong path.)
In part for that reason, very few schools fit in the group which produced the high average score Shyamalan cited.
That score was not produced by “white students.” That score isn’t what you get “if you pull out the inner city schools.”
That score was produced by a small number of schools which tend to be in our most affluent neighborhoods. That score doesn’t mean what Shyamalan thinks. It doesn’t mean anything like that.
Shyamalan made these mistakes for a fairly obvious reason. People like Ravitch keep pushing those bogus claims, year after year after year.
In fairness, Ravitch is a very fuzzy thinker. We’ll assume she doesn’t fully understand the statistics she keeps misinterpreting.
But Ravitch just keeps making these claims, even after acknowledging that they are “technically not valid.” When that sort of thing is done by people in rival tribes, their conduct gets scored as dishonest.
Who lost Shyamalan? Inevitably, Ravitch’s name comes to mind. Why on earth is Ravitch the best our uncaring liberal world has?
Tomorrow: Two sets of solutions