Part 3—Misled by Diane Ravitch: Yesterday, we criticized the New York Times editorial board.
We described its members as “functionally illiterate,” at least in their work on public schools and international test scores.
Today, we’ll apply the same term to the Times’ news reporting about New York City’s schools.
At issue is this very significant news report, a report the famous paper has buried deep inside this morning’s local section. Here’s what gives:
Yesterday, new test scores were released by the National Assessment of Educational Progress (the NAEP), our sole reliable domestic testing program.
These test scores track the progress of a group of city school systems, including New York City’s. Judging from this morning’s report, New York City’s progress seems to be quite substantial over the past decade.
But readers of today’s report won’t get any inkling of that apparent fact. The Times’ two reporters, Al Baker and Motoko Rich, seem to have no idea how to assess or evaluate these new scores.
For reasons which are blindingly obvious, we would say that this is the work of functional illiterates:
BAKER AND RICH (12/19/13): More than a snapshot of achievement, the scores released Wednesday illuminate overall increases the city’s fourth and eighth graders have made in math and reading since 2003, the year after Mayor Michael R. Bloomberg took office.For reasons which should be blindingly obvious, we’d call that the work of functional illiterates. That’s especially true when you see the boxed sub-headline appended by some editor:
For New York City’s fourth graders, the average reading score rose to 216 out of 500 this year, up 10 points from 2003. Nationally, the average fourth-grade reading score rose by four points, to 221. On math tests, the city’s fourth-grade average score rose to 236, up 10 points from 2003; the national score rose by seven points, to 241.
In the past decade, the city has chipped away at an achievement gap with the national average, even as cities with similar proportions of children from low-income families have risen from far lower bases of performance...
“Small but steady improvements in the Bloomberg era.”
Really? Do the reported score gains suggest small improvements during the last ten years?
Plainly, we’d say the answer is no. But there is no way to make an assessment from the text of this bungled report, in which Baker and Rich make no attempt to place the highlighted statistics in any kind of context.
According to Baker and Rich, New York City schools gained ten points in both reading and math during the Bloomberg decade. But would that be a lot or a little?
They make no attempt to say.
On some psychometric scales, like that of the SAT, a gain of ten points represents a trivial rounding error. On other scales, ten points might be a big deal.
In fact, ten points on the NAEP scale is often compared to one academic year. It would represent phenomenal progress if Gotham’s public school students have actually gained a full year in both reading and math over the past ten years.
That said, the New York Times is staffed by a gang of functional illiterates. In the full text of this buried report, its reporters never try to assess the size or significance of “ten points” worth of progress on the NAEP scale.
Gotham kids gained ten points in math! On its own, that statistic is meaningless, an import from an untranslated language.
But in this morning’s news report, the New York Times shows no sign of understanding that fact. It makes no attempt to tell readers how significant such a score gain might be.
Alas! This is the way our public schools are reported and discussed all through the wasteland we still describe as a “press corps.” This brings us back to our question from Tuesday:
Who lost M. Night Shyamalan?
Shyamalan is the Oscar-nominated director of the film, The Sixth Sense. He’s also a very lucid writer, one who has turned himself into a published, book-writing expert about the public schools.
Within our failing intellectual culture, no claim is too dumb to get widely repeated if it concerns public schools. That’s why Ali Velshi trumpeted this manifest nonsense, the fruit of a recent interview:
SHYAMALAN (12/11/13): You know how everyone says America is behind in education, compared to all the countries? Technically, right now, we're a little bit behind Poland and a little bit ahead of Liechtenstein, right? So that's where we land in the list, right? So that's actually not the truth.Are those the facts? If we look at international test scores, is it true that “our white kids are getting taught the best public-school education on the planet?”
The truth is actually bizarrely black and white, literally, which is, if you pulled out the inner-city schools—just pull out the inner-city, low-income schools, just pull that group out of the United States, put them to the side—and just took every other public school in the United States, we lead the world in public-school education by a lot.
And what's interesting is, we always think about Finland, right? Well, Finland, obviously, is mainly white kids, right? They teach their white kids really well. But guess what, we teach our white kids even better. We beat everyone. Our white kids are getting taught the best public-school education on the planet. Those are the facts.
If we don’t count our inner city schools, do “we lead the world in public-school education by a lot?”
In fact, those claims are crazily wrong. How did Shyamalan come to believe the things he has said, in various ways, in a series of interviews in which no journalist ever noted that his claims were wildly wrong?
Who lost M. Night Shyamalan, who seems completely sincere? We can get a clear picture of how he went wrong from a passage in his book.
The passage in question appears early on, right there on page 8. This passage starts to explain the way Shyamalan got fooled:
SHYAMALAN (page 8): The most important international comparison of educational performance is the Programme for International Student Assessment, or PISA, a test given every three years to fifteen-year-olds all over the world. Last time out, the U.S. average score was 500, just behind Poland and ahead of Liechtenstein.Ironically, Shyamalan starts by accepting the cult of the PISA, which mainstream “journalists” typically use to attack the quality of American schools. (American students rank higher on the TIMSS, a second major international test in which most developed nations take part.)
From the numbers Shyamalan provides, it’s clear that he is talking about only one part of the 2009 PISA—the reading test. Because reading is the subject on which U.S. students score best, his tilts his case a bit.
(The PISA also tests students in science and math. U.S. students rank worst in math.)
With that background established, here is his fuller statement from Shyamalan’s book. In this fuller passage, we see how he was misled:
SHYAMALAN (page 8): The most important international comparison of educational performance is the Programme for International Student Assessment, or PISA, a test given every three years to fifteen-year-olds all over the world. Last time out, the U.S. average score was 500, just behind Poland and ahead of Liechtenstein.Gack! Shyamalan is still talking about PISA reading scores. But as he does, he applies a widely-cited statistic in a deeply misleading way.
However—
If American scores were limited to those from schools in districts in which the poverty rate was less than 10 percent—Finland’s poverty rate is less than 4 percent—the United States would lead the world and it wouldn’t be close: 551 on the latest PISA test, compared to Finland’s 536 or South Korea’s 539. In fact, if all you did was exclude the American schools that have student bodies that are more than three-quarters poor, U.S. schools would still score 513, just behind Australia, but ahead of the Netherlands, Belgium, Sweden, Germany, Iceland—well, you get the picture.
Right up through this post last week, Diane Ravitch has spent the last number of years encouraging American to make such bogus analyses. Like many others, Shyamalan got misled by several of the bogus claims Ravitch won’t stop pushing.
Alas! Shyamalan is misapplying a set of real statistics. Below, you see the data in question. Click here, scroll to page 14:
Average scores of U.S. 15-year-old students on combined reading literacy scale, by percentage of students in public school eligible for free or reduced-price lunch: 2009You can see the average score of 551 which Shyamalan cited. Where does his average score of 513 come from?
Less than 10 percent: 551
10 to 24.9 percent: 527
25 to 49.9 percent: 502
50 to 74.9 percent: 471
75 percent or more: 446
United States, all students and schools: 500
To appearances, he simply averaged the average scores of the top four groups of schools. We’ll assume that’s a wildly bogus procedure, though it doesn’t affect what comes next.
Aside from accepting the cult of the PISA, where does Shyamalan go wrong? He goes wrong in two basic ways:
First, he seems to think that the percentages on that chart represent percentages of students living in poverty. That is flatly false, though Ravitch is still promoting this false idea in her new book, Reign of Error.
Here’s his second error:
Shyamalan cites three average scores on this PISA reading test. Korea averaged 539; Finland averaged 536.
In the United States, students in schools where fewer than ten percent get a lunch subsidy averaged 551. That’s higher than Korea’s average, though Shyamalan overstates the significance of the difference. (Twelve points is not a giant big deal on the PISA scale, where 100 points equals one standard deviation.)
Here’s what Shyamalan doesn’t seem to understand about those statistics:
Roughly half of all American students receive free or reduced price lunch. (This isn’t a measure of poverty.) For that reason, very few schools qualify for the category in which fewer than ten percent of students receive a subsidized lunch.
These schools tend to be found in our wealthiest districts and neighborhoods. As such, they house a small, unrepresentative sample of the student population.
It’s interesting that students at these schools produce average scores which surpass or rival those of Korea, even in math. But this is a very advantaged slice of the U.S. student population. Those students don’t in any way represent the overall student population.
They don’t represent the white population. They don’t represent the non-poverty population.
They don’t represent the overall population with inner city schools left out. They represent a narrow, upper-end slice of the student population.
Does that average score of 551 in reading mean that “our white kids” “beat everyone” “by a lot?” It doesn’t mean anything like that.
Does it mean that we beat everyone else by a lot if you don’t count our inner city schools? It doesn’t mean that either!
Simply put, Shyamalan’s statements to Velshi were wildly inaccurate. His formulation from his book isn’t a whole lot better.
In fact, it’s possible to see how white students in the U.S. score on PISA tests. In our view, they score surprisingly well, especially as compared to the inaccurate claims one hears from our elite doom-and-gloom propagandists.
(Bill Keller, New York Times, August 19: We have experienced “decades of embarrassing decline in K-12 education.” Disgraceful. Crazily wrong.)
It’s easy to see how our white students score. Here you see the relevant scores on the most recent PISA tests:
Average scores, 2012 PISA, math:Even on the PISA, white students in the U.S. score quite close to miraculous Finland, whose brilliant practices we’re constantly told to emulate, even by the very shaky Ravitch.
Korea, all students: 554
Finland, all students: 519
United States, all students: 481
United States, white students only: 506
Average scores, 2012 PISA, reading:
Korea, all students: 536
Finland, all students: 524
United States, all students: 498
United States, white students only: 519
That said, do “we beat everyone” “by a lot?” Judged by these international measures, are “our white kids getting taught the best public-school education on the planet?”
Judged by these international measures, those statements are wildly inaccurate. On the PISA, the measure Shyamalan selected, our white students trail Korea in math by a rather large amount—by roughly one half of a standard deviation.
In reading, they come fairly close, though Shyamalan mistakenly says that a 17-point gap actually wouldn’t be close. But they don't beat Korea at all, let alone by a lot.
What if we only measured our white kids who aren’t living in poverty? We’ll guess that such kids would be outscoring Finland. But no such data exist.
What if we simply left out our “inner city” schools, the framework Shyamalan described to Velshi? Almost surely, our scores would be lower than the scores shown above for white students. Those scores would be nowhere near the best in the world.
Who lost M. Night Shyamalan, a lucid writer who seems to be completely well intentioned?
On the one hand, he accepted the PISA as the sole international standard. That’s what the cult of the PISA wants us to do. We know of no reason to do it.
Beyond that, Shyamalan was misled by two bogus ideas being circulated by Ravitch, who is very shaky as an analyst and a rather unfaithful servant:
First, the measure Shyamalan was using isn’t a measure of poverty. (The misperception that it is starts us down a wrong path.)
In part for that reason, very few schools fit in the group which produced the high average score Shyamalan cited.
That score was not produced by “white students.” That score isn’t what you get “if you pull out the inner city schools.”
That score was produced by a small number of schools which tend to be in our most affluent neighborhoods. That score doesn’t mean what Shyamalan thinks. It doesn’t mean anything like that.
Shyamalan made these mistakes for a fairly obvious reason. People like Ravitch keep pushing those bogus claims, year after year after year.
In fairness, Ravitch is a very fuzzy thinker. We’ll assume she doesn’t fully understand the statistics she keeps misinterpreting.
But Ravitch just keeps making these claims, even after acknowledging that they are “technically not valid.” When that sort of thing is done by people in rival tribes, their conduct gets scored as dishonest.
Who lost Shyamalan? Inevitably, Ravitch’s name comes to mind. Why on earth is Ravitch the best our uncaring liberal world has?
Tomorrow: Two sets of solutions
As predicted.
ReplyDeleteMind-numbing gotcha!s addressing trivialities.
But Shyamalan's five points not touched at all.
Shyamalan says that the US is by far the best educator of non-poverty kids in the world. That is solidly wrong and not trivial. It isn't anywhere close to a gotcha to note that someone is making really simple stupid errors on large important statements.
DeleteIt's okay for our public discourse to contain small errors. It's okay for our public discourse to contain big errors on small points. But it's damaging to give a free pass to people with a large platform being very stupid.
I want people with a large platform to not suck at making points. Shouldn't be too much to ask. There are lots of people who don't suck at making points. They need to take the place of the people who do suck.
(I should add, it's also okay to make big errors in difficult subjects. This is how we learn. But Shyamalan's errors weren't difficult. They were simple and stupid.)
Deleteb. somerby says,
Delete“According to Baker and Rich, New York City schools gained ten points in both reading and math during the Bloomberg decade. But would that be a lot or a little?”
>>> 206 (216-10=206) divided by 500 (500 is the maximum score) is 41.20%. 216 divided by 500 is 43.20%. so the the reading score improved by 2.00% (43.20-41.20).
226 divided by 500 is 45.2%. 236 divided by 500 is 47.2%. so the math score improved by 2.00% (47.20 -45.20) as well.
Whether a 2.00 % improvement in both scores in one year is a little or a lot is subjective.
correction:
Deletethe 2% change was over 10 years, not one year. so that would be a change of only .2% per year which obviously is only a little.
Wrong.
DeleteThat 2% improvement is a large gain. It represents a full academic year.
Today's fourth graders are as smart as 2003's fifth graders. That's impressive improvement.
Justin Tyme
DeleteTest scores follow all kinds of weighted curves. Your analysis is logical, but completely facile.
Anonymous 11:04 --If in-depth analysis is too much for you, then at the very least stay quiet about it. TDH seemed to just blow Shyamalan's credibility out of the water and your response is boredom and condescension. It is good to know that you read this site, because you clearly are infected with the anti-intellectualism of our day, the very thing this site so ably and persistently fights against.
Bravo M. Carpenter!
DeleteThe Daily Show did an interview with a NY Times financial editor about why they don't cover scandals involving Wall Street greed. The editor stated that if the story couldn't be explained in a couple of sentences, it didn't run. If that is the criterion for financial stories, which do tend to be technical, it is probably the criterion for education stories or science stories or anything else complicated. She stated that there are plenty of stories that can be published using that criterion and thus no need to report on stories that would be more difficult to explain.
ReplyDeleteI think this applies to education reports, except there are no experts on staff capable of boiling the stories down to accurate summaries, because they are complicated. The issues described above show how complicated they can be for someone who is not an expert. My experience is that people in the field of education typically do not know a lot about statistics. I have seen people get doctorates in education by hiring a consultant to do their dissertation statistics for them (a routine practice not considered cheating, although maybe it should be). You would think that someone trained in education should understand what a standard deviation is and how to interpret test scores. But disdain for quantitative approaches has plagued the social sciences for decades.
I really appreciate this example of how innumeracy hurts our efforts to improve education. We are now an information society. Tools for understanding data are essential to success in it. I see this as a sign that not every profession is well-prepared to meet the demands of our changing society. Hopefully that will change. When it does maybe we will get better books on education and public policy will be better informed. Can we wait that long?
@ Justintyme "The 2% change was over 10 years, not one year. so that would be a change of only .2% per year which obviously is only a little."
ReplyDelete***
"A little learning is a dangerous thing
Drink deep, or taste not the Pierian spring." - Alexander Pope (1711)
Steeve,
ReplyDeleteIt matters little what misstatements Shyamalan made on TV except to get this blogger's panties in a bunch.
SHYAMALAN BOILED DOWN TWO YEARS OF RESEARCH TO FIVE RECOMMENDATIONS WRITTEN IN EVERYDAY ENGLISH.
What do you think the odds are that the blogger will critique them after lo this many posts lamenting librul scoundrels who don't care about education?
TDH spent a hell of a lot of years in a classroom and many years thinking and writing about our school system. People who have true experience and expertise often resist it when someone who does not prescribes simplistic solutions based on incomplete understanding.
DeleteYour snark toward Bob Somerby is ugly and juvenile. Why do you think you are smarter than him? Give us some evidence of your intelligence. Sarcasm is nothing.
Kool Aid Drinker.
DeleteSo your god is allowed score gotcha!s against trivialities but strictly ignore findings Shyamalan claims were found using statistical analysis methods used for clinical trials.
And these are not buried in clouds of smoke like the blogger's perpetual snit - they are direct and clearly stated. Surely this god of education should be able to critique them. To endlessly go on about Shyamalan's inaccuracies on trivial while strictly staying away from these recommendations is tolerable only to kool aid drinkers.
"Here’s what Shyamalan doesn’t seem to understand about those statistics:
DeleteRoughly half of all American students receive free or reduced price lunch. (This isn’t a measure of poverty.) For that reason, very few schools qualify for the category in which fewer than ten percent of students receive a subsidized lunch.
These schools tend to be found in our wealthiest districts and neighborhoods. As such, they house a small, unrepresentative sample of the student population.
It’s interesting that students at these schools produce average scores which surpass or rival those of Korea, even in math. But this is a very advantaged slice of the U.S. student population. Those students don’t in any way represent the overall student population.
They don’t represent the white population. They don’t represent the non-poverty population.
They don’t represent the overall population with inner city schools left out. They represent a narrow, upper-end slice of the student population.
Does that average score of 551 in reading mean that “our white kids” “beat everyone” “by a lot?” It doesn’t mean anything like that."
This is a very complicated issue that the blogger is oversimplifying. These people are a CASTE - not a race or ethnic group. The best non-white examples of this group are Indian-Americans who come in all colors, East Asians, Middle Easterners etc.
The upper caste of the US is second to none and if Shyamalan conflates them with non-poor or white its just casual talk. Shyamalan obviously knows this.
Anon 4:35,
Deletethe point of the TDH is to point out that misreporting "trivial" facts is the bread and butter of our media truth-spinners. Anybody can give out high-minding recommendations, but people with real on-the-ground experience generally just shrug them off.
Talk about drinking the Koolaid. Hollywood people have a long track record of giving out well-intentioned, yet superficial advice. Sometimes such people actually help a little but often they just get in the way.
M Carpenter -
DeleteSHYAMALAN'S PARENTS ARE DOCTORS.
HE IS FAMILIAR WITH CLINICAL TRIALS.
HE USED THE SAME TECHNIQUES TO ISOLATE FACTORS THAT HELP IMPROVE SCHOOLS.
There is no room for gotcha!s here. It is practically a scholarly work and the blogger is using gutter tactics to attack it.
SHYAMALAN'S PARENTS ARE DOCTORS.
DeleteHE IS FAMILIAR WITH CLINICAL TRIALS
None of this is impressive to me--what your parents did is a distant qualification. "practically a a acholarly work" is code for dilatant. Your boy has no relevant field experience--that is a huge disqualification.
I am familier with clinical trials too. Six times out of ten their results are misinterpreted by people with an agenda. People with good intentions but minimal experience almost never get it right, because reality gets more and more complicated the more you get into it. The real experts hate the dilatants. TDH is a real expert in education, Shyamalan is not.
M Carpenter, the word you are looking for is "dilettante". A dilatant is probably something that causes dilation.
ReplyDelete"I am familier with clinical trials too. Six times out of ten their results are misinterpreted by people with an agenda. People with good intentions but minimal experience almost never get it right, because reality gets more and more complicated the more you get into it. "
Correct.
Shyamalan's recommendations may all be wrong - but they need to be critiqued on their terms and how they were arrived at ( logging outcomes that result from lots of combinations of inputs and statistically inferring the inputs with the most significant causal influence on the outcome).
The blogger is only using falsehoods, slander etc. in place of criticism based on multivariate statistical analysis.
He is probably innumerate and would probably not recognize Decision Trees or Logistical Regression even if they bit him on the nose.
This was shooting fish in a barrel. Some of Shyamalan's major and fundamental assumptions are wrong, and TDH points this out. The basic assumptions then cause Shyamalan to make laughably off-base statements about the U.S. public education system. I wouldn't take it so personally: Shyamalan's probably a great guy and he's made a great movie several times.
Delete