The era of living incompetently: On Thursday morning, the New York Times reported, or tried to report, the new NAEP scores for New York City.
For reasons you can review, we described the news report as the work of functional illiterates. That was before we fact-checked the newspaper’s factual claims.
Yesterday, we fact-checked their claims. Truly, we are all trapped in the era of living incompetently!
What did Al Baker and Motoko Rich claim in their multiply bungled report? As you can recall at the link we’ve provided, this was the passage we found most striking:
BAKER AND RICH (12/19/13): More than a snapshot of achievement, the scores released Wednesday illuminate overall increases the city’s fourth and eighth graders have made in math and reading since 2003, the year after Mayor Michael R. Bloomberg took office.Baker and Rich tried to report the score gains achieved in the Bloomberg years. As it turns out, their highlighted statements are wrong.
For New York City’s fourth graders, the average reading score rose to 216 out of 500 this year, up 10 points from 2003. Nationally, the average fourth-grade reading score rose by four points, to 221. On math tests, the city’s fourth-grade average score rose to 236, up 10 points from 2003; the national score rose by seven points, to 241.
Did New York City fourth-graders gain ten points in both reading and math? As we noted in our original post, that would suggest strong academic gains, based on conventional rules of thumb for interpreting NAEP scores.
That said, Gotham’s fourth-graders didn’t gain ten points in either reading or math.
Miraculously, the writers got the current fourth-grade scores right. But in each case, they misstated the size of the gains.
Here are New York City’s actual scores for fourth-grade reading. We’ll give you two decimal places:
Score gains on NAEP, New York City, 2003-2013On its face, that isn’t a bad score gain. But no, it isn’t ten points.
All students, Grade 4 reading
Score gain: 6.39 points
Here are the scores for math, where the Times almost got something right:
Score gains on NAEP, New York City, 2003-2013By conventional rules of thumb, 9.48 points is a strong gain. As you can see, it’s a gain of almost ten points, although, by convention, 9.48 rounds off to nine.
All students, Grade 4 math
Score gain: 9.48 points
By conventional rules of thumb, those aren’t bad score gains at all. Across the nation, gains have been easier to achieve in math than in reading, a pattern which holds in this case.
For that reason, we were struck by the reporters’ original claim. On its face, a gain of ten points in reading would have been a very strong gain.
The real gain was 6.39.
As the reporters noted, these gains are larger than those of the nation as a whole. But as usual, Baker and Rich were weirdly wrong, even in their factual statements.
How does the New York Times manage to do it? How do they get so many things wrong, right down to their basic numbers?
In our view, Rich’s education reporting creates an especially deep puzzle. At the social club known as our greatest newspaper, this summa cum laude graduate of Yale get very few things right.
How is that even possible?
A few more points:
It was never clear why the reporters singled out fourth grade in their report. Completing the picture, here are New York City’s gains in eighth grade reading and math:
Score gains on NAEP, New York City, 2003-2013Those gains are farther from ten points. Did it make sense to (mis)report the fourth-grade gains while ignoring the smaller gains recorded in Grade 8?
All students, Grade 8 reading
Score gain: 4.64 points
All students, Grade 8 math
Score gain: 7.85 points
One more point:
It is always dangerous to work with aggregate scores—with the average scores achieved by all students. If demographic changes have occurred in a student population, aggregate scores can mask more significant trends.
Below, you see the score gains for black kids in New York City during the Bloomberg years.
On their face, these are good gains. None of the gains amount to ten points, although Grade 8 math comes close:
Score gains on NAEP, New York City, 2003-2013On their face, those eighth grade gains are strong. Final point:
Black students, Grade 4 reading
Score gain: 8.43 points
Black students, Grade 4 math
Score gain: 5.80 points
Black students, Grade 8 reading
Score gain: 7.85 points
Black students, Grade 8 math
Score gain: 9.44 points
Especially in tests where samples are tested, any individual score can be “wrong.” Any individual score can be unnaturally high or unnaturally low, for any number of reasons.
There is no perfect sampling. Unless you’re measuring height or weight, there is no perfect measurement—and not even then, of course.
Any individual score or score gain can be “wrong.” You have to look at the broad sweep of scores and make a broad assessment. On their face, the score gains were pretty good in Gotham during the past ten years.
Inevitably, though, the New York Times misstated the basic facts in Thursday’s multiply-bungled report. The Times, which is really a social club, is grossly incompetent in an amazingly wide range of ways.
In the next year, we’ll try to induce other people to discuss that remarkable fact. Given the national role of the Times, it’s very hard to get career writers to state that remarkable fact.
Coming tomorrow or Monday: The year of living propagandistically! Valerie Strauss’ propagandistic review of D.C.’s new NAEP scores
Where do our data come from: As far as we know, there isn’t any easy way to link you to these NAEP data. If you want to check for yourself:
Click here, then click on MAIN NDE. (Main NAEP Data Explorer.)
Click on “I agree to the terms above.” From there, you’re on your own.