New York Times bungles test scores again!

SATURDAY, DECEMBER 21, 2013

The era of living incompetently: On Thursday morning, the New York Times reported, or tried to report, the new NAEP scores for New York City.

For reasons you can review, we described the news report as the work of functional illiterates. That was before we fact-checked the newspaper’s factual claims.

Yesterday, we fact-checked their claims. Truly, we are all trapped in the era of living incompetently!

What did Al Baker and Motoko Rich claim in their multiply bungled report? As you can recall at the link we’ve provided, this was the passage we found most striking:
BAKER AND RICH (12/19/13): More than a snapshot of achievement, the scores released Wednesday illuminate overall increases the city’s fourth and eighth graders have made in math and reading since 2003, the year after Mayor Michael R. Bloomberg took office.

For New York City’s fourth graders, the average reading score rose to 216 out of 500 this year, up 10 points from 2003. Nationally, the average fourth-grade reading score rose by four points, to 221. On math tests, the city’s fourth-grade average score rose to 236, up 10 points from 2003; the national score rose by seven points, to 241.
Baker and Rich tried to report the score gains achieved in the Bloomberg years. As it turns out, their highlighted statements are wrong.

Did New York City fourth-graders gain ten points in both reading and math? As we noted in our original post, that would suggest strong academic gains, based on conventional rules of thumb for interpreting NAEP scores.

That said, Gotham’s fourth-graders didn’t gain ten points in either reading or math.

Miraculously, the writers got the current fourth-grade scores right. But in each case, they misstated the size of the gains.

Here are New York City’s actual scores for fourth-grade reading. We’ll give you two decimal places:
Score gains on NAEP, New York City, 2003-2013
All students, Grade 4 reading

2013: 216.27
2003: 209.88
Score gain: 6.39 points
On its face, that isn’t a bad score gain. But no, it isn’t ten points.

Here are the scores for math, where the Times almost got something right:
Score gains on NAEP, New York City, 2003-2013
All students, Grade 4 math

2013: 235.84
2003: 226.36
Score gain: 9.48 points
By conventional rules of thumb, 9.48 points is a strong gain. As you can see, it’s a gain of almost ten points, although, by convention, 9.48 rounds off to nine.

By conventional rules of thumb, those aren’t bad score gains at all. Across the nation, gains have been easier to achieve in math than in reading, a pattern which holds in this case.

For that reason, we were struck by the reporters’ original claim. On its face, a gain of ten points in reading would have been a very strong gain.

The real gain was 6.39.

As the reporters noted, these gains are larger than those of the nation as a whole. But as usual, Baker and Rich were weirdly wrong, even in their factual statements.

How does the New York Times manage to do it? How do they get so many things wrong, right down to their basic numbers?

In our view, Rich’s education reporting creates an especially deep puzzle. At the social club known as our greatest newspaper, this summa cum laude graduate of Yale get very few things right.

How is that even possible?

A few more points:

It was never clear why the reporters singled out fourth grade in their report. Completing the picture, here are New York City’s gains in eighth grade reading and math:
Score gains on NAEP, New York City, 2003-2013
All students, Grade 8 reading

2013: 256.43
2003: 251.79
Score gain: 4.64 points

All students, Grade 8 math
2013: 273.62
2003: 265.77
Score gain: 7.85 points
Those gains are farther from ten points. Did it make sense to (mis)report the fourth-grade gains while ignoring the smaller gains recorded in Grade 8?

One more point:

It is always dangerous to work with aggregate scores—with the average scores achieved by all students. If demographic changes have occurred in a student population, aggregate scores can mask more significant trends.

Below, you see the score gains for black kids in New York City during the Bloomberg years.

On their face, these are good gains. None of the gains amount to ten points, although Grade 8 math comes close:
Score gains on NAEP, New York City, 2003-2013
Black students, Grade 4 reading

2013: 209.70
2003: 201.27
Score gain: 8.43 points

Black students, Grade 4 math
2013: 224.70
2003: 218.90
Score gain: 5.80 points

Black students, Grade 8 reading
2013: 252.56
2003: 244.71
Score gain: 7.85 points

Black students, Grade 8 math
2013: 262.59
2003: 253.15
Score gain: 9.44 points
On their face, those eighth grade gains are strong. Final point:

Especially in tests where samples are tested, any individual score can be “wrong.” Any individual score can be unnaturally high or unnaturally low, for any number of reasons.

There is no perfect sampling. Unless you’re measuring height or weight, there is no perfect measurement—and not even then, of course.

Any individual score or score gain can be “wrong.” You have to look at the broad sweep of scores and make a broad assessment. On their face, the score gains were pretty good in Gotham during the past ten years.

Inevitably, though, the New York Times misstated the basic facts in Thursday’s multiply-bungled report. The Times, which is really a social club, is grossly incompetent in an amazingly wide range of ways.

Our pledge:

In the next year, we’ll try to induce other people to discuss that remarkable fact. Given the national role of the Times, it’s very hard to get career writers to state that remarkable fact.

Coming tomorrow or Monday: The year of living propagandistically! Valerie Strauss’ propagandistic review of D.C.’s new NAEP scores

Where do our data come from: As far as we know, there isn’t any easy way to link you to these NAEP data. If you want to check for yourself:

Click here, then click on MAIN NDE.
(Main NAEP Data Explorer.)

Click on “I agree to the terms above.” From there, you’re on your own.

29 comments:

  1. Any thing you say, blogger.

    Folks - I predicted the chickenshit coward would stay away from Shyamalan's 5 points and voila! - it looks like he has "disappeared" them.

    ReplyDelete
    Replies
    1. You're correct. He didn't talk about what you wanted him to talk about. I predicted he wouldn't talk about the upcoming Mizzou-Illinois game. I was right too.

      The "coward" did happen to ask a question that you can't answer. How can a top newspaper screw up everything so stupidly, right down to simple subtraction?

      It's not trivial or petty to note that the top of our media is very stupid. It's a message that needs to embed itself deep into the mind of every single citizen.

      Delete
    2. Hey, I get it.

      There is only one source of "truth", especially about education, and that source is Somerby, and only Somerby. The rest of the entire world is hopelessly inept.

      This is where Somerby and David Koresh find common ground.

      Delete
    3. Anonymous @11:40A,

      You get it only if ignorance is somehow contagious.

      There is only one source of topics, especially about education, for this blog, and that source is TDH, and only TDH. Because. This. Is. His. Blog.

      TDH doesn't demand that you read his blog and believe. In fact, if this is such a terrible place of mind control, what are you doing here?

      Delete
    4. now anon 11:40, i dont know about this koresh fella but i agree with you on mr. somerby. hes a giant in the computer box. what id like to know is whether ms. hathaway or mr. drysdale is doing a blog on here somewheres? smartest two city folk i ever met. havnet had the pleasure of meeting bob, so no offense there. hes as as smart as whip when comes to fancy book learnin, that bob somerby. i know that cuz i gots me a sixth grade education, so i knows a fellow intellect when i sees one.

      one little thing im not so partial to bob somerby on is this feud he seems to have going with the irish folk. now remember im typing this from the 1960's, so i dont what all has happened that they seem to be low man on the totem pole. you know, a lot of we mountain peoples blood is irish. but hes a good man in general, bob is. i think you know what i mean. all right then.

      oh, and stay tuned for my next comment. its a hum dinger, yes sir!

      Delete
  2. when i was doing the cyphering originally it struck me as odd that both scores' percentage changes were identical and an even whole number to boot, 2.00(0000000etc.). i had to take my shoes off again and redo the divideds and the multiplieds just to make sure i had it right. sure enuf uncle jed, they was the same dang number!

    well "hoooo doggies", im saying to myself now. what do we have here? did they work backwards from the conclusion, the percentage change, 2.00%, to the the support, the raw test results?

    i dont what to think about these yankee pencil pushers. they are northerners, arent they? well regardless we got a good yankee, ole bob somerby, to keep these boys in line. he knows all about the reverse argumentation. i gots me a sixth grade education, but let me you tell you what, i got nothing on ole bobby. you be good to him now. take your shoes off, stay awhile, you hear?

    ReplyDelete
  3. Superb post, Bob. I use each and every analysis you do on education.

    ReplyDelete
    Replies
    1. It is apparent the bulk of BOBfans prefer posts about sexist song lyrics.

      KZ

      Delete
  4. Bob, I mistake for 4th grade reading is startling. Really, really poor work and not corrected so far in the article. I was immediately impressed by the supposed gain on 10 points for 4th graders in reading but had not read the original report. Superb work.

    ReplyDelete
    Replies
    1. Bob, [the] mistake for 4th grade reading is startling....

      Delete
    2. But the hypocrisy by BOB in not letting us know the 10 point gain did occur, but was from 2002-2013 instead of 2003-2013 as reprted by the Times is typical.

      KZ

      Delete
  5. Since Zimmerman is never off-topic in these parts

    http://www.orlandosentinel.com/news/local/trayvon-martin/os-george-zimmerman-ebay-painting-20131221,0,3925055.story

    The modern day Jesus crucified by librulz seems to be doing well for himself.

    ReplyDelete
    Replies
    1. Thanks for the link. Enjoyed the sidebar NewsMax headline on Duck Dynasty merchanise selling out at WalMart as well.

      Is Zimm a better artiste than Prexy Bush?

      Are there Zimm black velvet paintings yet?

      If only Salon or Rachel would cover them, then we could read about it here!

      Delete
  6. http://www.npr.org/2013/09/12/221409659/filmmaker-turns-to-education-reform-gets-schooled

    shows how thoroughly dishonest (through omission) the blogger is.

    "The book debunks several myths. Perhaps the biggest is the notion that smaller class size makes for a better education."

    Its amazing that our education expert has no opinion to offer on this.

    And look - he debunked Finland even before our intrepid education expert

    "One bit of advice I'm ready to share is this: whenever anyone brings up Finland, back away slowly," he writes. "In fact it mystifies me that a country with fewer people than Greater Philadelphia, no civil rights problem, and virtually no significant income inequality is held up as a model for the United States."

    You slice the blogger any which way, he comes up despicable.

    ReplyDelete
    Replies
    1. On "he debunked Finland even before our intrepid education expert ...." Here's the Daily Howler from May 2005, beating Shymalan by eight and a half years: "Does Finland have 'the best school system in the world?' The claim is silly, and it’s amazing that the Post doesn’t know this." http://dailyhowler.com/dh052405.shtml

      Delete
  7. OMB (Where Dewey find dem Decimals)

    Well, if the NYT is a social club, BOB's sprawling campus blog is an isolation cell inhabitated by BOB and his imaginary analyst friends.

    Did the NYT make the mistakes BOB says they made when he fact checked them? Well, if you the follow the link to the NAEP data base
    BOB sends you to, it is easy to see they erred reporting on on reading. It is not so easy to see they erred reporting on math. That's because the NAEP data base doesn't seem to report scores to the decimal level BOB uses. We couldn't find one single set of data reported to the tenth of a point much less the hundredth of a point as reported here in BOBland. Follow BOB's link and see for yourself.

    How did the NYT err in reading? It seems they selected 2002 reading scores rather than 2003 for their starting year. So NY City fourth graders gained 10 points in reading in the last eleven years instead of the last ten. That is a whole grade level's worth of improvement using BOB's "widely known" but strangely never cited rule of thumb. In math there were no scores from 2002. But from 2003 to 2013 the NAEP data base reports a ten point gain. Perhaps the NYT did not make a "conventional thumb" rounding error because they consulted the decimal free data base BOB links us to.

    Did the New York Times select the wrong year on reading scores to fit their narrative of making the gains attributable to the Bloomberg years? Perhaps. We've seen this sort of misrepresentation before. BOB did it when he tried to make it seem like the US gains on PISA test scores were larger than that of Poland. He simply started with a base year that fit his narrative which conveniently allowed him to turn US test score drops into gains and erased a year in which Poland made great strides.

    Yep. The NYT is a gang of deceivers. Learning tricks from the looney bin. Shame on you NYT.

    KZ

    ReplyDelete
  8. Thanks KZ.

    The blogger is reduced to outright lying or obsessing on inadvertent errors that can happen to anybody to keep his flock worked up against librulz.

    Since he has all but come out as a winger, whats up with the "we liberals" shtick?

    ReplyDelete
    Replies
    1. Liberals are reality based, and Somerby tells it like it is. What "it is" is rank incompetence because the private-schooled elite just don't give a damn about the public school facts.

      Another important lesson to learn from Somerby is that the NYT and Rachel Maddow are not liberal on stuff that matters. On what matters they are thoroughly neoliberal and/or dedicated to wedge issues that unnecessarily divide real leftists from their natural allies on the 'populist right'.

      Delete
  9. Replies
    1. The NYT, like respectable blogs, employs a spam filter on trolls.

      Delete
  10. fairleft

    Your god targets Maddow et al. and since his transmogrification to neoconfederate angry white male (and a loser in life) strictly ignores the likes of Coulter ("liberals should fear that they too can be killed") or the blacker-than-the Ace-of-Spades Dinesh DeSouza ("OBama = Grown up Trayvon") or has only milquetoast criticism for them.

    Whatever floats your boat.

    However, there are bunch of us here who will rip him a new one (figuratively) constantly until he either comes to his senses or stops this train-wreck of a blog.

    ReplyDelete
    Replies
    1. A clear statement of troll intentions.

      Delete
    2. You're wasting your time and energy in a big way. One of the reasons why is because you posts come across as angry, stupid and tribal. They are boring. You are being boring as hell, Do whatcha like but I doubt blogger will stop based on your dumbass comments. (You may be right about what you say but you say it like a fool. No offense.) It's a shame to see someone waste their time and energy so completely.

      Delete
    3. I can't even take them seriously anymore.

      Whatever...

      Delete
    4. AnonymousDecember 22, 2013 at 3:03 PM

      "tribal"

      The freaking blogger even controls your vocabulary. THINK man - why do you look for others to do your thinking for you?

      Delete
  11. I don't know why I do that. I'm just saying you're probably wasting your time in trying bring blogger to his senses or have him stop. And that it's a little boring. But maybe you can do it. Maybe blogger will come to his senses due to your posse's constant figurative rips. Everything is possible. We'll just have to let time pass and see. Have a good holiday season!

    ReplyDelete
  12. Trolls talking to trolls.

    ReplyDelete