The Post and the Times report the NAEP scores!

THURSDAY, MAY 8, 2014

What we were talking about: Yesterday, new reading and math scores were released from the National Assessment of Educational Progress (NAEP), the widely-praised “gold standard” of domestic testing.

We refer to Grade 12 scores from the 2013 testing. Yesterday, we discussed the new math scores (click here).

This morning, the New York Times and the Washington Post tried to report the new data.

At the Washington Post, education reporter Emma Brown scattered errors, omissions and imponderables all through her report. Consider this early passage:
BROWN (5/8/14): Also called the nation’s report card, NAEP is widely regarded as the most consistent measure of U.S. student achievement over time. Since the 1990s, it has been administered every four years to high school students and every two years to students in the fourth and eighth grades.

Younger students’ results on the 2013 NAEP were released in November and showed incremental progress, continuing a slow but upward long-term trend. Twelfth-grade performance, by contrast, has been stagnant in recent years, and senior achievement in reading has declined since the early 1990s.
In fact, the NAEP conducts two parallel studies, each of which is widely discussed. The so-called “Main NAEP” did start in “the 1990s.” But the so-called “Long Term Trend” assessment started in 1971.

Charitably, you can write that off as a minor type of omission. But how about the highlighted statement? Is it true that “achievement in reading has declined since the early 1990s” on the twelfth-grade level?

We’re not sure. Here’s why:

First, it doesn’t really make sense to consider these scores without engaging in “disaggregation”—without checking the scores which have been attained by various student groups.

How have black kids done? How about Hispanics? Brown doesn’t bother with this.

Second, twelfth grade comparisons are complicated by changes in drop-out rate. Later in her report, Brown notes that Arne Duncan announced this week that last year’s graduation rate was the highest ever.

It’s good when fewer kids drop out, but it tends to weaken the overall twelfth grade population pool. In theory, this complicates comparisons over time.

Brown doesn’t mention this either. Meanwhile, note the way the New York Times’ Al Baker blows right past this point:
BAKER (5/8/14): In reading, 38 percent of seniors across the country achieved proficiency last year—compared with 40 percent in 1992, the first year for which data on seniors was available. The lack of progress was striking since elementary- and middle-school students have shown some growth during that time, as have graduation rates, suggesting that learning gains were wearing off in the high school years even as more students were earning diplomas.
In that passage, Baker puzzles over the drop in proficiency rate even as he notes that graduation rates have improved. It doesn’t occur to him, even theoretically, that the two factors could be related.

Are the two factors related? Given the way our press corps works, we don’t know the answer to that, and we never will.

As we look at the disaggregated data, another point jumps out from the reading scores, which start in 1992. Blacks, Hispanics, and Asians all showed weirdly large drops in average scores from 1992 to 1994. This suggests the possibility that sampling errors may have occurred in the first year of testing.

Adopting 1992 as the starting point, Brown reports gloomy news about reading. By way of contrast, if we take 2002 as our starting point, these score gains have occurred:
Score gains in reading, Grade 12 NAEP, 2002-2013
White students: 5.48 points
Black students: 0.71 points
Hispanic students: 3.58 points
Asian students: 11.45 points
Have those score gains been retarded by declines in the drop-out rate? We don’t know and we never will, given the norms of the press corps.

If you want to evaluate progress in schools, you pretty much have to disaggregate scores. On the Grade 12 level, the drop-out rate is a complicating factor.

It’s also true that initial scores in the 1992 reading testing seem to have been peculiarly high. If we were running the Post or the Times, we’d want our reporters to question NAEP officials on all these points.

Alas! At the Post and the Times, education reporters mainly seem to be trying to make it through the night. As a general matter, we stopped having a “press corps” long ago. Basic competence is especially low in the reporting of test scores.

How do Grade 12 math scores look? Pretty good! See yesterday’s post.

First, you cry: For unknown reasons, Brown did include the following point from one of our “education experts:”
BROWN: Some analysts contend that the 12th-grade scores are evidence only of an unsurprising truth: that high school seniors are not motivated to try their hardest on tests in which they have no real stake.

“We all remember exactly how engaged your 17-year-old high school senior is,” said Frederick Hess of the conservative American Enterprise Institute. Hess said skepticism about high school results should serve as a reminder not to read too much into younger students’ scores, as well.
According to Hess, “high school seniors are not motivated to try their hardest on tests in which they have no real stake.”

That may be true, of course. But presumably, it has always been true. That tends to wash things out.

Theoretically, if today’s unmotivated seniors know more than earlier unmotivated seniors did, they would tend to perform better than the earlier group, despite their ongoing lack of motivation. This undermines the attempt at a point by Hess, who may be experiencing a delayed onset of spring fever.

Also this:

Yesterday, we saw substantial score gains in Grade 12 math recorded by all four major demographic groups. The lack of motivation presumed by Hess didn’t stop these gains from occurring.

“Nothing gold can stay,” Frost insisted. When journalists like Brown talk to experts like Hess, few good things can result.

10 comments:

  1. OMB (Fat Drunk and Stupid* in the Howler's Gin Joint Tonight)

    We have been treated to a series of late in which we are told by educator par excellence BOB that "lazy liberals" feel good about articles "finessed" to make them feel good about school resegregation.

    How would you, the BOBreader, characterize the continuing effort of BOB to present numbers in a way that make you feel good about
    the performance of kiddos in America's public schools. Which "tribe" is BOB trying to make feel good about themselves?

    How does BOB finesse his numbers and make this tribe feel good. Anytime the message is unpleasant he attacks the messenger. Hence even good guy Kevin Drum walked into the gin joint and made a big mistake.

    In this piece BOB says this:

    "Is it true that “achievement in reading has declined since the early 1990s” on the twelfth-grade level?....We’re not sure. Here’s why:

    First, it doesn’t really make sense to consider these scores without engaging in “disaggregation”—without checking the scores which have been attained by various student groups....How have black kids done? How about Hispanics? Brown doesn’t bother with this."

    Then BOB proceeds to give you figures from the last ten years disaggregated to show gains in reading. See. nasty old Emma Brown
    is hiding things in her telling and BOB gioves you the latest scoop.
    Problem is the disaggregated numbers were on his own website since last night in a comment we made. And BOB could have run the numbers himself.

    Here's BOB in this post:

    Score gains in reading, Grade 12 NAEP, 2002-2013
    White students: 5.48 points
    Black students: 0.71 points
    Hispanic students: 3.58 points
    Asian students: 11.45 points

    Here is what BOB knows but won't tell you in order to finesse his own lazy rubes into feeling good.

    Score losses in reading, Grade 12 NAEP, 1992-2013
    White students: 0 points
    Black students: 5 points
    Hispanic students: 3 points

    We'll leave out those "tigers." They are leaving us behind anyway.

    And please BOB, quit hiding behind the dropout rate. We are supposed to feel good because scores are "dragged down" by educating kids we never should have lost in the first place?

    KZ

    * "Fat, Drunk, and Stupid" Even Dean Wormser is entitled to get one right.

    https://www.youtube.com/watch?v=bK-Dqj4fHmM

    ReplyDelete
    Replies
    1. Does Bob care about drop-outs?

      Apparently not since those pesky "at-risk" kids are staying in school and dragging down today's test scores, which would otherwise mirror the Polish Miracle.

      Delete
    2. We disagree. We think BOB cares about them. We think he also uses them as an excuse, hence this line from yesterday's post:

      "We don’t normally work with twelfth-grade scores, given the interpretive difficulties introduced by the drop-out question."

      We are definitely "suggesting, implying, and otherwise seeming" to say he ignores the final year of testing in both the Main and Long Term NAEP results because they don't fit his meme. They don't make "his" rubes feel good. They speak ill of "his" guild.

      BOB did a long series back in, we think, 2006 about the folly of the Los Angeles school system in requiring algebra to graduate because of its impact on increasing the dropout rate. We know from that he cares.

      KZ

      Delete
    3. Plus, the post numbering issue.

      Delete
  2. Bob explains that the 1992 12th grade scores are suspect. Still, he could have listed the scores from 1994-2013.

    I don't see why to discount the drop-out rate as having a negative impact on scores.

    What does "nothing gold can stay" mean? Things are always changing? Through suffering can come greater good? Anything good is always fleeting? Frost is a great poet.

    From Hess, the idea that conservatives don't care about anything unless it serves them, I think is dead on.

    ReplyDelete
    Replies
    1. When you do a better job of retaining kids who might otherwise drop out, those kids tend to be among the lower performers. That means including them will lower the averages. So, as retention rates go up, mean performance will go down, even though you may be doing a better job of educating more kids. That is why looking at just the test scores alone can be misleading.

      Nothing gold can stay means that if you want to do a better job of educating more kids, you have to accept that your mean scores will go down a bit.

      It is the same tradeoff that happens when more kids go to college. The ones going now (who might have skipped it in the past) are going to be more marginal students so they will tend to decrease graduation rates and have lower gpas than when they were discouraged from attending in the past. That means the education stats will look worse, even though a broader segment of the high school population is going on to college.

      Kicking out the kids that make your stats look bad always results in higher scores but is it the best thing to do -- for them and for our society?

      Delete
    2. I agree the decrease in drop-out rate will negatively impact scores. I don't understand why KZ has an issue with this, but maybe he can explain.

      I meant what does Frost mean? I think Bob was using it in reference to Brown/Hess, but I'm not sure how.

      Delete
    3. We have yet to see dropout figures we believe. Putting aside our fictional character we spent a number of years working on this issue closely in a major state. States and local districts disappear students at the high school level faster than BOB would have you believe MSNBC analysts disappear facts.

      We would agree that greater retention would have a theoretical negative impact on test scores. We haven't examined the numbers closely, but we believe the claim is being made that the lowest quartile of 17 year olds are showing improvement, which would suggest that the reverse is happening. The bottom is not dragging scores down, they are propping them up. We'll take some time,
      check this out, and report what we think we find.

      Our beef is not with taking a look at how retention rates may relate to scores, it is BOB using them as an excuse not to look at 12th grade scores at all in the past, then suddenly used them when they fit his meme. But that beef pales compared to BOB cherry picking his numbers to suit his meme while bitterly accusing others of either doing the same or being less intelligent than he is.

      But thanks for asking.

      KZ

      Delete
    4. KZ's a retard. That's pretty much the short and long of it.

      Delete
  3. I'm curious how much the effect of more privileged students moving to private schools in the last twenty years might have on overall results. I work at such a school and none of our students are tested. They would certainly score well above national averages. Still, I don't know how much the movement of such students would effect the overall pool. Seems a point worth considering, though.

    ReplyDelete