Supplemental: But the NAEP conducts two high school studies!

SATURDAY, SEPTEMBER 26, 2015

Which study is correct:
Are high school seniors doing better in math? We have no way of knowing!

We do know where the data are. The best data, from the so-called “Main NAEP,” seem to suggest that high school seniors are doing better in math—have been recording score gains in math over the past twenty-five years.

“Hold on!” the dead-enders will cry. What about the other NAEP study? What about the smaller score gains on the so-called Long-Term Trend Assessment?

The question is perfectly valid. In the study that’s called the Main NAEP, the NAEP tests high school seniors in reading and math. In the study that’s called the Long-Term Trend Assessment, the NAEP tests 17-year-old students, no matter what grade they’re in. (Most are juniors or sophomores.)

The two NAEP studies use different math tests. Because we read American newspapers, we’ve never encountered a real attempt to explain the difference between the math tests.

It’s our impression that the Main NAEP test is geared to the math curriculum as it currently exists, while the Long-Term Trend math test has largely stayed the same over its forty-plus years of use.

Warning! That’s our impression, but it could be totally wrong. As noted, we read the American press corps, which uses the NAEP, almost exclusively, as a vehicle for driving narratives about our failing schools, which clearly ought to be charters.

(For the record, we aren’t opposed to charter schools in any way at all. We are opposed to ludicrous claims which get made on their behalf.)

Whatever! The Main NAEP tests students in Grade 12. The Long-Term Trend Assessment tests students 17-year-old students—mostly juniors and sophomores.

Here’s the conundrum—score gains have been larger in math on the Main NAEP. Indeed, 17-year-old black students have recorded no score gains in math at all on the Long-Term Trend Assessment.

Adjusting for a procedural change which occurred in 2004, this is what score gains look like on the Long-Term Trend Assessment:
Gains in average math scores, 1990-2012
Long-Term Trend Assessment, 17-year-old students
National public schools

White students: 6.74 points
Black students: -0.32 points
Hispanic students: 8.45 points
Asian-American students: 19.84 points
For whites and Hispanics, those score gains aren’t huge, though they also aren’t non-existent. For black kids, there are no score gains at all.

A different pattern seems to obtain for black kids on the Grade 12 Main NAEP. And just for the record, these are the score gains recorded by 13-year-old and 9-year-old students on the Long-Term Trend Assessment:
Gains in average math scores, 1990-2012
Long-Term Trend Assessment, 13-year-old students
National public schools

White students: 18.33 points
Black students: 19.17 points
Hispanic students: 18.22 points
Asian-American students: 41.43 points

Gains in average math scores, 1990-2012
Long-Term Trend Assessment, 9-year-old students
National public schools

White students: 19.31 points
Black students: 22.25 points
Hispanic students: 20.95 points
Asian-American students: 30.75 points
Judged by normal rules of thumb, those are large score gains. (Warning! There’s a very large Standard Error for Asian-American scores in 1990, when the “N” for that group was small.)

Thirteen-year old students have recorded large gains on the Long-Term Trend math test. The same is true for 9-year-old students.

That's true of black students in each group. This leads to some obvious questions:

Why have score gains on the Long-Term Trend been smaller among 17-year-old students? Why have 17-year-old black students shown no gains at all?

Black kids have shown substantial gains everywhere else, including at all grade levels on the Main NAEP. Why are scores for 17-year-old black students flat? Why are gains by white and Hispanic students relatively small at that age?

We don’t know how to answer those questions. We’ll stick with our standard excuse:

Such questions are never asked by the American press, which never presents a serious discussion of any such topics at all. Quite plainly, the American press corps uses the NAEP in exactly one way—to cherry-pick data in support of the claim that our schools are stagnant or failing.

Some journalists may be doing this deliberately. Almost surely, most of our “education reporters” have never looked at a set of NAEP scores in their lives. They simply channel the standard claims which come to them from “educational experts,” almost all of whom are on the dole from Bill Gates and the rest of the billionaire “education reform” funder class.

As far as we know, Gates is fully sincere in his efforts. But on the whole, the narratives which control our discourse about public schools serve corporate and political interests which are blatantly obvious. It’s all about spreading gloom and doom and refusing to report the overall, rather large gains in test scores.

Why are math scores somewhat flatter among high school students on the Long-Term Trend? We have no idea! It may be because of the nature of the math test used in that study. If we had an actual press corps, somebody would have asked!

Why have black students recorded large score gains in math at ages 9 and 13, but not at age 17? Just a guess—the effects of declining drop-out rates would be in effect at age 17. For various reasons, we’ll guess that the black student population might have been more affected by that (desirable) change than the other three groups. Why haven’t reporters asked?

We’ll finish with a basic point, the most basic point of all:

Our mainstream “education reporting” is largely propaganda. It has been that way for a very long time. Whatever anyone’s intentions may be, it’s clear what interests are served.

You’re gloomily told, again and again, that our schools are stagnant or failing. You’re told that miraculous Finland is just sooo much better.

You’re told that nothing has worked in our schools. You’re told that we need more charter schools and a whole lot more “reform!”

Overwhelmingly, the pattern on our “federal tests” is hard to square with that narrative. For that reason, you’re never told about the score gains which predominate in both NAEP testing programs.

Routinely, you’re told about the gaps; the gains go unreported. When reporters do talk about gains, they turn to the Long-Term Trend Assessment, 17-year-olds only.

That’s what Michael Petrilli did in that horrible blog post. And by the way, note this:

Above, we said we were “adjusting for a procedural change which occurred in 2004.” At that time, the NAEP introduced “accommodations” which let them test kids with certain types of challenges—a relatively small group of kids whose counterparts hadn’t been tested before that.

Even with “accommodations,” including these kids in the NAEP tends to drive average scores down a few points. If you forget to make the basic adjustments, you’re absent-mindedly making test scores look “flatter” over the years.

Michael Petrilli forgot! If you look at his graphic for 17-year-old students, you’ll see a superscript 1 (essentially, an asterisk) appended to the scores for years before 2004.

But he never explains what the asterisk means! What kind of “expert,” in any field, conducts his business like that?

The asterisk meant he was flattening scores, in line with standard procedure. For many years, that has been the primary business of our “education experts” and the reporters who peddle their scripts.

They won’t tell you about score gains. Instead, they happily say that we need charter schools.

For ourselves, we aren’t opposed to charter schools. We are opposed to scams.

Where do test scores come from: For all data from the Long-Term Trend Assessment, just follow these easy steps:

First, click this. Then, click on LTT NDE (Long-Term Trend NAEP Data Explorer).

Click on “I agree to the terms above.” From there, you’re on your own! It’s just like Dylan said!

29 comments:

  1. It may well be the case that our schools are essentially stagnant in terms of how well those who finish high school have been educated. The idea that these test results are determined only by teachers in school is simply nonsense - there is a strong correlation between education level of parents and kids' achievement in schools. Whether kids learn a lot at home or just get their attitudes there - or from their peers - is important, but not directly relevant to how schools can be improved. Thus as the general population became better educated through earlier parts of the 20th century, test scores were bound to improve, regardless of how good teaching was. But high-school graduation rate actually peaked around 1970 and has slightly decreased since. I believe college degree attainment has continued to improve, but more slowly than in the past. These things must affect test scores and there is little hope of understanding school performance without taking them into account.

    ReplyDelete
  2. Your Howler Commenters Get Results!

    ReplyDelete
  3. If the NAEP scores are internally inconsistent, maybe they're not such a "gold standard".

    ReplyDelete
    Replies
    1. What do you mean by "internally consistent"? Which NAEP are you talking about?

      Delete
  4. The math Long-Term Trend Assessment, 17-year-old students shows a very slight decline from 1990 - 2012. OTOH Bob says, "A different pattern seems to obtain for black kids on the Grade 12 Main NAEP."

    ReplyDelete
    Replies
    1. Tell us again how great the Asian and white kids are doing, Dinky.

      Delete
  5. OMB (Still Disappearing With the OTB)

    Oh, them Golden Standards
    Oh, them Golden Standards
    Golden Standards BOB's Cherry Picked
    To Serve His Golden Meme

    Lord Have Mercy. Righteous is the BOB that answers the Valid Question though it be cried by only Dead Enders.

    It might seem,,,,no, it might be suggested....no, it is fact, that the Dead Enders, who are in double fact more real than the crying analysts, have asked far more than one question for their beloved BOB to address. In the course of this winding Sermon on the Mounting Test Scores which has now stretched from August to Autumn in a Four Series Eighteen Verse Back to School Revival, the biggest unanswered questions are why has BOB chosen to pose questions which were never asked? Why has he chosen to refute statements never made? We still just don't know.

    Whatever! BOB has chosen to address one of the cackling cries of the Dead End chorus by admitting there are two NAEP tests without explaining, why, if there are two, one of them Gold Standard assessments had to wait until Part 18 (count 'em yourself) to appear.

    We, of course, will wait until Part 2 and maybe even Part 3 of this little hymn to address the fact that even in his account of the late appearing second Gold Standard Test Bob is cherry picking, and Gack! and Alas!....There are not two tests, but four!

    ReplyDelete
    Replies
    1. Did you know that the SAT has 3 parts? That doesn't make it three tests. There are not four tests -- there are more. If you go to the NAEP website you will see there are many subjects it tests, not just Math and Reading.

      Delete
    2. That is technically accurate, to borrow a frequent term from BOB.

      There are four tests in two subjects that can currently be used to assess progress of American public high schools by viewing average test scores of randomly sampled students. The two subjects are math and reading.

      That said, since the Golden Standard Era of testing began, both the Long Term Trend and Main NAEP have tested seven other subjects. Unlike your SAT example, which is one test with three parts, each NAEP test, whether in the Long Term or Main series, is a separate test given to a separate group of students.

      Long Term Trend tests have been administered to 17 year olds in other subjects than math and reading, but not since 1999. The Main NAEP is given to selected 12th grade volunteers in seven other subjects. In our view that is nine Main NAEP 12th grade tests.

      That said, it would be more accurate to say the OTB has disappeared 9 of 11 tests.

      For our part we would be happy to detail the results of those tests of 12th graders for which tests have been given in more than one year.

      Gains in average test scores, 1990-2012
      Main Naep 12th Grade students
      National public schools


      Civics: - 1
      Economics: + 1
      Geography: - 2
      History: + 3
      Vocabulary: - 1

      Writing and Science given only once: 0

      Using BOBmath and the patented Rough Rule of Thumb we add total score gains together, subtract score losses, and divide by total of subjects tested.

      The total gain = Zero. Avg. Gain = Zero.

      We now apply tested vocabulary words from:

      Journo Anderson and Billionaire Pimp Petrilli:

      NAEP 12 Grade Tests - FLAT! (Stagnant for those who prefer polysyllabic piffle).

      (And thanks for asking!)



      Delete
    3. See, and here is the problem with your comments. You entirely ignore the purpose of each post -- why it is written, what it is about, in favor of your own agenda.

      I am tired of wasting time on you. I think I've said enough to show you don't know what you're talking about and your main goal is just to attack Somerby. You and mini-KZ can have the floor to engage in mutual admiration (or whatever it is that you do).

      Delete
    4. Can't do the math, is it? You can check ours.

      To quote your beloved BOB:

      "... click on LTT NDE (Long-Term Trend NAEP Data Explorer).

      Click on “I agree to the terms above.” From there, you’re on your own! It’s just like Dylan said!"

      KZ

      Delete
    5. If you were to subscribe to Kaiser you could reorder your meds online and get them by mail. Then you wouldn't run out.

      Delete
  6. OMB (Still Disappearing With the OTB) Part 2

    Two, Four, Six, Eleven
    BOBmath Takes You Straight to Heaven

    We thought we would devote Part 2 to the Cherry Picking Problem plaguing bewildered believers in BOBworld. But Anon. @ 11:16 brought fourth a challenge to our assertion that there are four tests instead of the two proclaimed herein by the OTB.

    Is there a single Gold Standard Test or many tests which meet a single Gold Standard?

    In the Back to School Revival we have been worshipfully following BOB has been concerned with questions concerning the performance of our high schools. More specifically the performance as measured and mismeasured and as reported, misreported, distorted and disappeared by our mainstream media, professors, and experts Pimps.

    During this Revival, there was only one accurate measure. The 12th grade NAEP Math test. It was to BOB, the Gold Standard test. For this Revival spanning four Series, 18 Posted parts and supplementals. Before this series, we think one could make a strong argument that the 8th Grade NAEP Math Test was the Gold Standard. We base that judgement on what seems to be most frequent invocation of test results in any post containing the magic phrase "___________ doesn't care about Black kids!"

    The grade may change depending on the journalist BOB is cuffing aside, but invariably in later years, BOB has shown a distinct favoritism for math scores/Main NAEP test.

    In this post, after many preceding posts in which it disappeared, BOB has focused on another series of NAEP tests, the Long Term Trend test, which, given that it came first, and uses a testing methodology that allows modern scores to be compared back to the 1970's ought to deserve equal billing as a Gold Standard test. The same government body administers it, after all. As BOB notes, it is given to students based on their age at the time of the test.

    Why did BOB leave out mention of the Long Term Trends test? Is it any less valid? Is there something about it that makes it Silver instead of Gold compared to the Main NAEP test series? We don't know. We will make no excuses. We will show what we think is the reason when we get to Part 3 during cherry picking time.

    For now let us note that the two NAEP test series have covered many, many subjects over time at different age and grade levels of administration. But only two subjects have been tested in each series at all grades and ages overf the course of each test....math and reading.

    BOB, for some reason, has disappeared all tests and scores in reading.

    There are four tests you can use to measure progress of high schools over time. The LTT Reading, LTT Math, Main Reading and Main Math.
    One has been with us all month. One has been recently found. Two have gone missing. Both in Reading.

    Where can they be? When did we last see them?

    Back in 2011 BOB invoked reading scores from NAEP regularly. See the Series NEWT VERSUS NAEP in December of that year. BOB used NAEP scores (disaggregated, of course) to level Gail Collins. But lately it has been all about math.

    Back in the (very near and far) past it wasn't just Reading that had disappeared. It was tests involving high school students. The Dead Enders used to cry BOB never talked about any NAEP testing of 12th graders or 17 year olds.

    BOB and his believers had their standard excuses. 12th graders did not care about the test and did not try. Upper grades were flooded with kids who would have dropped out years ago and were now lowering the scores (even though they were raising them four years earlier as 13 year olds and/or 8th graders).

    Then came journalist, Nick Anderson, quoting "expert" Michael Perilli, and BOB is all over the 12th grade test and even mentions the LTT for 17 year olds. He is out to prove these two are wrong when they say progress has stalled and scores are stagnant.

    And our OTB is just the man to do it. He can do the math. As long as the test is about math.

    ReplyDelete
    Replies
    1. This is what a comment looks like when someone cares more about attacking another person than discussing education of children.

      Delete
    2. Poor @ 6:43 let her/his guard down. Is BOB discussing education of children in this post? We thought he was pissing on the boots of plutocrat pimps and journalists with worm eaten brains.

      We could be wrong. That said we just don't know.

      KZ

      Delete
    3. Right. You know nothing and yet you keep making noise here. Why not shut up and try to learn something?

      Delete
    4. @ 10:22

      "This is what a comment looks like when someone cares more about attacking another person than discussing education of children." @ 6:43

      Delete
    5. "I know you are, but what am I?" -- Peewee Herman

      Delete
  7. First rule of commenting: your comment should not be longer than Bob Somerby's post.

    ReplyDelete
    Replies
    1. Which at the Howler is an impossible rule to brak.

      Delete
  8. Dylan Stableford said today:

    "...Trump has a razor-thin lead over retired neurosurgeon Ben Carson...The same poll found. Hillary Clinton's. lead on the Democratic side has largely evaporated, with the former secretary of state's edge on Vermont Sen. Bernie Sanders down to 15 points..."

    If anyone but Hillary were running, 15 points would be considered a substantial lead.

    ReplyDelete
    Replies
    1. Our guess? Such cluelessness from Clinton supporters may represent her “biggest problem.”

      Delete
    2. Since we have reintroduced the campaign, on a topic Somerby mentioned briefly, Bush's tax proposal, another surprising journalist seems to be doing his job. And another blog, too.

      http://crooksandliars.com/2015/09/thats-just-way-it-jeb-bush-explains-why-1

      Delete
    3. The New York Times this morning suggests that liberal feminists are enamored of Carly Fiorina. In what universe? They quote Jennifer Weiner and someone who writes for Cosmopolitan magazine. I cannot imagine any liberal feminist wasting even a moment thinking about Carly Fiorina, no matter how many times she talks back to Trump. She is not only a female candidate, she represents the War on Women exemplified by all of the Republican candidates. There have always been female Republicans. There is no novelty in a woman standing on a platform and speaking against the interests of women.

      Delete
    4. Commenter @ 10:02 writes like Bob Somerby talking about the gender pay gap.

      Delete
  9. OMB (Still Disappearing With the OTB) Part 3

    We mordantly chuckled reading the previous post. The Dead Ender who forced BOB to make the Long Term Trends Test reappear is Kevin Drum. It took five days from the date of Drum posting the LTT scores on September 21 to this post. BOB spent the better part of a week wandering the wilderness in search of cherries.

    Nick Anderson's article forced BOB to use high school test scores for the first time in ages, Drum's post may have caused the OTB to feel the need for a new rebuttal as well. Alas, he could not find black student test numbers in LTT to do justice to those in the Main NAEP math test. He simply pronounced the whole thing a conundrum and gave his "standard excuse," it's the media's fault for not discussing the topic and relieving our ignorance.

    We see here a sad array of the best cherries BOB could find. His cherry is the selection of 1990 as a starting year. That way he could show gains for everyone on the 17 year old NAEP math test except for black students. He threw the black test takers under the bus.

    "How could that be?" you ask. "BOB cares about black kids."

    Not especially. BOB seems to care about disparaging people who write about a topic he feels he knows more about than they do. BOB appears to care less about black kids than saying others don't care about them.

    The point of this long series is to attack articles that angered BOB and those who authored them. None pissed BOB off more than Nick Anderson, his expert Perilli, and now Kevin Drum. This isn't about BOB's concern for black kids, it is about refuting the contention that test scores have stalled in public high schools.

    Rising black test scores have been one of BOB's favorite cherries over the years, particularly 8th grade NAEP math. He discovered that 12th grade NAEP cherries worked quite well to refute Anderson/Perilli.

    Unfortunately along came Drum showing the NAEP LTT math scores flat for black kids since 1990. Drum also called American high schools a pit. This caused the analysts, BOB said, to cry. BOB does care about his analysts. So he set out to prove Drum wrong too in today's post. By selecting the same year as Drum, BOB can show gains for white students and Hispanic students on the NAEP LTT math test. BOB declares victory for American Public High Schools over Drum and blames the press for not explaining the disparity between the Main and LTT math scores of black high schoolers.

    There was no particular reason for Drum or BOB to select 1990. If BOB had selected 1992, which gives a nice round twenty years of testing in which to measure progress, black kids on LTT math would have shown a 2 point gain. So would white kids and Hispanic kids. So why not do this? Because a 2 point gain is not significant according to the NAEP. On a 500 point scale, 2 points shows no gain or loss. What words come to mind? Stalled? Stagnant? The terms Anderson and Perilli used? But 1990 allows significant gains to be claimed in math for 3 of the four ethnic groups. BOB started from a low point to measure his gains.

    The same cherry picking to serve the meme caused Bob to ignore NAEP reading entirely. Those results are also flat for the most part. On the Main NAEP reading tests one group of seniors did score significantly lower in 2013 than they did back in 1992. Sadly, it was black kids.

    On three of the four NAEP Gold Standard high school tests, the results are exactly what Anderson and Perilli claimed them to be. On the Main NAEP tests in subjects other than reading and math, the test results are flat.

    Are American high school seniors doing less well in math? Bob asked. Nobody suggested they were. Nobody suggested anybody was doing worse in math or reading. They suggested scores were flat. Progress had stalled. The "P" in NAEP stands for progress. According to the NAEP, on 10 of 11 tests given to high school students, 9 of which are given to seniors, the results are flat.

    ReplyDelete
    Replies
    1. Take your meds.

      Delete
    2. Will they make us higher than NAEP results on 10 of 11 tests?

      KZ

      Delete
    3. Sadly, they won't help you recognize the point of an essay, sufficiently to pass a NAEP reading test yourself. But they will cut down your word count and prevent you from making such a fool of yourself here daily.

      Delete