DISAGGREGATION NATION:
Molly Ivins made a mistake!

THURSDAY, SEPTEMBER 27, 2012

Part 2—Why Maine seemed so special: Friend, do you want to use test scores in reading and math to judge a school or school district? To judge an entire state’s schools?

To compare school performance in one state to school performance in another?

If you have reliable test scores to use, that’s a sensible thing to do. But if you want to evaluate schools that way, you have to “disaggregate” test scores.

You can’t compare the overall scores from some state to the overall scores from another. You have to break the test scores down to see how different groups of students in the two states did.

If one state has a high proportion of low-income kids, that state may have low overall scores, even though those low-income kids are outscoring their peers around the nation. And uh-oh! Because white kids currently outscore black and Hispanic kids, a state with a high proportion of white students has a built-in advantage.

If you want to rate the true performance of a state’s schools, you have to “disaggregate” its scores. In her unfortunate book, As Texas Goes, Gail Collins sings the praises of disaggregation—then fails to practice the technique as she rolls her eyes at the state of Texas for its low overall scores. (See THE DAILY HOWLER, 9/26/12.)

That’s par for the course with a writer like Collins. But uh-oh! Twelve years ago, the late Molly Ivins made the same mistake in one of her syndicated columns.

Collins quotes that Ivins column in her massively bungled new book. For that reason, it may be worth taking a trip back in time to see what happens when journalists fail to practice “disaggregation.”

Twelve years ago, Molly Ivins made a substantial mistake.

Given the time frame, her mistake may have been understandable. It was a mistake all the same.

The episode started when the Rand Corporation released a major report on the performance of the various states’ public schools on the National Assessment of Educational Progress (NAEP). The report appeared in July 2000. Rand heaped praise on the Texas schools for the high performance of its various students—after disaggregation.

Rand stressed the progress Texas schools had made on the NAEP between 1990 and 1996. Beyond that, it said that black and Hispanic kids in Texas were scoring right at the top of the nation as compared to black and Hispanic peers in the other states.

That sounded like good news for Texas. But at the time, a presidential campaign was under way, featuring the Republican governor of Texas. And sure enough! As Ivins described the Rand report, she stressed the idea that Governor Bush had nothing to do with the progress displayed by kids in the Texas schools.

That may or may not have been true. But in the process, Ivins made grossly misleading remarks about her state’s public schools.

The Texas schools were still “slightly below average,” the liberal columnist wrote. At one point, she used a statistic which made it sound like things might be somewhat worse:
IVINS (7/29/00): The study shows that Texas is improving fast. Our scores are still slightly below the national average (27th of the 44 states that use the national tests); but we're moving up—second in improvement on math scores, and our minority kids are outperforming others around the country.

So the governor stood up and took a bow. Excuse me.

The report was based on tests between 1990 and 1996. One thing we know about education reform is that it takes 10 to 20 years before we can see any results, before we can tell whether what we've tried is working.

The real story on how our schools rocketed from abysmal to only slightly below average in a mere 30 years starts in 1968, with a lawsuit...
Do we really know that education reform “takes 10 to 20 years before we can see any results?” Ivins may have stretching a bit, denying credit to Bush.

But according to a wisecrack by Ivins, the Texas schools had “rocketed from abysmal to only slightly below average in a mere 30 years.” And she included a statistic which sounded gloomier till: She said Texas was still “27th of the 44 states that use the national tests.”

That statistic can be defended as technically accurate. But it was grossly misleading.

Let’s forget about credit and blame. Instead, let’s consider Ivins’ claims about the Texas schools.

Were Texas schools “below average” in 2000, when Ivins’ column appeared? Were they “below average” in 1996, the last year considered in the Rand report? In fact, once you “disaggregated” their scores, Texas students were outscoring their peers around the nation by very significant margins as of 1996.

Once again, here are some of the 1996 scores which led Rand to praise the Texas schools, followed by the corresponding scores from the year 2000, when Ivins wrote her column:
Texas students, fourth-grade math, 1996 NAEP
White kids: First in the nation (of 43 participating states)
Black kids: First in the nation (of 35 states)
Hispanic kids: Second in the nation (of 25 states)

Texas students, fourth-grade math, 2000 NAEP
White kids: Second in the nation (of 40 participating states)
Black kids: First in the nation (of 32 states)
Hispanic kids: Second in the nation (of 21 states)
On the basis of those 1996 scores, the Rand study heaped praise on the Texas schools. But Rand had disaggregated the data.

Misleadingly, Ivins did not.

To those who read the Rand report, the strong performance of Texas students wasn't a mystery. Rand stressed the high performance of Texas kids. Other journalists were able to see this.

Example: A few days before Ivins’ column appeared, Melanie Markey did a detailed report on the Rand study in the Houston Chronicle. With perfect accuracy, Markley described Rand’s findings about the high performance of the Texas schools.

Texas students “outperformed all other states when variations in demographics were taken into account,” Markley correctly reported:
MARKLEY (7/26/00): Texas a qualified No. 1 in U.S. education study

Texas students, building on reforms launched in the 1980s, outperformed peers with similar backgrounds on national tests measuring education progress, the California-based Rand research group reported Tuesday.

Smaller class sizes, well-funded preschool programs and adequate resources for teachers are key factors that separate top-ranked Texas from the likes of California, which came in last among 44 states participating in the three-year private study, Rand found.

[...]

Based on an analysis of the National Assessment of Educational Progress tests given from 1990 to 1996, the study ranks states by raw scores, scores that compare students with similar backgrounds and score improvements.

When comparing raw scores, Texas ranks below most other states. Maine, North Dakota, Iowa, New Hampshire and Montana are at the top, while Mississippi, Louisiana, California, Alabama and South Carolina are at the bottom.

Researchers said the states with high raw scores tend to have fewer minorities, higher family incomes and better-educated parents.

But Texas ranked second only to North Carolina in improved scores—about twice as great as the national average—and outperformed all other states when variations in demographics were taken into account.
Texas “outperformed all other states when variations in demographics were taken into account,” Markley wrote. That is to say, when test scores were disaggregated.

After disaggregation, Texas kids were outperforming their peers in all other states! This point was made a bit more clearly in Anjetta McQueen’s report for the Associated Press:
MCQUEEN (7/25/00): Researchers used specific categories to see how well each state is educating children regardless of background. For example, on the 1996 math test of fourth-graders, black students in Texas ranked first when compared with blacks in other states; Hispanic students in Texas ranked fifth. Meanwhile, California's black students ranked last; California's Hispanic students ranked fourth from the bottom.
McQueen seems to have erred on one point. According to official NAEP data, Hispanic kids in Texas scored second in the nation, behind only Maryland, in fourth-grade math in 1996. To give you a rough idea of their relative success, they outscored the national average for their peers by more than twelve points—and by a very rough rule of thumb, ten points on the NAEP scale is often said to equal one academic year.

After disaggregation, how well were Texas students doing in 1996? Here’s where the three major demographic groups stood in fourth-grade math by the final year Rand studied:
Texas students, fourth-grade math, 1996 NAEP
White kids: Outscored white kids nationwide by 10.1 points
Black kids: Outscored black kids nationwide by 12.8 points
Hispanic kids: Outscored Hispanic kids nationwide by 12.4 points
On fourth-grade math, Texas kids were kicking the nation’s backside. How did Ivins end up saying that the state was still 27th, out of just 44 states?

Where did she get that gloomy statistic? Therein lies several tales.

Ivins’ statement can be defended as technically accurate. One lone graph in the lengthy Rand report compared the average scores the various states had achieved on all NAEP tests from 1990 through 1996. (For the Rand report, just click here. Scroll to page 14.)

In this one solitary graph, there was no attempt to “disaggregate” scores—to adjust for income, race or ethnicity. And sure enough! On this measure, Texas did finish 27th, out of 44 states. And surprise! These were the five top-scoring states, along with the percentage of their students who were white:
Top five states on Ivins’ preferred measure, with percentage of students who were white:
1. Maine (98)
2. North Dakota (96)
3. Iowa (93)
4. New Hampshire (97)
5. Montana (unavailable)

[...]

27. Texas (50)
Maine finished first on this ill-conceived measure—but then, 98 percent of Maine’s students were white! North Dakota finished second on this measure; its kids were 96 percent white. (The data for race come from 1992.)

Texas did finish 27th on this particular measure. The main reasons: The state had large numbers of low-income students. And its student population was only about 50 percent white.

Duh! As anybody could have guessed, success on this measure correlated strongly with the percentage of white kids in a gievn state’s schools. Presumably because it ranked Texas so low, Ivins pulled this statistic from a mammoth report which lavishly praised the Texas schools for outscoring all other states—after disaggregation.

Can we talk? In her column, Ivins dogged Texas for its rank on this single, ill-conceived measure. In essence, she was complaining that Texas had too many minority kids and too many kids with low incomes.

We’ll assume that Ivins didn’t fully understand that fact. But that was the nature of the statistic she pulled from that lengthy report—a lengthy report which explicitly stressed the high achievement of Texas kids if you compared such kids with their nationwide peers.

Final point: Was Maine really first in the nation, while Texas languished in 27th? Only because the state of Maine had so many white students! For more proof of Texas’ high performance, just consider this:

Was Maine really best in the nation? In fact, if you consider white kids only, Texas was generally outscoring Maine by the period under review! Here’s how white students from the two states ranked on a string of NAEP tests:
White students only, fourth-grade math, 1996 NAEP
Texas: First in the nation
Maine: Eleventh

White students only, fourth-grade reading, 1998 NAEP
Texas: Second in the nation
Maine: Twelfth

White students only, eighth-grade math, 2000 NAEP
Texas: Sixth in the nation
Maine: Eighteenth

White students only, fourth-grade math, 2000 NAEP
Texas: Second in the nation
Maine: Twenty-fifth
By 1996, Texas tended to outscore Maine, often by fairly large margins, even if you looked at white students only. Maine came in first on that overall ranking only because the state had almost no minority kids.

Texas finished 27th! In such ways, we get misled when we don’t “disaggregate” test scores. Ivins could have told the world about the relative success of all the kids in Texas. Instead, she cherry-picked a single statistic—a statistic which gave a grossly misleading picture of the performance of the Texas schools.

We will assume that Ivins may not have fully understood her topic that day. Twelve years later, in her unfortunate book, Collins praises the wonders of disaggregation—after which, she utterly fails to employ the praised technique.

Ivins’ column was greatly misleading, but she may not have understood. Twelve years later, what’s Collins’ excuse?

Why does this nonsense continue?

Tomorrow: Quoting Barbara Bush!

29 comments:

  1. Your argument is a good one, Bob, but let's take a step back and take a bit wider view.

    Minority students don't perform as well as white kids? Why is that? I'm sure one can make long and varied lists of reasons, but such a list wouldn't be complete unless it noted that states and school districts often provide substandard teachers, facilities, and resources to schools with high proportions of minority students.

    While disaggregation provides better apples-to-apples comparison of data, relying solely on disaggregated data lets legislatures and school boards off the hook for problems they had a hand in creating.

    In the very same paragraph where Ivins dinged the Texas school system for mediocrity, she also notes the gains made by the state's minority students. However, one can't simply accept the performance gap between white students and minority students as a law of nature. At some point, you have to recognize that minority kids deserve the same education as white children. Ivins perspective fairly balances recognition of performance with recognition of the performance gap.

    ReplyDelete
    Replies
    1. "states and school districts often provide substandard teachers, facilities, and resources to schools with high proportions of minority students"

      How would you propose that aggregate date gives any insight about this problem?

      [crickets]

      "At some point, you have to recognize that minority kids deserve the same education as white children."

      Yes! And the problem addressed by this series is that Gail Collins is singularly failing to do this.

      If you want to talk about what "minority kids deserve" it's best if you don't bullshit yourself and your readers with aggregate data that HIDES what those kids are doing, and falsifies the story on where they're having the most success.

      Delete
    2. How would I propose that aggregate dat[a] provide any insight about this problem?

      Simple. I don't. If a school system is providing a quality education to all students, there's no problem that requires the special insight of disaggregated data.

      Pie in the sky? Perhaps. But if a state school system helps create disparate results, we shouldn't excuse them on the basis of...disparate results.

      As far as "bullshitting myself" goes, I'm not recommending using ONLY aggregated data. I'm seeing value in a comparison of systems on both aggregated and disaggregated results--exactly what Ivins did in the cited passage.

      Delete
    3. Data, schmata!

      Delete
  2. Bob, sometimes you really get lost in the weeds. Not only are you selectively quoting a column from someone who has been dead for six years, you are selectively highlighting as well. If you simply reverse the highlights, you could write an essay saying that she was making the exact point you wanted her to make! Try it like this:

    IVINS (7/29/00): THE STUDY SHOWS THAT TEXAS IS IMPROVING FAST. Our scores are still slightly below the national average (27th of the 44 states that use the national tests); BUT WE'RE MOVING UP—SECOND IN IMPROVEMENT ON MATH SCORES, AND OUR MINORITY KIDS ARE OUTPERFORMING OTHERS AROUND THE COUNTRY.

    As for the argument that she is simply trashing Bush because that's what she did, I'd invite you to look at the education chapter in her contemporaneous book, Shrub. I believe it is titled, "The Bright Spot: Education". So she sure didn't trash him about education there. Since you didn't link to the article, it sounds like she was making the argument that there was a 1968 lawsuit that either opened up access to decent education in Texas or shifted some state money to lower income areas. I don't know, since you cut off the quote there. Either way, that lawsuit appears to be the main focus of the column.

    Trash Collins all you want. She is a clown, and she is around to defend herself. But you have lost your mind on this one. Sorry, but we can't go back to 2000 and have a disaggregation argument with Molly Ivins. She may well have agreed with you! But you have to have the good sense to let it go now.

    ReplyDelete
    Replies
    1. Anon, here's a link to the full text:

      http://www.creators.com/opinion/molly-ivins/who-deserves-credit-for-texas-education-improvements.html

      Delete
    2. That link seems to be broken. Here's where I found it.
      http://www.creators.com/opinion/molly-ivins/who-deserves-credit-for-texas-education-improvements.html

      It's a great column, passing credit around generously for a wide range of contributors of different political persuasions for responding positively to the famous San Antonio equal protection case. The phenomenon being described is the tremendous improvement over a generation, and the point is that George Bush could hardly take credit for improvements between 1990 and 1996. The most you can do is quibble with Ivins' phrasing, but the two references to the aggregate score being "slightly below average" are, to anyone not searching for things to argue with, overwhelmed by the dominant positives about the Texas schools. Moreover, as noted in a comment above, she does do some primitive disaggregation in noting that Texas minorities were outperforming minorities in other states. It is hard to expect more from any journalist.

      Perhaps Ivins was prescient: for some reason -- like maybe the George Bush accountability "reforms" -- the disaggregated Texas scores for whites, blacks and Hispanics between 1996 and 2011, have been more likely to lag behind the national rate of improvement done: of the 12 "cells" tested, six for each grade level (two subjects, three ethnic groups), Texas improvement beat the national average in four cases, tied it in one case, and lagged behind the national average in seven. For some reason, the spectacular Texas improvement in 8th Grade math by whites and blacks is not matched by similar performance in 4th Grade math. Indeed, it is quite the opposite in math for 4th Graders: each group's improvement substantially lagged behind the national improvement.

      One potential problem with disaggregation -- which I agree (despite its fundamental flaw of assuming each ethnic group to be the same in each state) is the best way to compare states -- is that it may reduce the reliability of the data.

      Delete
    3. The Real Urban LegendSeptember 28, 2012 at 7:48 AM

      Shorter urban-legend:

      Everything I said about TX two days ago was unfounded bullshit.

      So now I admit Bob's right, but I imply maybe the data's unreliable, on the basis of zero information.

      I also QUIBBLE: Sure deserving low income TX kids scores were better than average, and remained better than average as everyone in the nation improved -- but the average is catching up with TX, so TX isn't improving as much, it's "lagging behind" in its rate of improvement! How measurable and important is that "lag" I don't know, and you don't know either!


      Sure, TX is improving. But some are improving more! TX has an improvement gap!! I can't say how big, or important it is, but I think it retroactively proves Ivins correct!

      Yup, I now agree "disaggregation is the best way to compare states" -- so Somerby's correct -- but maybe I should leave you with this unfounded thought: the data are probably unreliable, yeah, that's it!

      You can't make me up, I'm real!

      Delete
    4. Feel better now?

      I conceded two days before my error in reacting without engaging the disaggregation argument. Guess you missed that but reacted anyway.

      The fact remains that the rate of improvement since the mid-90s is not as fast as it has been in the rest of the country. There's a legitimate question whether it represents a real slowing of progress or just a matter that a rate of improvement is bound to slow down when you're already at the top. I see no evidence that you have an answer to that question. Snark doesn't get it done.

      The fact also remains that the NAEP scores for Texas are high, so the essential point of the post, at least as to Collins, seems valid. Since Ivins was not an education policy analyst, was not writing a book about it, and the overwhelming thrust of her column was about who deserves credit for the enormous progress of Texas schools, with special mention of the disaggregated high scores of minority students, it is a lot less so as to Ivins. It was worth a corrective comment at best, not an attack implying she said a lot more than she did. Invoking her heroic status with progressives as a way to counter any defense of her has no value.

      Nor did I say nor did I imply that the data are "probably" unreliable. Lower statistical reliability is the inevitable byproduct of breaking a sample into a smaller sample. Lower reliability does not translate into unreliability, and I neither said nor implied that. The point was simply that it is worth examining, and being careful with going too far with language based on the simple assumption that the data shows the truth.

      I do know how measurable the lag is, and if you bothered to take the time you could, too. I did not claim to know how important it is. One thing it does suggest strongly, however, is that any "reforms" instituted by George Bush and successors beginning in 1995 (when Bush became Governor, a year before the 1996 tests) have not improved the schools much if at all. If anything, when combined with the opinion of the experts -- i.e., the teachers who actually teach children in the schools -- the lag behind the national rate of improvement suggests the possibility that the reforms might have actually truncated Texas' progress. But the lag does not apply to all categories. It's a complex question.

      Delete
  3. I've gotten a lot a pleasure over the years feeling smugly superior to Texas, knowing that everything about it was retarded and laughable. Then YOU come in, with your fancy facts and figures, and screw everything up! Goddamn it!

    ReplyDelete
  4. the daily howler

    musings on the arcane and my grudges

    ReplyDelete
  5. Despite the outpouring of praise for Ivins -- the mistake is worth noting and correcting.

    Why?

    Because it's being used in the present day, by Gail Collins, who quite undeservedly is getting a reputation for *knowing* something about education -- and Texas schools in particular -- being used in service of Collin's own bullsh!t.

    ReplyDelete
    Replies
    1. Then go after Collins! If she's taking one sentence out of 12 year old column and foolishly running with it, then blame her! But to cherry-pick one line you don't like out of an otherwise positive column about Texas schools is crazy. And to say that someone "implicitly praised the schools of Maine because they contained nothing but white kids" when they are dead and can't defend themselves is wrong. Period. The fact that Somerby can't figure that out is scary.

      Bob, you do good work, but you need an editor and a sounding board deseperately. Somebody who will let you know when you are too stubborn and have gone too far. So Bob, you are too stubborn and you've gone too far. Drop this stupid side-show and go back to the issues at hand.

      Delete
    2. "Then go after Collins!"

      Why, yes...

      Yes, I am too stupid to realize that this column is part of a series that is doing exactly that!

      I am too stupid to realize that a sidetrack into Ivin's error was probably necessary, because Collins is using it -- trying to ride the deep respect many have for Ivins -- in pursuit of her bullshit agenda.

      Yes, yes I am too deeply imbedded in my own love affair with the departed Ms. Ivins to allow any correction of her past mistakes.

      She's dead! How dare you correct her!

      The nerve!

      Yes, that's how stupid I am.

      Delete
    3. The Real "Real Other Commenter" GuySeptember 28, 2012 at 1:51 PM

      Put words in other people's mouths? Who, me?

      Delete
    4. “. . . the Rand study heaped praise on the Texas schools. But Rand had disaggregated the data.”

      “Misleadingly, Ivins did not.”

      “. . . Ivins’ preferred measure. . .”

      “Ivins pulled this statistic from a mammoth report”

      “Ivins dogged Texas for its rank on this single, ill-conceived measure. In essence, she was complaining that Texas had too many minority kids and too many kids with low incomes.”

      “Instead, she cherry-picked a single statistic. . .”

      It's a hell of a lot more than a sidetrack.

      Delete
    5. And it's all accurate.

      Look, urban legend, you've been dead wrong in virtually every thing you've said about NAEP during this series.

      You don't like seeing Ivins corrected? (Apparently in part because it's not fair to correct the dead?!?!)

      And, she was mostly right, anyway? Whatever!

      No one, no one at all is really talking about the mis-use of education data - other than Somerby. For that alone, it's a valuable series.

      Ivins was wrong on this point.

      You yourself, urban legend, have pretty clearly been taken to school on the whole issue of disaggregation. But still you grouse, over this little point: Leave poor Ivins alone!! Wahh!

      Delete
  6. Maу I sіmplу јust say what a cоmfort tо discοvеr somеbοdy that really knows whаt thеy
    are discusѕing on thе nеt. Yοu actually realize how to bring
    an issue to light and make it impоrtаnt.
    More peορle really neeԁ to checκ thiѕ out аnԁ undеrstand this ѕіde οf the storу.
    I was ѕurргised you're not more popular given that you surely have the gift.

    Have a look at my weblog ... http://fraythemovie.com/
    my web site :: samsung galaxy note 2

    ReplyDelete
  7. Hello, ӏ would likе to ѕubѕcribe for this weblog
    to taκe most up-to-datе updаtes, thеrefore
    wheгe can і do it please help out.
    Feel free to visit my site : purchase likes

    ReplyDelete
  8. An impressive share! I have just forwarded this onto a coworker who has been conducting a
    little homework on this. And he actually ordered me dinner because I
    stumbled upon it for him... lol. So allow me to reword this.
    ... Thanks for the meal!! But yeah, thanx for spending the time to discuss this subject here on your web page.



    Here is my blog: http://www.forexbroker.info
    Also visit my site : vapornine

    ReplyDelete
  9. Whoa! This blog looks just like my old one! It's on a entirely different topic but it has pretty much the same layout and design. Wonderful choice of colors!

    My webpage ... increase twitter followers

    ReplyDelete
  10. Everyone loves what you guys are up too.

    This sort of clever work and exposure! Keep up the very good works guys I've you guys to my personal blogroll.

    my web blog; sites to get more followers on twitter
    My website - targeted twitter followers

    ReplyDelete
  11. Thank you for the good writeup. It in fact was a amusement
    account it. Look advanced to more added agreeable from you!
    However, how could we communicate?

    Also visit my webpage :: get followers
    My website - get followers

    ReplyDelete
  12. I all the time emailed this website post page to all my associates, since if like to
    read it then my contacts will too.

    my web page :: get followers
    Also see my site: get followers

    ReplyDelete
  13. My brother suggested I might like this blog. He was totally right.
    This post truly made my day. You cann't imagine simply how much time I had spent for this information! Thanks!

    my web site; ava fx

    ReplyDelete
  14. What's up to every one, because I am in fact keen of reading this weblog's post
    to be updated daily. It carries good stuff.

    Here is my site ... get followers

    ReplyDelete
  15. Its such as you learn my mind! You appear to grasp a lot about this, like you wrote the e-book in it or something.

    I think that you simply could do with a few % to power the message home a bit, but instead of that, this is wonderful blog. A fantastic read. I will certainly be back.

    my site; investment trading

    ReplyDelete
  16. Simple, Texas needs to enforced e-verify and many of the 1.6 million illegal immigrants that have kids will return to Mexico, so the excuse of having a minority population will dropped by as much as 5 to 10 percent.

    ReplyDelete
  17. Maine (98)
    2. North Dakota (96)
    3. Iowa (93)
    4. New Hampshire (97)
    5. Montana (unavailable)

    [...]Molly's right, states need to get rid of their illegal population. Blacks are a different story.

    ReplyDelete