DISAGGREGATION NATION: No bogus analysis left behind!

FRIDAY, SEPTEMBER 28, 2012

Part 3—Gail Collins quotes Barbara Bush: If you want to judge how well a school or school district is doing, you pretty much have to “disaggregate” the relevant test scores.

If you just go by the overall scores, you will think that schools (or states) with lots of white kids are the best-performing schools (or states) in the nation. You will think that schools with lots of upper-income kids are performing better than schools with kids from low-income backgrounds.

You will think that the schools in Maine are performing better than those in Texas, even when “disaggregation” would show you results like these:
White students only, fourth-grade math, 1996 NAEP
Texas: First in the nation
Maine: Eleventh

White students only, fourth-grade reading, 1998 NAEP
Texas: Second in the nation
Maine: Twelfth

White students only, eighth-grade math, 2000 NAEP
Texas: Sixth in the nation
Maine: Eighteenth

White students only, fourth-grade math, 2000 NAEP
Texas: Second in the nation
Maine: Twenty-fifth
If you want to conduct a sane analysis, you have to disaggregate test scores! But alas! In her unfortunate book, As Texas Goes, Gail Collins praises the wonders of disaggregation, then utterly fails to employ the practice. This leads her to roll her eyes at the Texas schools—even though Texas students persistently outperform their peers around the nation on our most reliable tests, the National Assessment of Educational Progress (NAEP).

Collins has explicitly said that NAEP data are the best we have. (Everybody agrees, for perfectly obvious reasons.) That said, how well did Texas schools perform in 2011, the last year for which we have data?

Thanks for asking! In 2011, Texas kids outscored their peers around the nation in both reading and math on the NAEP. But their performance in math was especially strong—and this is a phenomenon which dates to the mid-1990s:
Texas students, rank among the fifty states, fourth-grade math, 2011 NAEP
White students: Sixth in the nation
Black students: Fourth in the nation
Hispanic students: Eleventh in the nation

Texas students, rank among the fifty states, eighth-grade math, 2011 NAEP
White students: Third in the nation
Black students: Second in the nation
Hispanic students: Second in the nation
One year later, Collins published her unfortunate book. Confronted with that type of success, Collins ginned up a different portrait.

Today, we include and highlight an unfortunate reference to 87-year-old Barbara Bush. That said, this entire passage is grotesquely misleading—an act of journalistic fraud:
COLLINS (page 91): You may be wondering how things are going, education-wise, in the state that deeded its reform plan to the nation. Paul Sadler, the Democrat who led the effort in the Texas house, complains that the state is “testing our kids to death.” (Under the state’s newest regimen, students take seventeen high-stakes tests between third and eighth grade, and up to a dozen more while they’re in high school.) A survey by the Texas State Teachers Association showed that 43 percent of its members were seriously thinking of looking for another line of work.

Sadler still believes that the leaps made in the 1990s are holding up. “I think most of our schools do a pretty good job,” he said. Other observers are, at a minimum, disillusioned. “The eighth grade reading scores were exactly the same in 2009 as 1998,” said Diane Ravitch, referring to the scores Texas received in the national NAEP test results. (I know we were trying to avoid them, but sometimes it's unavoidable.) “The whole country is now embarked on remedies that didn’t do anything for Texas.”

David Grissmer, the author of that glowing RAND study, says that since 2000, when the study came out, Texas students’ scores on national tests have begun to “flag.” Perhaps coincidentally, that was exactly the time when Bush stopped being governor and turned the state over to Perry, whose interest in K-12 education was minimal. When the state’s budget developed a monster hole in 2011, Perry refused to raise taxes—or even dip into state savings—to avoid enormous cuts in school aid. As the impact began to hit districts, schools began cutting back on programs that had been in place since the Perot commission, seeking waivers on class size and preschool requirements. Former first lady Barbara Bush wrote an opinion piece in the Houston Chronicle protesting the lack of financial support for public schools. “We rank 36th in the nation in high school graduation rate,” she wrote. “An estimated 3.8 million Texans do not have a high school diploma. We rank 49th in verbal SAT scores, 47th in literacy and 46th in average SAT scores.”

It all sounded sort of familiar.
“It all sounded sort of familiar,” Collins wrote—having offered a parody of journalism, in which she extended a very familiar, but grossly misleading, bogus old portrait of Texas.

The statement by Ravitch is inexcusable, for reasons we will review again next week. The one-word quotation from Grissmer is hard to parse: What exactly does it mean to say that scores “have begun to flag?”

But Collins tops even herself when she includes those statements by Barbara Bush. There’s no sign that the former first lady has any idea what she’s talking about. And her data are basically worthless, if we’re trying to assess the performance of the Texas schools.

A bit of background:

Barbara Bush wrote her unfortunate column in February 2011. (To read her full piece, click here.) As she wrote, the Texas legislature was considering large cuts to statewide education funding. At the time, education cuts which were being imposed in many states, one reaction to the funding problems brought on by the nation’s economic collapse.

Barbara Bush argued against the funding cuts. She may have been on the side of the angels, but her portrait of the Texas schools was uninformed and grossly misleading. Two obvious problems:

Graduation rate: Does Texas really rank 36th in high school graduation rate? It’s certainly possible! Later in her book, Collins approvingly reprints other data which say the state ranks 43rd! Rather typically, Collins doesn’t seem to notice that she has presented two different figures.

Does Texas rank 36th in high school graduation? It’s possible—but Texas will rank rather low on many measures until you disaggregate data. The Texas schools have high proportions of students from low-income families. They have very high proportions of black and Hispanic kids.

As part of our continuing national tragedy, minority kids drop out of school at substantially higher rates than white kids. All things being equal, states with large proportions of low-income and minority kids will have higher drop-out rates than states with lots of white kids.

Where would Texas rank in graduation rate if we compared its student groups to their peers nationwide? For example, where would Texas rank in graduation rate for black kids? Trust us—Barbara Bush has no idea! Neither does Collins, who effusively praises disaggregation, then fails to employ the practice.

SAT scores: Even worse is Barbara Bush’s use of those SAT scores. Collins has said that the NAEP is our best source of educational data. If we’re trying to compare the performance of schools in the various states, the SATs may be our worst.

The SATs are not designed for this sort of comparison. In some states, every high school student is required to take the SATs. In other states, the SATs are taken by a narrow range of top students.

Comparisons between the average scores in such states are completely meaningless. And here we go again! In the case of Texas, large numbers of its high school students are black and Hispanic—and minority kids still score substantially lower on the SATs than their white peers.

What would those SAT scores look like if we were able to disaggregate the data? For example, if we could compare a representative sample of black high school kids in Texas to their peers nationwide?

Barbara Bush has no idea—and neither does Collins. But so what! Those SAT scores look very bad, and that of course is why Collins used them. But it’s impossible to say what those scores really mean. Meanwhile, you can look at those data from the NAEP, in which carefully selected samples of students are compared from one state to the next.

Texas students score very high on that measure—on a measure which is specifically designed to permit such state-to-state comparisons. Result? Collins ignores the best data, grotesquely clowns with the worst.

The full passage we have quoted above is pure journalistic porn. As our series continues, we’ll look at other spots in Collins’ book where she builds a grossly misleading picture of the performance of the Texas schools.

We think her work is deeply cruel and deeply unfeeling. Here’s why:

For the past fifteen years, Texas schools have been performing extremely well on NAEP math tests. In 2011, the state’s reading scores were good, often quite good. The math scores were exceptional.

On the eighth-grade level, black kids in Texas scored ahead of their peers in every state but Hawaii. The state’s Hispanic kids outscored their counterparts in all states except Montana. (White kids finished third among the fifty states.)

A decent person would wonder what might account for this. What have the Texas schools been doing to produce those high math scores?

But people like Collins don’t care about that. They exist to mock the red-state rubes, thus entertaining their pseudo-liberal audience. They exist to keep telling us a “sort of familiar,” very old story which makes pseudo-liberals feel good.

They don’t care about an important question: Why are black kids in Texas scoring so high in math?

Gail Collins doesn’t care about that. Black kids in Texas can hang in the yard as far as she is concerned.

As our series continues, we will continue to examine Collins’ clowning approach to these topics. But how little does Collins care about this? Enjoy this window into the way this “journalist” gathers her facts:
COLLINS (page 197): Acknowledgments

Normally, this is my favorite part of writing a book—when you get to thank all the people who helped you along the way. In the case of this particular book, however, I got guidance, advice, support and good information from so many incredibly smart and helpful people it’s a little embarrassing.

The idea for As Texas Goes came from Bob Weil…who was also, to my incredible good fortune, my editor at every step along the road. Although it’s certainly true that without him the book would never have been written, I’d rather point out that without him the book would have been unreadable.

At the beginning of this project, Abby Livingstone of Roll Call did me the favor of recommending a fellow Texan, [name withheld], as a researcher. Before she went off to pursue an advanced degree in architecture at Harvard, [name withheld] got me through all the initial chapters, as well as the education section.
Trust us—at its core, this horrible book is “unreadable.” That said, we direct your attention to that third paragraph.

We’ve withheld the name of that young Texan because this book’s Texas mess isn’t her fault. But note the way the journalist Collins undertakes her research:

Collins’ book has three chapters on educational topics—and she had the “research” done by an architecture student! The young woman whose name we have withheld may well be “incredibly smart and helpful.” But the evidence suggests she may have known little about educational data.

She may not have understood disaggregation, even as Collins was praising the practice. She may not have known that you can’t use SAT scores in the way Barbara Bush did.

She didn’t know what was wrong with that disgraceful statement by Ravitch. We'll assume she wasn't involved in that one-word "quotation" of Grissmer.

Just a guess: To this day, Collins’ architecture student has never seen Texas NAEP scores after you disaggregate the data. When she was conducting her research for Collins, she didn’t know that Texas kids have been kicking the nation’s keister for years, especially in math.

Twelve years ago, Molly Ivins implicitly praised the schools of Maine because they contained nothing but white kids. Twelve years later, this time inexcusably, Lady Collins has done the same thing in an ugly but typical book.

She tells a very familiar old story, one designed to amuse and mislead.

Pseudo-liberals have played this game for years. Why is this conduct accepted?

Monday—part 4: Additional misleading data—and some additional questions

Why not conduct your own research: The NAEP web sites provide tons of data, none of which Collins seemed to review.

To conduct your own state-by-state comparisons, click here, then click again on "State Comparisons." You will be doing the type of work Collins seems too lazy to do.

23 comments:

  1. The Real Urban LegendSeptember 28, 2012 at 1:21 PM

    What stupid goddamn thing am I going to say today?

    The only certain is that I will arrive to offer an irrelevant half-baked criticism!

    ReplyDelete
    Replies
    1. You mean, just like Somerby does?

      Good grief, how many days and how many thousands of words has he already spent this week saying the same things about the same thing over and over and over and over and over . . .

      Delete
    2. Yeah, it's not as if how minority kids are actually doing matters...

      Delete
    3. Oh, I forgot. Somerby and his small band of sycophants are the only people on earth who care about minority kids.

      I really shouldn't have so much trouble remembering that. They keep saying is over and over and over and over . . .

      Delete
    4. "Oh, I forgot. Somerby and his small band of sycophants are the only people on earth who care about minority kids."

      >>> dissmissive missive uncharacteristically agreeing with a commenter,

      ...this helps maintain his/their apparent liberal status as they go about the business of false equivocating on behalf of the gop, hoping to pull his audience back at least somewhat from the left to the right.

      Delete
    5. "Good grief, how many days and how many thousands of words has he already spent this week saying the same things about the same thing over and over and over and over and over . . ."

      >>> dissmissive missive is happy to see a comment to be applauded and says,

      the repetitive meandering is a mind control technique akin to hypnosis. lord somerby has been honing his psi-op techniques on his many missions of national import.

      Delete
    6. lowercaseguys case managerSeptember 28, 2012 at 7:19 PM

      Shorter dismissive_missive (aka lowercase guy):

      I have no way of arguing coherently against Somerby's point today (or, yes, any other day!), so I instead argue (and applaud) the idea that he ought not to make his point at all!

      Anon: When you find yourself being seconded from Bellevue, you ought to reconsider.

      Delete
    7. extremely dismissive missiveSeptember 28, 2012 at 10:54 PM

      @ lowercaseguys case manager

      first, can i fire you?

      second, do i hear you right? old hit-and-run, a. d. hominem, is knocking *me* for a lack of substance? laugh.

      third, you might want to look up "projection", de riguer by lord somerby and his subjects. whats in your head?

      Delete
  2. I am still at a loss as to how Texas students can score so well on the NAEP, and so poorly on the SATs. I could understand it if they dropped, because of the reasons Bob mentions, from near the top to middle of the pack, but to drop from near the top to the bottom, that seems fishy. It really ought to be looked at, but people like to talk about education a lot more than they like to think about it.

    ReplyDelete
    Replies
    1. " like to talk about education a lot more than they like to think about it."

      Or, like you do here, raise a "fishy" red herring when you have no data about it.

      Whatever, right?

      "how Texas students can score so well on the NAEP, and so poorly on the SATs"

      Are you paying attention?

      TX students as-a-whole DON'T score so well on the NAEP!!! They're below average as-a-whole.

      So the first half of your conditional "how Texas students can score so well on the NAEP" is just wrong.

      Given that, the predicate "and so poorly on the SATs" is just irrelevant.


      So, again, Texas kids, taken as a whole (the same way your're taking the SAT data), are *not* scoring so well on the NAEP.

      When you know that, and when you know that as-a-whole TX kids' parents are less white and less-wealthy -- are you really still "at a loss" to understand how they could be doing poorly on the SATs?

      Because if you are still "at a loss," knowing those *facts* then you are willfully dumb.

      Delete
    2. One of the factors involved here is that the NAEP and the SAT look at two very different things. (Cautionary note: I make my living prepping students for standardized tests.) The NAEP attempts to measure academic achievement, while the SAT attempts to measure ability to apply logic and, in some cases, academic subject matter to solve problems. Saying that a state does well on the NAEP and poorly on the SAT is like saying that it does well on NAEP but poorly on Sudoku.

      Delete
    3. Well, Nick, doesn't that bring us to the whole purpose of education?

      Or to paraphrase the good book, what good is it to do well on the NAEP if you can't apply logic and solve problems?

      Delete
    4. I feel fairly certain there's a high correlation between good problem solving skills and academic achievement. It's just a suspicion, mind you, but, well, you get the idea.

      Delete
    5. God, you two (three?) are tools:

      "what good is it to do well on the NAEP if you can't apply logic and solve problems"

      What evidence is there that someone (or some state) is doing well on the NAEP but "can't apply logic and solve problems"?

      Answer: You have none. You are talking out your ass.

      "Saying that a state does well on the NAEP and poorly on the SAT is like..."

      What state are you even talking about?? TX? But we've already established that TX *didn't* do especially well on the NAEP!

      The sounds keep coming out of your asses. But they still aren't making sense.

      And *you* guys are worried about the problem solving skills of the *kids*!!

      Hilarious!!!!

      Delete
    6. What do you want to bet that these Anglo schools in Texas teach to the test?

      Much like what conservative districts do in Orange County Ca. Here they even have big assemblies where they'll highlight individual kids and their test scores, cheering them on to a higher score and always exhorting them to greater glory.

      It's like athletics-- and with this kind of attention to wholesale testing -- especially in the White schools-- you just know it's going on en masse.

      I mean, come on. It's Texas.

      And it's not good pedagogy.

      Delete
    7. "What do you want to bet that these Anglo schools in Texas teach to the test?

      Much like what conservative districts do in Orange County Ca. Here they even have big assemblies where they'll highlight individual kids and their test scores"

      On the NAEP?

      You're a liar. Or an idiot. Probably both.

      Delete
    8. At this late date, I'm sure no one (besides me) is following the conversation, but a couple of points. First, to the "tool" reference above, I wasn't in fact talking about ANY state. If I were talking about a state, I would have mentioned the state. What I was talking about was the SAT. If you're going to mock me for being a tool, Swan, at least mock what I actually said and not the point YOU think I was trying to make.

      As for the folks suggesting that SAT results are, indeed, a worthwhile component of this debate because logic and problem solving skills are an important part of learning, well, I partially agree. I do think problem solving skill is important. I just don't think the SAT actually measures that skill, at least not in any meaningful way. If it did, I wouldn't be able to teach students how to improve their scores without substantially improving their problem-solving skills.

      As for suggesting that the anonymous commenter above is a liar, an idiot, or both, well... It's possible (just possible) that, in fact, the commenter isn't talking about the NAEP. Many states have state-level mastery tests--you've heard of them, I'm sure--whose results are widely celebrated within school districts, and whose high scorers are often congratulated publicly. I'd suggest that you shouldn't be so quick to label someone so harshly without knowing for sure what they're talking about.

      Delete
    9. "without knowing for sure what they're talking about"

      If they weren't talking about the NAEP, they could have clearly said so -- and it would have made clear that they were raising an irrelevancy with regard to the topic at hand.

      But given that they began with the referent "the test" in their reply it would be reasonable to think they were on the same page with the rest of the conversation. The fact that the initial post was talking (albeit incorrectly) of good performance on the NAEP and poor performance on the SAT suggests this is the right interpretation of the "teaching to the test" comment -- they meant the NAEP [if not, then their idiocy is guaranteed].

      Nevertheless, Nick, I'll allow that you may be correct -- the commenter at 9/29 1:08 AM may NOT be an idiot or a liar: they may simply be a blowhard wasting time on irrelevancies.

      As for the original "Saying that a state does well on the NAEP and poorly on the SAT" -- what was the point of that line of argument?

      It's a straw argument.

      It was raised as a counter to Somerby, as a suggestion that it made something look "fishy."

      Well, it doesn't make anything look fishy, because we simply Don't Have Any Evidence of "a state does well on the NAEP and poorly on the SAT."

      TX doesn't do well on the SAT? Okay!

      But TX ALSO doesn't do well on the NAEP!

      Delete
    10. I'll try again, Swan... My intent in my original post was not to insert a straw man, but rather to point out that any time spent discussing the SAT in the context of academic performance is itself a straw man. I was not countering Bob, whom I consider to be consistently spot on in his analysis.

      Perhaps I could make myself more clear to you if I were to insert the clause "Even if we accepted the highly dubious claim that State X had high NAEP scores and low SAT scores," but I suspect you'd still miss my point. After all, you think I disagree with you; therefore, I must be an idiot, a liar, or a blowhard.

      Such is life.

      Delete
  3. Unless something has changed very recently, there are no states that require students to take the SAT. There are at least six states where 100% take the ACT.

    There is nothing inherently wrong with comparing state scores. Both the ACT and SAT list state scores. The first order of business is to look at the participation rate, since there is a very high correlation between scores and participation rate. Then you have to do the disaggregation for closer comparability, but I do not believe that data by state, which has to be based on test takers and not general student populations, is made available.

    It's still data, though, and no data is completely meaningless. No state will ever be immune to comparing the top-line scores, because disaggregation outside of policy analysis (1) is a bit complex for most of the population; (2) looks like an excuse even if it's not; and (3) is unpleasant for minority populations even when progress borders on spectacular.

    For the relatively low percentages who take either the SAT or the ACT in Texas, their Texas scores on first glance look a bit lower than one would expect even from the aggregated NAEP data. High schools worse than those in other states? Probably not, but who knows? No data can simply be ignored.

    Of course, yes, Barbara Bush doesn't know shit. As far as minorities are concerned, I'm betting she actually thinks the schools are "working out pretty well for them."

    ReplyDelete
  4. "Collins’ book has three chapters on educational topics—and she had the “research” done by an architecture student!"

    Right, Bob. We all know how incredibly complex educational data is. Only highly trained professionals should even begin to "research" it.

    And you know what? That could be part of the problem. Career educators are so taken by what they do that they have developed their own jargon, their own sets of numbers, and truly believe it can't be understood by us "common people" like Harvard archecture students.




    ReplyDelete