DISAGGREGATION NATION: No bogus analysis left behind!

FRIDAY, SEPTEMBER 28, 2012

Part 3—Gail Collins quotes Barbara Bush: If you want to judge how well a school or school district is doing, you pretty much have to “disaggregate” the relevant test scores.

If you just go by the overall scores, you will think that schools (or states) with lots of white kids are the best-performing schools (or states) in the nation. You will think that schools with lots of upper-income kids are performing better than schools with kids from low-income backgrounds.

You will think that the schools in Maine are performing better than those in Texas, even when “disaggregation” would show you results like these:

White students only, fourth-grade math, 1996 NAEP
Texas: First in the nation
Maine: Eleventh

White students only, fourth-grade reading, 1998 NAEP
Texas: Second in the nation
Maine: Twelfth

White students only, eighth-grade math, 2000 NAEP
Texas: Sixth in the nation
Maine: Eighteenth

White students only, fourth-grade math, 2000 NAEP
Texas: Second in the nation
Maine: Twenty-fifth

If you want to conduct a sane analysis, you have to disaggregate test scores! But alas! In her unfortunate book, As Texas Goes, Gail Collins praises the wonders of disaggregation, then utterly fails to employ the practice. This leads her to roll her eyes at the Texas schools—even though Texas students persistently outperform their peers around the nation on our most reliable tests, the National Assessment of Educational Progress (NAEP).

Collins has explicitly said that NAEP data are the best we have. (Everybody agrees, for perfectly obvious reasons.) That said, how well did Texas schools perform in 2011, the last year for which we have data?

Thanks for asking! In 2011, Texas kids outscored their peers around the nation in both reading and math on the NAEP. But their performance in math was especially strong—and this is a phenomenon which dates to the mid-1990s:

Texas students, rank among the fifty states, fourth-grade math, 2011 NAEP
White students: Sixth in the nation
Black students: Fourth in the nation
Hispanic students: Eleventh in the nation

Texas students, rank among the fifty states, eighth-grade math, 2011 NAEP
White students: Third in the nation
Black students: Second in the nation
Hispanic students: Second in the nation

One year later, Collins published her unfortunate book. Confronted with that type of success, Collins ginned up a different portrait.

Today, we include and highlight an unfortunate reference to 87-year-old Barbara Bush. That said, this entire passage is grotesquely misleading—an act of journalistic fraud:

COLLINS (page 91): You may be wondering how things are going, education-wise, in the state that deeded its reform plan to the nation. Paul Sadler, the Democrat who led the effort in the Texas house, complains that the state is “testing our kids to death.” (Under the state’s newest regimen, students take seventeen high-stakes tests between third and eighth grade, and up to a dozen more while they’re in high school.) A survey by the Texas State Teachers Association showed that 43 percent of its members were seriously thinking of looking for another line of work.

Sadler still believes that the leaps made in the 1990s are holding up. “I think most of our schools do a pretty good job,” he said. Other observers are, at a minimum, disillusioned. “The eighth grade reading scores were exactly the same in 2009 as 1998,” said Diane Ravitch, referring to the scores Texas received in the national NAEP test results. (I know we were trying to avoid them, but sometimes it's unavoidable.) “The whole country is now embarked on remedies that didn’t do anything for Texas.”

David Grissmer, the author of that glowing RAND study, says that since 2000, when the study came out, Texas students’ scores on national tests have begun to “flag.” Perhaps coincidentally, that was exactly the time when Bush stopped being governor and turned the state over to Perry, whose interest in K-12 education was minimal. When the state’s budget developed a monster hole in 2011, Perry refused to raise taxes—or even dip into state savings—to avoid enormous cuts in school aid. As the impact began to hit districts, schools began cutting back on programs that had been in place since the Perot commission, seeking waivers on class size and preschool requirements. Former first lady Barbara Bush wrote an opinion piece in the Houston Chronicle protesting the lack of financial support for public schools. “We rank 36th in the nation in high school graduation rate,” she wrote. “An estimated 3.8 million Texans do not have a high school diploma. We rank 49th in verbal SAT scores, 47th in literacy and 46th in average SAT scores.”

It all sounded sort of familiar.

“It all sounded sort of familiar,” Collins wrote—having offered a parody of journalism, in which she extended a very familiar, but grossly misleading, bogus old portrait of Texas.

The statement by Ravitch is inexcusable, for reasons we will review again next week. The one-word quotation from Grissmer is hard to parse: What exactly does it mean to say that scores “have begun to flag?”

But Collins tops even herself when she includes those statements by Barbara Bush. There’s no sign that the former first lady has any idea what she’s talking about. And her data are basically worthless, if we’re trying to assess the performance of the Texas schools.

A bit of background:

Barbara Bush wrote her unfortunate column in February 2011. (To read her full piece, click here.) As she wrote, the Texas legislature was considering large cuts to statewide education funding. At the time, education cuts which were being imposed in many states, one reaction to the funding problems brought on by the nation’s economic collapse.

Barbara Bush argued against the funding cuts. She may have been on the side of the angels, but her portrait of the Texas schools was uninformed and grossly misleading. Two obvious problems:

Graduation rate: Does Texas really rank 36th in high school graduation rate? It’s certainly possible! Later in her book, Collins approvingly reprints other data which say the state ranks 43rd! Rather typically, Collins doesn’t seem to notice that she has presented two different figures.

Does Texas rank 36th in high school graduation? It’s possible—but Texas will rank rather low on many measures until you disaggregate data. The Texas schools have high proportions of students from low-income families. They have very high proportions of black and Hispanic kids.

As part of our continuing national tragedy, minority kids drop out of school at substantially higher rates than white kids. All things being equal, states with large proportions of low-income and minority kids will have higher drop-out rates than states with lots of white kids.

Where would Texas rank in graduation rate if we compared its student groups to their peers nationwide? For example, where would Texas rank in graduation rate for black kids? Trust us—Barbara Bush has no idea! Neither does Collins, who effusively praises disaggregation, then fails to employ the practice.

SAT scores: Even worse is Barbara Bush’s use of those SAT scores. Collins has said that the NAEP is our best source of educational data. If we’re trying to compare the performance of schools in the various states, the SATs may be our worst.

The SATs are not designed for this sort of comparison. In some states, every high school student is required to take the SATs. In other states, the SATs are taken by a narrow range of top students.

Comparisons between the average scores in such states are completely meaningless. And here we go again! In the case of Texas, large numbers of its high school students are black and Hispanic—and minority kids still score substantially lower on the SATs than their white peers.

What would those SAT scores look like if we were able to disaggregate the data? For example, if we could compare a representative sample of black high school kids in Texas to their peers nationwide?

Barbara Bush has no idea—and neither does Collins. But so what! Those SAT scores look very bad, and that of course is why Collins used them. But it’s impossible to say what those scores really mean. Meanwhile, you can look at those data from the NAEP, in which carefully selected samples of students are compared from one state to the next.

Texas students score very high on that measure—on a measure which is specifically designed to permit such state-to-state comparisons. Result? Collins ignores the best data, grotesquely clowns with the worst.

The full passage we have quoted above is pure journalistic porn. As our series continues, we’ll look at other spots in Collins’ book where she builds a grossly misleading picture of the performance of the Texas schools.

We think her work is deeply cruel and deeply unfeeling. Here’s why:

For the past fifteen years, Texas schools have been performing extremely well on NAEP math tests. In 2011, the state’s reading scores were good, often quite good. The math scores were exceptional.

On the eighth-grade level, black kids in Texas scored ahead of their peers in every state but Hawaii. The state’s Hispanic kids outscored their counterparts in all states except Montana. (White kids finished third among the fifty states.)

A decent person would wonder what might account for this. What have the Texas schools been doing to produce those high math scores?

But people like Collins don’t care about that. They exist to mock the red-state rubes, thus entertaining their pseudo-liberal audience. They exist to keep telling us a “sort of familiar,” very old story which makes pseudo-liberals feel good.

They don’t care about an important question: Why are black kids in Texas scoring so high in math?

Gail Collins doesn’t care about that. Black kids in Texas can hang in the yard as far as she is concerned.

As our series continues, we will continue to examine Collins’ clowning approach to these topics. But how little does Collins care about this? Enjoy this window into the way this “journalist” gathers her facts:

COLLINS (page 197): Acknowledgments

Normally, this is my favorite part of writing a book—when you get to thank all the people who helped you along the way. In the case of this particular book, however, I got guidance, advice, support and good information from so many incredibly smart and helpful people it’s a little embarrassing.

The idea for As Texas Goes came from Bob Weil…who was also, to my incredible good fortune, my editor at every step along the road. Although it’s certainly true that without him the book would never have been written, I’d rather point out that without him the book would have been unreadable.

At the beginning of this project, Abby Livingstone of Roll Call did me the favor of recommending a fellow Texan, [name withheld], as a researcher. Before she went off to pursue an advanced degree in architecture at Harvard, [name withheld] got me through all the initial chapters, as well as the education section.

Trust us—at its core, this horrible book is “unreadable.” That said, we direct your attention to that third paragraph.

We’ve withheld the name of that young Texan because this book’s Texas mess isn’t her fault. But note the way the journalist Collins undertakes her research:

Collins’ book has three chapters on educational topics—and she had the “research” done by an architecture student! The young woman whose name we have withheld may well be “incredibly smart and helpful.” But the evidence suggests she may have known little about educational data.

She may not have understood disaggregation, even as Collins was praising the practice. She may not have known that you can’t use SAT scores in the way Barbara Bush did.

She didn’t know what was wrong with that disgraceful statement by Ravitch. We'll assume she wasn't involved in that one-word "quotation" of Grissmer.

Just a guess: To this day, Collins’ architecture student has never seen Texas NAEP scores after you disaggregate the data. When she was conducting her research for Collins, she didn’t know that Texas kids have been kicking the nation’s keister for years, especially in math.

Twelve years ago, Molly Ivins implicitly praised the schools of Maine because they contained nothing but white kids. Twelve years later, this time inexcusably, Lady Collins has done the same thing in an ugly but typical book.

She tells a very familiar old story, one designed to amuse and mislead.

Pseudo-liberals have played this game for years. Why is this conduct accepted?

Monday—part 4: Additional misleading data—and some additional questions

Why not conduct your own research: The NAEP web sites provide tons of data, none of which Collins seemed to review.

To conduct your own state-by-state comparisons, click here, then click again on "State Comparisons." You will be doing the type of work Collins seems too lazy to do.