DISAGGREGATION NATION: As found in Collins’ appendix!

TUESDAY, OCTOBER 2, 2012

Epilogue—More deception and bile: For whatever reason, Texas schools and Texas students score well on the National Assessment of Educational Progress (NAEP)—if you disaggregate test scores, as everyone knows you should.

Texas schools have scored quite well on the NAEP for the past fifteen years. In math, they score near the top of the nation—after “disaggregation,” that is.

To show you what we mean, here are math scores from Texas and Massachusetts in last year’s NAEP. Massachusetts is commonly regarded as our highest-scoring state—our own little answer to Finland.

Compared to their peers in the other states, Massachusetts students do rank high in math. But omigod! So do students in Texas:

Texas and Massachusetts, 2011 NAEP
Grade 4 math, rank among the fifty states:
White kids: Massachusetts 1, Texas 6
Black kids: Massachusetts 2, Texas 4
Hispanic kids: Massachusetts 7, Texas 11

Grade 8 math, rank among the fifty states:
White kids: Massachusetts 1, Texas 3
Black kids: Texas 2, Massachusetts 3
Hispanic kids: Texas 2, Massachusetts 16

These states are both high-scorers in math. But so what? While Massachusetts gets praised as our bright shining star, Texas gets trashed by the likes of Gail Collins, in the insulting ways we included in yesterday’s post.

Why does Massachusetts attract so much praise? In part, because the state deserves it! Even after disaggregation, Bay State students score extremely well, as you can see in those data. (For reading scores, see below.)

And why does Texas get trashed by Collins? In part, because of what happens to some states’ test scores if you don’t disaggregate. To wit:

Texas has lots of black and Hispanic kids; Massachusetts doesn’t. For that reason, this is how the two states compare in eighth-grade math if you go by overall scores—if you don’t disaggregate:

Texas and Massachusetts, 2011 NAEP
Grade 8 math, rank among the fifty states:
All students: Massachusetts 1, Texas 10

Because its schools serve so many minority kids, Texas’ overall ranking slides all the way down to tenth in the nation, even though each of its main population groups ranked second or third on this test.

Yes, you read that correctly! White kids in Texas ranked third in the nation. Black and Hispanic kids each ranked second. But overall, Texas ended up ranked at tenth, because it has such a large proportion of minority kids!

Tenth place out of fifty is still a strong ranking, of course. Reading Collins’ misleading book, you would never dream that Texas kids ranked that high.

But tenth in the nation is much lower than the rank achieved by each of the state’s major student groups. Conclusion? You need to remember how this works when people like Collins fail to disaggregate test scores (and other types of data).

To some extent, Massachusetts gets praised because it has lots of white kids! Texas ends up getting handed the bile you see in Collins’ appendix.

Leave it to Collins! In the appendix to her slender book, she reprints a propagandistic report from the Texas Legislative Group, a group of Texas Democrats. She reprints their report in full—but not before offering this introduction:

COLLINS (page 231): Appendix

Texas on the Brink
A Report from the Texas Legislative Study on the State of our State

“Texas on the Brink” is a measure of how the state stands up to the rest of the country, published at the beginning of every two-year session by the Legislative Study Group in the Texas House of Representatives.

When this 2011 version was produced, Representative Garnet Coleman was chair of the group. [Names of more members follow.]

Just spending money isn’t a sign of progress. If your state came in at the bottom on student achievement, you probably wouldn’t be comforted by the news that it had paid the most per capita on its students. And Texas does have a low cost of living. So the fact that it ranks forty-fourth in per pupil expenditures might not mean anything at all—if the high school graduate rate didn’t clock in at forty-third.

[...]

Gail Collins

There she went again, conveying the impression that Texas “comes in at the bottom on student achievement!” More specifically, does Texas really rank 43rd in “high school graduate [sic] rate?” It’s certainly possible, but we’re not sure—and Collins doesn’t seem to know either. Earlier in this embarrassing book, she approvingly quoted Barbara Bush saying the state ranked 36th! (See THE DAILY HOWLER, 9/28/12.)

Without seeming to notice the contradiction, Collins bumped the state down seven more places in her appendix.

(Note: Graduation rates are famously difficult to compute. Under present circumstances, Texas will inevitably have a high overall drop-out rate, given its demographic mix. That said, Collins apparently failed to notice the change in the ranking she assigned.)

After this renewal of snark, Collins reprints “Texas on the Brink” as her formal appendix. When it deals with the Texas schools, the report is a barely coherent, largely cherry-picked mess.

It’s the kind of report an actual journalist would be expected to challenge! Collins simply reprints it in full, padding her slender tome.

What is wrong with the way this report presents the Texas schools? At several places, the report presents data straight from the NAEP. But it cherry-picks the data it uses, and it often presents those data in ways which are barely coherent.

To wit:

Starting on page 239 of Collins’ book, you will see a page of data which make it sound like Texas students have low “proficiency rates” on the NAEP—rates which fall below national norms. At no point does the report disaggregate the data. Had this been done, those proficiency rates would of course have exceeded national norms.

All the selections in this passage are designed to convey the impression that Texas schools are low-performers. Often the writing is barely coherent. This bullet-point is pure statistical porn:

TEXAS LEGISLATIVE STUDY: 79 percent of 4th graders in families with low incomes were at a basic performance level on math in comparison to 95 percent of whites.

Presumably, that is meant to show that white kids in Texas are doing better than kids from low-income families. But that would be true in every state, including states which had nothing but white kids!

Meanwhile, a large percentage of white kids in Texas come from those very same low-income families! This is incoherent, incompetent work. It exists for one reason—to convey an unflattering impression about the performance of Texas schools among low-income and minority children.

In fact, Texas scores high among low-income and minority kids as compared to the other states. But a reader is never told any such thing at any point in Collins’ book, or in the highly propagandistic “Texas on the Brink.”

At a second point, “Texas on the Brink.” presents data from the NAEP concerning achievement gaps. This starts on page 240 in Collins’ book.

The presentation is grossly misleading.

The report simply presents the gaps observed on the 2009 NAEP. For fourth and eighth grade reading and math, the report shows how much higher white students in Texas scored as compared to black and Hispanic students.

Those achievement gaps do exist. In eighth-grade math, to cite one example, the gap between white and Hispanic students in Texas was 24 points on the NAEP scale.

It would be a better world if those gaps didn’t exist. It’s important to remind citizens that these gaps exist (although they’ve been getting smaller). But this report presents the gaps without noting a very key point—in Texas, that achievement gap is occurring at a very high level as compared to most other states.

Example:

Alabama has a smaller achievement gap in eighth grade math. But Alabama's white and Hispanic kids are both scoring roughly twenty points below their peers in Texas! Meanwhile, Massachusetts has a larger gap than Texas—the second largest gap in the nation. But its gap is occurring at a very high level, with both its white and its Hispanic kids scoring above the national average.

It’s important to document those gaps. But achievement gaps aren’t the only measure of a state’s performance—unless you’re doing the kind of work found all through this horrible book.

When it comes to the public schools, “Texas on the Brink” cherry-picks its data furiously. As such, it fit the unholy template of this horrible, unholy book.

People who read Gail Collins’ book are grossly misled about Texas schools. That means that they’re also grossly misled about the beautiful children and hard-working teachers found within those schools.

Those children and teachers seem to be doing something right, especially in math, as compared to their national peers. A decent person might want to try to figure out what that might be.

Collins isn’t that person. She isn’t even decent enough to tell you that Texas schools are high achievers, after you disaggregate test scores. In fairness, we’ll guess that she didn’t know that. If she knew, she wasn’t telling.

Manifestly, Gail Collins doesn’t seem to care about Texas kids. Collins only seems to loves the thrust of her dumb, ugly novel.

Tomorrow: Ruminations on a very old novel

Massachusetts and Texas in reading: As shown above, Texas schools score especially well in math. On the 2011 NAEP, its three major student groups scored near the top of the nation, as did students in widely-heralded Massachusetts.

After you disaggregate test scores, Texas kids run neck-and-neck with their Bay State peers! Texas doesn’t do quite as well in reading, although its students still rank high among the fifty states:

Texas and Massachusetts, 2011 NAEP
Grade 4 reading, rank among the fifty states:
White kids: Massachusetts 1, Texas 10
Black kids: Massachusetts 3, Texas 7
Hispanic kids: Massachusetts 7, Texas 14

Grade 8 reading, rank among the fifty states:
White kids: Massachusetts 3, Texas 13
Black kids: Massachusetts 7, Texas 10
Hispanic kids: Texas 23, Massachusetts 39

As compared to their peers, Texas students score high in reading, very high in math. No reader would ever dream such a thing from reading Collins’ book.