The remarkable problem of basic test scores!

THURSDAY, JANUARY 9, 2014

Is San Diego the best: Is San Diego the nation’s best urban school system?

We have no opinion about that. We don’t know if there is one single best urban district.

Diane Ravitch says San Diego is best; she may be right, of course. On Tuesday, she started a post on the subject like this:

RAVITCH (1/7/14): Something magical is happening in San Diego. It is a good school district. Teachers and administrators and the school board are working towards common goals.

San Diego, in my view, is the best urban district in the nation.

I say this not based on test scores but on the climate for teaching and learning that I have observed in San Diego.

Ravitch goes on to offer a set of observations concerning the San Diego schools. Eventually, she complains about the way the local paper, the San Diego Times Union, has criticized the way the system is being run.

The Times Union made invidious comparisons with the school system in Houston, where San Diego’s previous superintendent had gone. Ravitch challenged the newspaper’s view:

RAVITCH: There was pushback. One board member wrote a letter to the editor pointing out that the dropout rate in Houston was nearly double the dropout rate in San Diego and commending Cindy Marten for avoiding the polarizing tactics associated with certain other unnamed superintendents.

But whoa! There are also some basic facts that the Union Tribune should have noticed. On the 2013 NAEP, San Diego’s public schools outperform those of Houston in math and reading, in grades 4 and 8. San Diego is in the top tier of urban districts; Houston is not. San Diego’s scores on the NAEP have steadily improved over the past decade. The proportion of students who score “below basic” has dropped significantly, and the proportion who score at or above proficient has increased significantly over the past decade. Why does the UT envy a lower-performing district and dismiss the solid, steady, persistent gains of its own district?

Gack! If you simply disaggregate the two systems’ test scores, Ravitch’s claims are wrong. Here you see the two systems’ average scores in Grade 8 math:

Grade 8 math, 2013 NAEP,
Houston versus San Diego
Black students: 271-260
White students: 312-300
Hispanic students: 279-260

In each case, the higher score belongs to Houston. In Grade 8 math, the scores weren’t really that close.

In other areas, the average scores after disaggregation tended to be close, with Houston tending to score higher. San Diego’s aggregate scores tend to be higher because it has quite a few more white and Asian students.

“San Diego’s scores on the NAEP have steadily improved over the past decade?” That is true, but so have Houston’s, from a higher starting point. (Except in Grade 8 math, San Diego has shown larger score gains over the past ten years.)

We aren’t trying to pick and choose between these two school systems. We hope Ravitch is right about the changes she sees occurring in San Diego. We hope the kids in Houston are doing better too.

But what is a person to think about those claims about test scores? All through her new book, Reign of Error, Ravitch disaggregates scores, especially when this basic practice helps her advance her key points.

In this case, she didn’t disaggregate. Apparently working from aggregate scores, she made a claim about magical San Diego which is just basically wrong.

This is a bone-simple statistical matter. At present, Ravitch is the liberal world’s most prominent “educational expert.”

But so what? Our country is still the Wild, Wild West when it comes to statistical claims about test scores. Four examples:

*In her ballyhooed pro-reform book, The Smartest Kids in the World, Amanda Ripley does quite a bit of picking and choosing, completely ignoring one whole set of international scores—the ones which don’t help advance her gloomy thesis.

*In adopting a decade-old script about miraculous Finland, the entire world of educational experts ignored a deflating fact—white students in this country score about as well as Finland on the PISA, and better than Finland on the TIMSS. Seeing this, why should we feel that Finland is working miracles?

*Mainstream journalists seem to have no earthly idea about our most reliable test scores. As Ravitch explains in Reign of Error, scores are substantially up for all demographic groups over the past two decades (and more). But so what? The press corps keeps saying, in the words of Bill Keller, that we have experienced “decades of embarrassing decline in K-12 education.” Their claims are persistently bogus, but they just keep churning them out.

*In this instance, Ravitch failed to disaggregate, something she does all through her book, especially when it advances her thesis. There’s much to be said for Ravitch’s book, but she does tend to play games with basic facts this way.

When it comes to the basics of test scores, we seem to live in the Wild, Wild West. Our leading journalists and experts seem to feel they can pretty much do as they please.

Why do these people behave that way? If our experts and journalists function this way, can we complain about the skills of our 10-year-old students?

A commenter corrects: In comments, a commenter named Tim supplied the fuller scores for Houston and San Diego. In our view, Ravitch’s replies are worth perusing.