The remarkable problem of basic test scores!

THURSDAY, JANUARY 9, 2014

Is San Diego the best: Is San Diego the nation’s best urban school system?

We have no opinion about that. We don’t know if there is one single best urban district.

Diane Ravitch says San Diego is best; she may be right, of course. On Tuesday, she started a post on the subject like this:
RAVITCH (1/7/14): Something magical is happening in San Diego. It is a good school district. Teachers and administrators and the school board are working towards common goals.

San Diego, in my view, is the best urban district in the nation.

I say this not based on test scores but on the climate for teaching and learning that I have observed in San Diego.
Ravitch goes on to offer a set of observations concerning the San Diego schools. Eventually, she complains about the way the local paper, the San Diego Times Union, has criticized the way the system is being run.

The Times Union made invidious comparisons with the school system in Houston, where San Diego’s previous superintendent had gone. Ravitch challenged the newspaper’s view:
RAVITCH: There was pushback. One board member wrote a letter to the editor pointing out that the dropout rate in Houston was nearly double the dropout rate in San Diego and commending Cindy Marten for avoiding the polarizing tactics associated with certain other unnamed superintendents.

But whoa! There are also some basic facts that the Union Tribune should have noticed. On the 2013 NAEP, San Diego’s public schools outperform those of Houston in math and reading, in grades 4 and 8. San Diego is in the top tier of urban districts; Houston is not. San Diego’s scores on the NAEP have steadily improved over the past decade. The proportion of students who score “below basic” has dropped significantly, and the proportion who score at or above proficient has increased significantly over the past decade. Why does the UT envy a lower-performing district and dismiss the solid, steady, persistent gains of its own district?
Gack! If you simply disaggregate the two systems’ test scores, Ravitch’s claims are wrong. Here you see the two systems’ average scores in Grade 8 math:
Grade 8 math, 2013 NAEP,
Houston versus San Diego

Black students: 271-260
White students: 312-300
Hispanic students: 279-260
In each case, the higher score belongs to Houston. In Grade 8 math, the scores weren’t really that close.

In other areas, the average scores after disaggregation tended to be close, with Houston tending to score higher. San Diego’s aggregate scores tend to be higher because it has quite a few more white and Asian students.

“San Diego’s scores on the NAEP have steadily improved over the past decade?” That is true, but so have Houston’s, from a higher starting point. (Except in Grade 8 math, San Diego has shown larger score gains over the past ten years.)

We aren’t trying to pick and choose between these two school systems. We hope Ravitch is right about the changes she sees occurring in San Diego. We hope the kids in Houston are doing better too.

But what is a person to think about those claims about test scores? All through her new book, Reign of Error, Ravitch disaggregates scores, especially when this basic practice helps her advance her key points.

In this case, she didn’t disaggregate. Apparently working from aggregate scores, she made a claim about magical San Diego which is just basically wrong.

This is a bone-simple statistical matter. At present, Ravitch is the liberal world’s most prominent “educational expert.”

But so what? Our country is still the Wild, Wild West when it comes to statistical claims about test scores. Four examples:

*In her ballyhooed pro-reform book, The Smartest Kids in the World, Amanda Ripley does quite a bit of picking and choosing, completely ignoring one whole set of international scores—the ones which don’t help advance her gloomy thesis.

*In adopting a decade-old script about miraculous Finland, the entire world of educational experts ignored a deflating fact—white students in this country score about as well as Finland on the PISA, and better than Finland on the TIMSS. Seeing this, why should we feel that Finland is working miracles?

*Mainstream journalists seem to have no earthly idea about our most reliable test scores. As Ravitch explains in Reign of Error, scores are substantially up for all demographic groups over the past two decades (and more). But so what? The press corps keeps saying, in the words of Bill Keller, that we have experienced “decades of embarrassing decline in K-12 education.” Their claims are persistently bogus, but they just keep churning them out.

*In this instance, Ravitch failed to disaggregate, something she does all through her book, especially when it advances her thesis. There’s much to be said for Ravitch’s book, but she does tend to play games with basic facts this way.

When it comes to the basics of test scores, we seem to live in the Wild, Wild West. Our leading journalists and experts seem to feel they can pretty much do as they please.

Why do these people behave that way? If our experts and journalists function this way, can we complain about the skills of our 10-year-old students?

A commenter corrects: In comments, a commenter named Tim supplied the fuller scores for Houston and San Diego. In our view, Ravitch’s replies are worth perusing.

13 comments:

  1. Diane Ravitch didn't do her credibility any favors in that exchange with commenter Tim.

    ReplyDelete
    Replies
    1. Tim was remarkably precise and Ravitch was careless and simply evaded Tim's important criticism.

      LTR

      Delete
  2. Diane Ravitch answered this way:

    Bob Somerby,

    I will repeat what I said to the previous similar comment. San Diego’s scores on NAEP have steadily improved. But, I did not say San Diego was best because of its test scores; other districts have higher scores.

    I said it was best because of the spirit of collaboration and respect that prevails there. It is a good district in which to be a student, a parent, a teacher, and an administrator. I think we have to get past the idea that the best district has the best test scores; if the district spends vast mounts of money on test prep and drops the arts and social workers and closes libraries, just so it can get higher scores, it’s not a good district. That’s my view. Disagree if you choose. It is a free country.

    ================
    I will add that my sister lives in San Diego, and my three nephews have gone or are going through the public school system. She is very happy and has nothing but praise for it. She herself, attended Hunter, a selective public High School in NYC, and got a very good education there and now works as an engineer.

    ReplyDelete
    Replies
    1. This was a nonsubstantive reply. As other commenters pointed out, Ravitch herself introduced the statistics. If she is going to do that, she needs to be accurate about it. Her response to Somerby is beside the point.

      Delete
  3. Glad to see the blogger is still reading and recommending comments on blogs other than his own.

    ReplyDelete
    Replies
    1. And writing comments as well I see. He left that out.

      Delete
  4. Sadly, the current trend seems to be to deny errors that are pointed out. I'm familiar with two cases where statistical errors in journal articles were formally pointed out -- one by my wife, the other by a friend. In both cases, the original authors gave a bullshit explanation denying the error.

    The world would work better if the person whose error was pointed out simply admitted their mistake.

    ReplyDelete
    Replies
    1. DAinCA,

      Stop trying to break my irony meter. I have a mil-spec, built-in-the-USA model.

      By the way, your last sentence has a number disagreement.

      Delete
  5. I gotta say I'm with Diane on this one. Although she quoted NAEP scores, she affirms several times that the scores are not the basis for her conclusion that San Diego is a superior school district. I completely agree with her statement: "...if the district spends vast mounts of money on test prep and drops the arts and social workers and closes libraries, just so it can get higher scores, it’s not a good district."

    ReplyDelete
  6. DH is playing "gotcha" with Ravitch. Her blog post was not about test scores, though she mentioned them. She is not obfuscating or distorting figures, much less denying them or their importance. But other things are also very important. DH and the other posters here seem to think test scores trump everything. Yet he continuously points out that test score have been rising *on their own* for many years (albeit slowly).

    ReplyDelete
    Replies
    1. True if and only if "trump everything" equals "shouldn't be misrepresented."

      Ravitch's presentation of stats was misleading. Her reply conveniently doesn't address that.

      Delete
  7. Over Christmas, I made a huge effort to find the actual black poverty rates for school districts in NAEP TUDA, as opposed to FRL. You might find it interesting:

    Finding the black poverty rates: http://educationrealist.wordpress.com/2013/12/31/naep-tuda-scores-detroit-isnt-boston/

    Does black poverty rate affect NAEP scores? http://educationrealist.wordpress.com/2013/12/31/naep-tuda-does-black-poverty-matter/

    ReplyDelete