THE SEARCH: We sought the truth about one school's scores!

WEDNESDAY, MAY 31, 2023

The history of that search: In February 2006, one of our favorite journalists—a person we flatly admire—made a substantial mistake. 

As you know, we human beings sometimes make mistakes. In this case, the mistake was bannered across the top of the front page of the Washington Post, where a banner headline said this:

A Study in Pride, Progress

The front-page report told a (very) familiar story about a (nearly miraculous) elementary school in Alexandria, Virginia.

The front-page report told a highly novelized story about the progress and pride now on display at one low-income school. Under the lead of a new, "energetic principal," the school had shown remarkable one-year growth in its scores on Virginia's statewide tests. 

Unfortunately, a mistake lay at the heart of that front-page report. A few days later, the Washington Post editorial board got taken in by the mistake:

WASHINGTON POST EDITORIAL (2/5/06): A profile of once-disastrous, now-successful Maury Elementary School in Alexandria by The Post's Jay Mathews last week showed what can be achieved if teachers and administrators use the law well. It's an odd idea, getting the Democrats to embrace a Republican project. But if they are brave enough to do it, thousands of inner-city children will be better off.

As you can see, the editors were thrilled by the large score gains at the "once-disastrous, now-successful school." 

As you can see, the editors attributed the score gains to President George W. Bush's No Child Left Behind law. According to the editors, if other teachers would employ the law the way the staff at this one school had, "thousands of inner-city children will be better off."

Unfortunately, this upbeat editorial was based on a large mistake. Everything was not as it seemed with this school's improved test scores.

In yesterday morning's report, we reminded you of the way impressive score gains can sometimes result from outright cheating—from deliberate, fraudulent conduct by the staff of a school or school system. 

We first became aware of this problem over dinner one night, with three other Baltimore City teachers, in 1971 or maybe 1972. Finally, almost forty years later, USA Today and the Atlanta Journal-Constitution researched this phenomenon to devastating effect. 

For one brief shining moment, that reporting—and the subsequent criminal trials in Georgia—created a world in which mainstream journalists were willing to acknowledge the existence of this long-running phenomenon. 

That said, we humans love the familiar, highly novelized tale of The Little Low-Income Elementary School That Could. Sometimes, major journalists may love that novelized story so much that they fail to check the small print in the data sets which accompany the upbeat report on the miraculous scores at the school. 

Did the AP fail to check the small print concerning the "Mississippi miracle," the topic it explored in this lengthy, upbeat report back on May 17? 

We're going to examine that question before our current series is done. But for today, we thought we'd review the remarkable error which led to that front-page report in the Washington Post—that front-page report about a small "inner city" elementary school which had previously been disastrous but was now successful. 

As far as we know, there was no "cheating"—none at all—on the part of the Maury staff. As far as we know, that school's apparent test score gains did not result from fraudulent conduct on the part of Maury's teachers or its "energetic" new principal.

Instead, the test score gains. which turned out to be bogus, had resulted from a bizarre policy move on the part of the state of Virginia's Department of Education. That said, please understand this:

There are different ways in which apparent test score gains can turn out to be bogus. Flagrant cheating can produce such gains, but so can policy decisions made at a higher level.

Briefly, this is the remarkable story concerning what happened at Maury:

For starters, our search began with this. Here at THE HOWLER, we didn't believe that front-page report about those heartening score gains. 

We didn't necessarily disbelieve that news report. But through long experience, we knew that upbeat news reports of that type often turn out to be bogus.

In the case of that front-page report, our search went on for several months. By the time we were done, everyone from the chairman of the Virginia Board of Education on down had agreed that a deeply ridiculous policy provision had led to the misimpression conveyed in that front-page report.

What actually happened at Maury Elementary? Crazy as it's going to seem, the story turned out to be this:

At that time, the state of Virginia was testing Grades 3, 5 and 8 in its statewide testing program. At some point, the state board had come up with a truly crazy idea—a crazy idea which would vastly inflate the official passing rates at large numbers of schools on the statewide tests.

Crazy as it seems—and no, we really aren't making this up—the official procedure was this:

Kids would take the statewide reading and math tests when they were in Grade 3. If they failed to achieve a passing grade, they would be promoted to Grade 4—and at the end of their Grade 4 year, those kids would take the Grade 3 test again!

Depending on how this was reported, this could have made perfect sense. But what follows is the way it was actually reported—and no, we aren't making this up:

Consider Maury Elementary, a small, low-income school. In the school year under review, there were only 19 third graders in the school. 

Only five of them passed the state's reading test. This created a dismal passing rate of 26.3 percent.

Statewide, the passing rate had been 77 percent! With that in mind, how did a school with a 26.3 percent passing rate quality as "now successful?"

The answer to your question is this, and no, we aren't making this up:

At Maury that year, there were a bunch of kids in the fourth grade who had failed the Grade 3 reading test the year before. In accord with official state policy, those kids took the Grade 3 test again at the end of the year—and 12 of those kids now attained a passing grade on the Grade 3 test.

So far, this still could have made a type of sense. But here's what happened next:

In accord with official state policy, the twelve fourth graders who passed the test were lumped in with the five third graders who passed it. This meant that seventeen kids had passed the Grade 3 reading test, in a school with only 19 third graders!

By now, you'll surely think we're kidding. Sadly enough, we aren't! In accord with official state procedure. the state reported that 17 kids had passed the Grade 3 test at Maury—17 kids, in a school with only 19 third graders. 

On that basis, the state reported that Maury Elementary had an 89% passing rate on the Grade 3 reading test! That was the official report, despite the fact that only five of the school's 19 third graders had actually passed the test.

We know, we know, we know, we know—it sounds we're making that up. It's hard to believe that something so crazy could have been happening as a matter of official state policy—but that is the way it was working in Virginia at the point in time.

By the time we spoke with the chairman of the state board, everyone had come to agree that this had been a crazy procedure which shouldn't have been adopted. But on the basis of that crazy procedure, the state had reported that Maury had a 89% passing rate on the Grade 3 reading test, and the Washington Post had accepted that claim at face value.

The Post got fooled atop its front page, then in an upbeat editorial. Based on long experience with bogus test scores, we didn't (necessarily) believe what we read that first day, and we decided to check it out.

It took several months of effort to get clear on what actually happened. You can review the search in our 2006 archives, running from February 6 through March 23 on a nearly daily basis, and then occasionally after that.

Just for the record, here's how crazy it was! Below, you see one torrent of official language explaining the way this policy worked.

We highlight the last part only. In that part of this torrent of language, the state explains what to do if more than 100 percent of your school's kids pass a given test!

Remediation Recovery, which started with the 2001 SOLs, is a third reason for apparent score disparities. Students in grades 4, 6, or 9 may retake failed English: Reading/Literature and Research or mathematics tests for grades 3, 5, or 8, respectively, following a Remediation Recovery program. Additionally, students who failed Algebra I, Geometry, or Algebra II and who are enrolled in a Remediation Recovery program may retake a given EOC mathematics test. Tables 6, 7, 15, 17, and 20 display the number of students who retook the failed SOL, the percentage who passed, the number who passed (Bonus number), and the potential benefit to the school (Recovery Bonus or Unadjusted + Recovery score). In the State's calculations to determine accreditation, the number of students who pass the targeted test following a Remediation Recovery program will be added to the number of students who passed the SOLs in the same content area. For example, a fourth grader’s passing grade 3 mathematics score will be added to that school’s grade 3 mathematics passing scores. At other grade levels, the passing mathematics score will be added to the school’s “collapsed” SOL mathematics scores (for accreditation calculations, all mathematics scores are collapsed or averaged together to create one passing percentage). Remediation Recovery students will be included in the unadjusted number of students who passed, but not in the number of students tested, hence the term Recovery Bonus. Said another way, passing Remediation Recovery students are added to the numerator, but not to the denominator. What this means is that a passing percentage exceeding 100 percent is possible (Note: while this overview reports percentages more than 100 percent, the State caps pass rates at 100 percent).

Good God! Under the state's official policy, "a passing percentage exceeding 100 percent is possible!" 

Also, note the boondoggle about adding the number of "Remediation Recovery students" to the numerator, but not to the denominator. Under this absurd procedure, it could turn out that more than 100 percent of your school's kids had passed some particular test!

At that point, the state would step in! When schools ended up with passing rates which exceeded 100 percent, the state of Virginia had decided to "cap" that (impossible) passing rate at a mere 100 percent!

Was this policy adopted in good faith? We aren't mind-readers here.

That said, the Post believed what it was told about Maury's passing rates. It didn't examine the small print in the state's reports on individual schools like Maury, where puzzling statistical contradictions immediately began to turn up.

Based on long experience with fraudulent test scores, we pretty much didn't believe what we read in the Post about those high scores. We spent several months conducting a search.

Back to the question which has triggered the present search:

Everything was not as it seemed with the passing rates at Maury. Is it possible that things are not completely as they seem with those improved Mississippi test scores?

Tomorrow, we'll start to examine that question. We'll offer a bit more of this history of bogus test scores in this week's afternoon submissions.

Tomorrow: The AP's explanations


32 comments:

  1. From Peter Greene at Curmudgucation:

    "At Forbes, I’m taking a look at Anya Kamenetz’s book about the pandemic.

    In The Stolen Year, NPR education reporter Anya Kamenetz doesn’t definitively answer the question, “Who stole it?” But she is absolutely clear on who the year was stolen from.

    Children.

    Early on in the book, Kamenetz quotes Rebecca Winthrop, a senior fellow at the Brookings Institute and expert on education in crisis situations. “Amazingly,” Winthrop observes, “kids are pretty resilient. But you have to find ways to give them two things.”

    [First], a normal sense of routine. It doesn’t have to be the exact replica of schooling, but if you give them a normal sense of routine with enough activities throughout the day, it really helps reduce their anxiety and supports their overall psychosocial well-being.

    The other thing you need to do is really find deep ways of supporting kids’ caregivers, their parents, and their teachers. Often people forget until halfway through the emergency response that parents and teachers are also affected by the crisis and they’re dealing with their own multitude of problems.

    “During COVID,” Kamenetz writes, “the United States didn’t do either.”

    The book is unflinching in looking at how “the crisis exposed fault lines that run to the core.” Hindsight is already allowing folks to pretend that there were simple, clear answers. The book’s chronological approach reminds the reader just difficult it was at the time, in an atmosphere charged with too much politics and too little information, to choose from among a small menu of options that were all bad."

    https://www.forbes.com/sites/petergreene/2023/05/30/hard-pandemic-lessons-from-the-stolen-year/?sh=7c3f87481ff4&utm_source=substack&utm_medium=email

    Somerby said not one single word during about the impact of the pandemic on schools, parents, and kids. If someone cared about children and education, he would have cared during the 2020 crisis and following recovery. Instead Somerby wastes his breath sniping at the progress that is evident in Mississippi and elsewhere in the South.

    Somerby talks about triggered searches. I find myself easily triggered by Somerby's utter neglect of what matters in education, in favor of undermining the efforts of educators, parents, and yes, children, during these fraught times. I do not have words for how angry this series of essays by Somerby has made me. The biggest fraud is not NAEP scores but Somerby's pretense at caring for kids and their education. There is not a bigger asshole on this planet today than Somerby.

    ReplyDelete
  2. Our schools have just come through harrowing times but Somerby is obsessed with a tiny classroom of 3rd graders (19) in 2006? What is wrong with him?

    ReplyDelete
    Replies
    1. What’s wrong with you, reading a blog you don’t like?

      Delete
    2. How do you know whether you like it or not without reading it?

      Delete
    3. You look at the first letter and guess the rest.

      Delete
  3. "We didn't necessarily disbelieve that news report. But through long experience, we knew that upbeat news reports of that type often turn out to be bogus."

    Somerby has said that he knows of no cheating on the NAEP test by Mississippi, Alabama or Louisiana. He has said that he knows of no cheating by Maury. But he also says that upbeat news reports of educational progress "often" turn out to be "bogus". And yet he has made no case that this is true -- the "often" part. Yes, there have been a few cheating scandals, but those are few compared to the very large number of school districts nationwide. What turns an outlier event into "often"? Nothing that Somerby has reported. Today he reaches into 2006 for an example of statistical oddity that has nothing to do with cheating, hoping his readers will think he has provided evidence that results are bogus.

    This isn't what evidence looks like, but it is what a kook with a grudge looks like.

    Most parents judge the effectiveness of their schools by their own kids' experiences, not by esoteric statistics that mostly confuse readers. Somerby's vitriol against schools that work is the most puzzling part of today's essay. He needs to come clean about why he cannot stand it when a school makes progress -- because that is not anyone else's usual reaction to good news about reading scores.

    ReplyDelete
  4. Three posts all saying the same thing in the space of 10 minutes.

    Nice work “a”!

    ReplyDelete
    Replies
    1. "a" troll. ;)

      Delete
    2. If you think the three posts were saying the same thing, you need to actually read them. The first was about Peter Greene's recommendation of a book about how the schools fared during covid. The second is asking why a trivial statistical problem in 2006 is important to anything. The third one objects to Somerby calling good results "bogus" without evidence. I wouldn't call that "all the same thing" but your point is to call other people names, not to discuss anything about kids or education.

      Delete
    3. They all share a number of logical and reasoning fallacies. I would be glad to document each and every one of them if you would like.

      Delete
    4. I don't care.

      Delete
    5. Ha. The bottom line indifference from Mr. Logical Falicy.

      Delete
    6. Anonymouse 1:55pm, you could start with tu quoque.

      Somerby didn’t care about that, so how can he care about this.

      The anonymouse 10:53am, goes beyond that fallacy by negating the importance of such student progress testing by arguing that there are few cheating scandals as compared to the number of school districts (!), and making an appeal to the authority of parents in gauging academic progress.

      (Irony, thy name is anonymouse”.)

      Delete
    7. Somerby has never made the case that there has been enough cheating to remain eternally skeptical.

      Delete
    8. Anonymouse 5:07pm, are you serious?

      Zero incidents of cheating are enough for everyone to harbor healthy skepticism and to be watchful of student progress testing. Let alone when the results are described as being a “miracle”.

      Several incidents of cheating or of faulty processes is merely reinforcement.

      Delete
    9. Going from almost last to 21st is outstanding. Why get hung up on the word “miracle”? It looks as though Mississippi is doing something right. Why won’t Somerby say what his argument is?

      Delete
    10. Once you get suckered into believing CRT was being taught in grammar schools, you can believe almost anything is possible.

      Delete
    11. Goldilocks and the 3 trollsJune 1, 2023 at 12:14 PM

      "several" is doing a lot of heavy lifting for you, box o' whine.

      Delete
  5. Karen is innocent.

    ReplyDelete
  6. The second amendment is evil.

    ReplyDelete
  7. Somerby calls this a study in pride, deliberately misusing the term applied to LGBTQ+ people. Pride in achievement is fully appropriate, but it was the NAEP who released the scores and the AP who wrote a report about them. It isn't as if individuals are profiting off the so-called miracle, the way Rhee did decades ago. Cui bono in the current situation? Hopefully the kids did.

    ReplyDelete
  8. "Based on long experience with fraudulent test scores, we pretty much didn't believe what we read in the Post about those high scores. We spent several months conducting a search."

    Was this a good use of Somerby's time? What exactly was at stake for anyone that would justify this use of time?

    ReplyDelete
    Replies
    1. This blog is not a good use of YOUR time.

      Delete
    2. I am retired. If I want to use my time trying to educate the misguided trolls who inhabit this blog, or warn the readership that Somerby isn't always on the up-and-up, that is my business. What do you suppose Somerby's motive was for using his time to try to discredit a small school in VA?

      Given all the time Somerby says he has invested in looking for fraud in public schools, do you think he has provided the evidence to support a claim that test scores are bogus in most school today?

      Was the Maury school committing fraud or was it keeping a running total of how many students had passed the 3rd grade test and how many were still lagging behind? Where is the statement showing that Maury used a 100%+ figure to claim wonderful results? Did they broadcast any miracle to the press based on the fact that most of its kids could pass a 3rd grade test by the end of 4th grade?

      Or did you not bother trying to follow Somerby's narrative, and are you just swallowing his unsupported claims because he is promoting a preferred narrative about those ratting public schools?

      Delete
    3. correction: ratting should be "ratty," a term Somerby used to apply to teachers and their unions.

      Delete
    4. Your criticisms are rarely not full of mistakes, misunderstandings, misreadings and bizarre, nonsensical paranoia. You're wasting your life and your retirement.

      Delete
  9. Defund the Supreme Court.

    ReplyDelete
  10. Somerby reports the 3rd & 4th grade test procedures as if there were some shenanigans taking place designed to inflate the success rate. But consider this. Children develop at different rates. If a child cannot pass a tough reading test (based on the 26% pass rate) at the end of third grade, they are given more instruction in reading in 4th grade and then take the test again, to see if they have improved sufficiently to pass the 3rd grade test. Most do by then, with the additional instruction. That is a good thing, but Somerby's focus is on how close they are to 100%, not the fact that nearly all kids can pass the test by the end of 4th grade. Recall that educators use test scores for assessment, to figure out what kids need, not to brag about being closer to 100% a year later. The small number of kids who still cannot pass the test need to be identified so that they can get extra help.

    Somerby presents these procedures as if they made no sense, when they do, in the the context of the varying developmental maturation rates of actual children. Somerby only cares about the pass/fail rate, not the need to identify what kids need and where they stand in their reading progress. Somerby thinks the procedures are conniving to inflate perceived success, but how can that be when even Somerby was aware of how many initially failed the test, and how many were successful a year later.

    For schools, the numbers are not the point. Children's learning is the point. For politicians or critics of public education, the scores may be the point, not the students. And that obsession with scores and tests is exactly why some districts are driven to cheat. But I don't see Maury's procedures as an example of cheating or craziness. I see it as Somerby's failure to understand how one small school keeps records of which students are doing well and which are struggling. And it doesn't seem to me that Somerby cares much about that -- given his obsession with proving that educators are doing something wrong with numbers Somerby clearly doesn't understand, given that he thinks their purpose is accountability to the public and not assessing the ongoing needs of children.

    ReplyDelete
  11. We live in a barred spiral galaxy.

    ReplyDelete
  12. Again Somerby implies that there may be something bogus about Mississippi’s naep test scores. This is the sixth day running that he fails to show what he is talking about. The discussion of cheating on other tests at one time gives the reader the impression that something may be amiss in Mississippi.

    ReplyDelete
  13. The board should never have capped the success rate. If more than 100% of the kids succeed, it should be reported.

    ReplyDelete
  14. Harry Litman is God.

    ReplyDelete