INSTANT ASSESSMENTS: Ravitch blasted Kristof's claims!

MONDAY, JUNE 5, 2023

Her work was astoundingly bad: All last week, we outlined some of the background to our latest search. 

(Note: We'll be AWOL for several days later this week. For that reason, we're covering a lot of ground in this morning's report.)

As you may recall, the current search concerns a set of claims by Nicholas Kristof—a set of claims which followed, and echoed, this May 17 analysis piece by the Associated Press.

We love Kristof's values, but how good was his analytical work? The headline on his essay in the New York Times went exactly like this:

Mississippi Is Offering Lessons for America on Education

"Thank God for Mississippi," Kristof said, at the end of his lengthy June 1 essay. The following excerpts give the flavor of what he said he said he had seen in the public schools of that state:

KRISTOF (6/1/23): [I]t’s extraordinary to travel across this state today and find something dazzling: It is lifting education outcomes and soaring in the national rankings. With an all-out effort over the past decade to get all children to read by the end of third grade and by extensive reliance on research and metrics, Mississippi has shown that it is possible to raise standards even in a state ranked dead last in the country in child poverty and hunger and second highest in teen births.

In the National Assessment of Educational Progress, a series of nationwide tests better known as NAEP, Mississippi has moved from near the bottom to the middle for most of the exams—and near the top when adjusted for demographics. Among just children in poverty, Mississippi fourth graders now are tied for best performers in the nation in NAEP reading tests and rank second in math.

[...]

Mississippi has achieved its gains despite ranking 46th in spending per pupil in grades K-12. Its low price tag is one reason Mississippi’s strategy might be replicable in other states...

“This is something I’m proud of,” said Erica Jones, a second-grade teacher who is the president of the Mississippi affiliate of the National Education Association, the teachers’ union. “We definitely have something to teach the rest of the country.”

Readers, how about it? Does Mississippi "definitely have something to teach the rest of the country?"

After reading the original AP report, we initiated a search of an answer to that question. At this point, three weeks later, we remain unsure of the answer.

Back to Kristof's essay:

"Thank God for Mississippi," Kristof said at the end of his lengthy piece. How had this low-income state achieved such success? Kristof said it was largely due to three policies:

He said it was due to the teaching of phonics. He said it was due to early intervention with kids who are struggling. 

He also said the success was due to a third-grade retention policy, in which kids have to repeat the third grade if they can't pass an end-of-year reading test.

We have learned, through long experience, to question claims of surprising test score gains. For that reason, we set out on a search.

So did education writer Diane Ravitch—and she drew an instant conclusion. After a search of roughly one day, Ravitch's headline said this:

Nicholas Kristof Does Not Know How to Fix Education

So went Ravitch's instant assessment. In our view, her work was astoundingly poor and her instant assessment was worthless.

Dear reader, don't get us wrong! As she conducted her overnight search, Ravitch gave voice to several points of concern we'd be inclined to share.

As she started, Ravitch said that Kristof "is terrific on many issues but consistently wrong when he writes about education." As we noted last week, we've had the same impression in the past.

As we noted, it has seemed to us that he's too inclined to accept the pronouncements of the educational experts—for example, of those who said that "nothing was working" even as Naep scores kept rising for all demographic groups over an extended period of time.

We tend to agree with Ravitch's assessment of Kristof's past work.  Beyond that, Ravitch quickly challenged the third-grade retention policy. 

As with Ravitch, so too here. This is an issue which jumped out as us in Kristof's essay, and in the earlier report by the AP. 

Concerning the retention policy, here's the start of what Ravitch said:

RAVITCH (6/2/23): The 2013 legislation [in Mississippi] also enacted third-grade retention. Any child who didn’t pass the third-grade reading test was retained. Most researchers think retention is a terrible, humiliating policy. But Kristof assures readers that failing students get a second chance to pass. 9% of students in third grade flunked. He considers this policy to be a great success, inspiring third graders to try harder, citing a study funded by Jeb Bush’s foundation (Florida also practices third grade retention, which lifts its fourth grade reading scores on NAEP).

[...]

It seems fairly obvious that the big gains in NAEP in fourth grade were fueled by the policy of holding back third graders. Jeb Bush boasted of the “Florida Miracle,” which was based on the same strategy: juice up fourth grade scores by holding back the lowest performing third graders.

Long story short! Ravitch was saying that those Grade 4 score gains have been caused by the retention policy in a way which is artificial. Briefly, let's consider why Ravitch would say that:

One thing seems fairly obvious. A state which makes a lot of third graders repeat third grade will almost surely show an improved average score on the next year's Grade 4 tests.

In Mississippi's case, here's how would work. Like Ravitch, we'll use the nine percent retention figure which Kristof cites in his piece.

As the Grade 3 retention policy went into effect, the lowest-performing nine percent of Mississippi's third graders didn't move on to fourth grade, as they otherwise would have. In this way, they were eliminated from participation in the next year's Grade 4 reading test. 

Presumably, this gave a rocket boost to the average score on that Grade 4 reading test. Here's why:

All those kids who had to repeat third grade were good, decent kids. That said, the lowest scorers in any grade group exert a large downward pull on a state's average score. 

How large might that downward pull be? Consider these two data points from last year's Naep:

U.S. public schools
Grade 4 reading, 2022 Naep
Average score nationwide: 216.11 
10th percentile score: 160.47

Nationwide, the average score was 216.11—but good grief! According to official Naep data, the lowest performing ten percent of Grade 4 test takers scored 160.47 or lower—and many scored quite a bit lower than that! 

Stating the obvious, last year's nationwide average score would have been much higher if that lowest performing ten percent had been eliminated from the exercise. Presumably, Mississippi benefitted from a large score boost on Grade 4 tests in the year its Grade 3 retention policy went into effect.

Key point! Presumably, the benefit would have been smaller in subsequent years. Here's why:

In the second year of this policy's operation, the kids who had to repeat third grade in Year 1 would now be in the fourth grade, after spending two years in third grade. It's likely that they would have remained among the lower scoring fourth graders—but after two years in third grade, plus the subsequent year in fourth grade, their scores would presumably have been better than they would have been the year before.

(We'll show you Naep data to that effect when we convene tomorrow.)

However you slice it, the basic point seems obvious. A state which retains a lot of third graders likely has a statistical advantage on the Grade 4 Naep when compared to other states which don't retain lots of kids. 

This doesn't tell us whether the retention policy is a good or a bad idea. This does mean that it's hard to make a valid statistical comparison between the different states with different retention policies.

In our view, Ravitch should probably have spent a bit more time on this particular point. Instead, she focused on a second criticism of Kristof's analysis. That critique went like this:

The so-called "Main Naep" tests reading and math in Grades 4, 8 and 12. Echoing a statement by Kristof himself, Ravitch says that Mississippi's Grade 8 gains haven't kept pace with its Grade 4 gains—score gain which (in her view) were basically phony anyway.

Ravitch hammers Kristof very hard on this point. Based on an overnight search, she offered these (bizarre) assertions about Mississippi's performance in Grade 8 reading and in Grade 8 math:

RAVITCH: Eighth grade reading scores in Mississippi have gone up over the past two decades, but scores went up everywhere. In the latest national assessment (NAEP), 37 states had scores higher than those of Mississippi on the NAEP eighth grade reading test. Only one state (New Mexico) was lower. The other 13 were tied. In Mississippi, 25% of the state’s students in 2019 (pre-pandemic) were at or above proficient, compared to 20% in 2003. Nationally, in 2019, 29% of students were at or above proficient*.

In 2019, 42 states and jurisdictions outperformed Mississippi in percentage of students at or above proficient in eighth grade math, eight were tied, and only two scored below Mississippi. 24% were at or above proficient in 2019, a big increase over 2009 when it was 15%. But Mississippi still lags the national average, because scores were rising in other states.

Has Mississippi made progress in the past decade? Yes. Is it a model for the nation? No. When impressive fourth grade scores are followed by not-so-impressive scores in eighth grade, it suggests that the fourth grade scores were anti Oakley boosted by holding back the 9% who were the least successful readers. A neat trick but not an upfront way to measure progress.

Bizarrely, Ravitch seems to think that the most recent Naep tests were conducted in 2019. 

She offers data from that year's Naep testing only. In the first of those two paragraphs, she even offers this link to an overview of Mississippi's performance on the Naep—to an overview of Mississippi's performance on the Naep in 2019!

She then proceeds to hammer Kristof on the basis of Mississippi's allegedly poor performance at the Grade 8 level in 2019. Bizarrely though, she offers only "aggregate" data, failing to make the statistical "adjustments for demographics" which let us perform a valid assessment of Mississippi's performance as compared to more affluent states.

Ravitch remains influential, especially in blue tribe circles. The link to her attack on Kristof was sent to us by a long-time local education activist, a person of justified high standing.

That said, Ravitch's account of Mississippi's Grade 8 performance is astoundingly incompetent. So it goes, again and again, when a person conducts a search of the high-end American discourse concerning the public schools.

Tomorrow, we'll show you how Mississippi's eighth graders actually performed on the most recent Naep—on the Naep which was conducted last year, in the spring of 2022.

Partisan furies to the side, we're forced to tell you this:

After making the obvious statistical adjustments, Mississippi's eighth graders outpaced almost all other states on the 2022 Naep. After making the obvious statistical adjustments, the state's eighth graders performed above the national average in almost all demographic categories, often by substantial margins.

Some of that may still reflect a misleading statistical advantage derived from that third-grade retention policy. But Ravitch—who doesn't even seem to know when the latest Naep testing occurred—failed to make the obvious statistical adjustments as she hammered a series of partisan points against Kristof.

The woods are lovely, dark and deep. But many searches, over many years, have convinced us of a surprising fact:

This is the way the discourse tends to work in this badly floundering, vastly self-impressed nation, especially when people pretend to talk about the public schools and the good, decent people within them.

Tomorrow: Mississippi's eighth grade scores