Workshop: How can score gains look like that!

WEDNESDAY, NOVEMBER 13, 2013

Imagine two groups of kids: In our last post, we presented some data which may not seem to make sense. Here you see score gains recorded on the NAEP over the past several decades:

Score gains by public school students, 1990-2013
Grade 8 math, NAEP
All students: 21.88 points
White students: 24.12 points
Black students: 26.61 points
Hispanic students: 26.36 points

How weird! If the average score of each group went up by more than 24 points, why did the overall average score go up by less than 22 points? That doesn’t seem to make sense.

Don’t blame it on the smaller group of Asian-American kids. Their average score went up by 31.10 points! Start here, then click MAIN NDE. Then, continue clicking.

Unless we're mistaken, this is a version of Simpson's paradox. But whatever you call it, it’s easy to demonstrate the way this counterintuitive manifestation can work.

Imagine two groups of kids, the “mountain kids” and the “valley kids.” They constitute an entire school system.

Members of these two populations are tested in Grade 8 math in two successive years.

For each group, the average score goes up in the second year—but the overall average score goes down! Here’s how that can work:

In the first year of testing, there are 800 mountain kids and only 200 valley kids. The mountain kids average 100. The valley kids average 40.

That produces an overall average of 88. You can work it out.

In the second year of testing, there has been a population shift. There are now 200 mountain kids and 300 valley kids.

This year, the mountain kids average 110. The valley kids average 60. For each group, the average score has gone up. But uh-oh:

The overall average has dropped to 80! Again, you can work it out.

This seems counterintuitive, paradoxical. The average score for each group goes up. But the overall average score goes down.

That may seem weird. But that’s the way it can work when the relative size of the sub-groups changes.

Question:

Should a reporter only say that the overall average score went down? Or should he also report the fact that the average score for both groups of kids went up?

On the NAEP, black kids have been recording large score gains. Almost invariably, mainstream reporters report the achievement gap, fail to report the score gains.

This produces the gloomy impression elite “reformers” love. It keeps the public from learning about the large score gains which have been produced by our black kids. And by our Hispanic kids. And by our white kids too.

Reporters have to report those score gains! From there, you could even start to ask what the large score gains might mean.