WHERE THE NAEP SCORES ARE: Reporting the gaps, ignoring the gains!

FRIDAY, SEPTEMBER 16, 2016

Part 3 in this series

Part 4—Keller's folly: The National Assessment of Educational Progress provides our only reliable educational test scores.
Colloquially, the federally-run program is known as the NAEP. Journalists routinely describe it as "the gold standard" of educational testing, as "America's report card."

As far as anyone has ever said, the NAEP has not been subject to the cheating scandals which have frequently rocked our high-stakes, state-run testing programs. Now for a brief side trip:

(We first wrote about this type of outright cheating in the Baltimore Sun in the mid-1970s. Forty years later, the nation's educational experts and education reporters began to notice the problem. In 2006, we exposed a statewide score-inflation scam which was being conducted by the state of Virginia. A leading educational expert sat on the state of Virginia's board as this scam was being conducted. Our inquiry began when a leading education reporter at the Washington Post was taken in by the scam, penning a glowing front-page report about the wonders of a certain miraculous low-income school. As we demonstrated, the miraculous school actually had the second lowest passing rate in third-grade reading in the whole state of Virginia. Eventually, the chairman of the Virginia board graciously acknowledged the scam, which he said had been unintentional. The Washington Post, where our interest in this matter began, failed to report any part of this matter. This is the way our leading newspapers report on the nation's schools.)

As far as anyone knows, the NAEP hasn't been subject to the cheating scandals which have frequently rocked our state-run testing programs. Beyond that, the NAEP is believed to be run by people who are technically competent. It isn't clear that this is the case with our various statewide testing programs, whose profusion is sometimes said to have placed a strain on the nation's psychometric resources.

As far as we know, no one has ever said that the NAEP isn't a competent program. As far as anyone can say, the major problem with the NAEP is, alas, journalistic:

Routinely, the nation's education reporters report the substantial "achievement gaps" which can be found in NAEP data. They fail to report the "truly spectacular gains" (Richard Rothstein) which can also be found there.

They report the gaps, disappear the gains! It's hard to know why this shouldn't be seen as journalistic malpractice.

Let's offer some basic background. The NAEP came into existence long ago, under President Nixon. In this overview, the nation's leading authority spells out most of the acronyms and offers some basic info:

The National Assessment of Educational Progress (NAEP) is the largest continuing and nationally representative assessment of what American students know and can do in various subjects. NAEP is a congressionally mandated project administered by the National Center for Education Statistics (NCES), within the Institute of Education Sciences (IES) of the U.S. Department of Education. The first national administration of NAEP occurred in 1969...

NAEP results are designed to provide group-level data on student achievement in various subjects, and are released as The Nation’s Report Card. There are no results for individual students, classrooms, or schools. NAEP reports results for different demographic groups, including gender, socioeconomic status, and race/ethnicity.

As the program has developed, its reach and utility have grown. For our purposes, it now has three major components:

The Long-Term Trend study has been in operation throughout the life of the NAEP. In this program, the NAEP tests 9-year-old, 13-year-old, and 17-year-old students, most notably in reading and math.

The program called The Main NAEP came into existence in xxx. In this program, the NAEP tests students in Grade 4, Grade 8 and Grade 12.

A third program, the Trial Urban District Assessment, continues to grow in its value and reach. In this branch of The Main NAEP, the NAEP tests representative samples of students in participating urban school districts. In the 2015 assessment, twenty-two urban districts took part, including those which serve the nation's five largest cities.

The aforementioned National Center for Education Statistics provides voluminous data from all these testing programs. Substantial achievement gaps can be seen in the data from all three programs. But all three programs also display those "spectacular gains."
Your nation's press corps reports the gaps, persistently skips past the gains.

Just how large are the score gains in the NAEP's voluminous data? We've often detailed those gains at this site. Doing so is utterly pointless within our journalistic culture, which is allergic to information and is almost wholly driven by adherence to preapproved story-lines (by devoted adherence to "narrative").

For today, we'll restrict ourselves to a sample from the Main NAEP. We'll restrict ourselves to the three largest groups within our student population. For a reason which will be clear below, we'll stop at 2013:

Average scores, Grade 8 math, NAEP
2000 / 2013
White students: 282.98 / 293.19
Black students: 243.27 / 262.73
Hispanic students: 251.75 / 271.02

Black kids gained almost twenty points over that 13-year period. Hispanic kids recorded a similar gain.

As you can see, substantial "achievement gaps" still existed, in 2013, in the scores of those three groups of kids. (Though you'll note that the gaps had grown smaller.) It's also true that very large gains could be observed in those scores.

This brings us to a basic question—how large are those gaps and those gains? The average score of black eighth-graders rose by 19.46 points in those thirteen years. Should that be seen as a lot or a little? Absent some sort of measuring stick, there is no way to say.

This brings us to a rule of thumb which is often applied to NAEP scores. When Richard Rothstein and Sean Reardon wrote about NAEP scores in 2011 and 2013, each man seemed to apply this rule of thumb is a fairly straightforward fashion.

(Reardon said this: "The average 9-year-old today has math skills equal to those her parents had at age 11, a two-year improvement in a single generation." Presumably, he was applying some form of the ten-point rule to math scores from the Long-Term Trend study.)

In our view, the rule of thumb to which we refer should be regarded as an extremely rough rule. But according to that rule of thumb, ten or eleven points on the NAEP scale is often equated to one academic year.

If we apply that rule of thumb, we can draw two conclusions about those recent math scores:

The achievement gaps they reveal are somewhat smaller than before, but they're still quite substantial. But so are the academic gains recorded by all three groups of kids. That's especially true of the gains recorded by the black and Hispanic students.

If we go all the way back to the start of the NAEP, the score gains recorded by these groups are substantially larger. It's astounding to think that these basic facts are almost wholly withheld from the public.

Everyone hears about the gaps. Very few people have heard of the gains. When our journalists play this double game, we'd say they're involved in malpractice.

Gains of this type can be seen in the data from all the NAEP's major programs. We'll offer a few points of caution:

Has cheating ever occurred on the NAEP? Generally, it's assumed that cheating doesn't occur on the NAEP. The NAEP has never been a high-stakes test. No one has ever seemed to have anything to gain from invalidating the program. We'll voice a concern below.

How competent is the NAEP? People who love the gloom we're handed won't want to believe that those gains are real. They may wonder if something is "wrong" with the NAEP.

It's a perfectly valid question. If any newspaper ever chose to report the score gains found on the NAEP, we'd expect its reporters to explore that question, along with the question of cheating. (Now that the NAEP is widely discussed, state superintendents have an incentive to garner high statewide scores. Trust us: If there's a way to gimmick the sample of students who get tested, some superintendent has already done so.)

How large are those gains? Thirteen years later, did black eighth-graders really know that much more math? Were they really two years ahead of their counterparts from the 2000 testing?

That is an excellent question. Because the nation's major news orgs refuse to report those impressive score gains, no one has ever stooped to the task of asking such obvious questions.

A few more points should be mentioned:

We've been looking at "disaggregated" scores—at the scores which have been recorded by different population groups. Overall score gains tend to be smaller, for reasons we'll discuss before our report is done.

Beyond that, some readers may be disturbed by the relatively low number of students who are rated "proficient" on the NAEP. Gloom and doom are always appealing, but we should consider two points:

First, the question of what counts as "proficient" is a subjective assessment. The late Gerald Bracey frequently argued that the NAEP has set its bar for "proficiency" at unnaturally high levels. If newspapers ever return to reporting, they should examine this question.

A second point remains basic. The number of students rated "proficient" is higher than it was in the past. This fact is lost when major newspapers refuse to report those score gains.

It's also true that many of Finland's miraculous kids would presumably fail to rate "proficient" on the NAEP, just as our own kids do. As we noted last week, Massachusetts students matched their Finnish counterparts on the 2012 PISA, exceeded them on the 2011 TIMSS. Despite those facts, our journalists keep flying to Finland to gaze on the wonders found there! The American public is never told that Bay State kids are exceeding the distant miracle workers about whose ballyhooed greatness we are constantly and propagandistically told.

We hear about gaps, but not about gains. We hear about the wonders of Finland. Those Bay State kids can go hang.

What happens when a nation's press corps plays such a double game? Consider what Bill Keller said, back in 2013.

Keller is a major American journalist. In 1989, he won a Pulitzer for his international reporting. From 1997 through 2001, he was managing editor of the New York Times. He was executive editor of the Times from 2003 through 2011.

In 2013, Keller was writing a weekly column for the Times. He had never been an education reporter, but he felt he knew enough to make a dramatic statement:

KELLER (8/19/13): The Common Core, a grade-by-grade outline of what children should know to be ready for college and careers, made its debut in 2010, endorsed by 45 states. It is to be followed in the 2014-15 school year by new standardized tests that seek to measure more than the ability to cram facts or master test-taking tricks...

This is an ambitious undertaking, and there is plenty of room for debate about precisely how these standards are translated into classrooms. But the Common Core was created with a broad, nonpartisan consensus of educators, convinced that after decades of embarrassing decline in K-12 education, the country had to come together on a way to hold our public schools accountable.

When he wrote that column, Keller apparently thought that our public schools had experienced "decades of embarrassing decline." But in the previous thirteen years, black and Hispanic eighth-graders had recorded very large score gains on the NAEP math test.

Presumably, Keller had never heard about that. Very few people had. In fairness, the gentleman had an excuse:

He reads the New York Times!

Coming—Week 3 of this report: "Where the gaps are"