Plus, the allure of the gloom: The National Assessment of Educational Progress (Naep) has long been praised as our most reliable educational testing program.
As far as we know, the Naep is our only (presumptively) reliable domestic testing program. Our statewide testing programs have routinely been compromised, in a wide array of ways, including in the widespread cheating scandals which finally came to light through the work of USA Today and the Atlanta Constitution.
(Full disclosure! We first wrote about such problems in the late 1970s, in the Baltimore Sun.)
The statewide programs are hard to trust; only the Naep seems to soldier on. How well regarded is the Naep? Way back when, Kevin Drum described the Naep as shown below in a report for the September 2012 print edition of Mother Jones.
Kristina Ryzga had written a longer report about San Francisco's Mission High School. In a companion piece, Drum started like this:
DRUM (9/12): Standardized tests may not tell us everything there is to know about a school like Mission High, but that doesn’t mean they don’t have their uses. And one of those uses is myth busting—in particular, the myth that America’s schools are in a state of terminal decline and students aren’t learning as much as those of a generation ago.Drum's report appeared beneath an upbeat pair of headlines. The headlines chided those who would roll their eyes at the spectacular dullards long known as These Kids Today:
The real story is more complicated, and the best place to see it is the federal government’s National Assessment of Educational Progress, otherwise known as the Nation’s Report Card, the “gold standard,” or just plain NAEP. It provides a long-term set of results going back to the early 1970s, and unlike state tests, which vary substantially and are sometimes dumbed down to produce higher scores, the NAEP is widely trusted in the educational community. It can’t tell us anything about particular students or schools (since only a fraction of the nation’s schools participate), but it can give us a pretty good idea of how national averages have changed over time.
The Kids Are All Right.Drum cited data from the Naep's Long Term Trend Assessment, one of two complementary studies conducted by the federal program. The first two myths he busted were these:
Students today score better on tests than you did.
That day's kids were scoring much better on the Naep than their parents and/or grandparents had done back in the 1970s, Drum noted And not only that:
"In fact, scores for blacks and Latinos are up more than scores for whites," Drum correctly reported. These were the first two points Drum made in his upbeat, myth-busting piece.
As far as we know, Drum was right about the Naep; the program is "widely trusted in the educational community." It's also routinely described by mainstream journalists as the "gold standard" of educational testing, to the extent that mainstream journalists ever discuss the Naep at all.
In fact, mainstream journalists rarely discuss the Naep. As we've noted in the past, the Naep is the educational testing program which is widely praised but almost never analyzed or discussed.
Mainstream news orgs almost never report the basic facts which Drum reported that day. If you read the New York Times or the Washington Post, you have basically never been told about the large score gains Drum cited.
Instead, you've had a basic piece of propaganda hammered into your head—propaganda advancing the gloomy myths Drum debunked and busted that day.
America’s schools are in a state of terminal decline? Students aren’t learning as much as those of a generation ago?
Variants of these gloomy myths have dominated our upper-end "education reporting" over the past many years. Starting in the 1990s, these myths were widely used to recommend certain kinds of "education reform"—types of reform which were being recommended by the a group of wealthy benefactors including Bill Gates and the Waltons.
If you read the Post or the Times, you've been served the myths of decline over and over again. On the rare occasions when major news orgs bother to discuss public schools at all, the gloom has tended to be general all over the upper-end press.
Indeed, as of last month, the gloom even seemed to have captured our analysts' Uncle Drum! They came to us with tears in their eyes on two separate occasions.
Skillfully, we tried to explain their uncle's gloomy heresies. It started on December 17 with a gloom-inflected piece in which Drum offered this:
DRUM (12/17/19): [P]erformance in 4th and 8th grade on the national NAEP test has indeed improved. But a lot of that improvement washes out in high school.Drum was now working from the so-called "Main Naep," the companion study to the Naep's Long Term Trend Assessment. He presented charts showing that average scores at Grade 12 have slightly declined in reading since the early 1990s, even after "disaggregation," and have grown by only about 5 percent in math.
Oof! The nation's declining drop-out rate makes comparisons of this type difficult at the Grade 12 level. Where lower-performing teens were once more likely to leave school, they're now more likely to stay in school through graduation.
This decline in the drop-out rate is generally viewed as an educational success. But it likely has the effect of reducing average scores at the Grade 12 level, invalidating comparisons over time for most purposes—and explaining why larger improvement at Grade 8 may seem to have "washed out.".
We talked our crying analysts down, but their uncle struck again! On December 30, he authored a somewhat puzzling chart which led him to offer this gloomy assessment (sub-headline included):
DRUM (12/30/19): The Black-White Test Score Gap Remains IntractableOn this occasion, Drum used figures from the Grade 4 and Grade 8 Naep, but he eschewed Grade 12. Instead, he compared black/white average scores on the SAT and the LSAT, a procedure which makes no apparent sense.
The gap in achievement test scores between blacks and whites starts in kindergarten and rises steadily with age. The chart below shows the gap measured in standard deviations, which normalizes the size of the gap between different tests with different scales. It starts at about 0.5 standard deviations in kindergarten and rises to just over 1.0 standard deviations by college.
The worst part of this is that we’ve made no recent progress on it. The gap closed some during the ’70s and ’80s, but since then nothing has changed. This is one of our greatest failures as a society, and it represents something real, not just an artifact of the way we do testing. Until we address it, we will never even begin to make progress on racial justice in America.
(There is no control on who takes the SAT or the LSAT. Black and white average scores are not drawn from "representative samples" of the two groups. There is no way to have confidence in such comparative scores.)
Set the SATs and the LSATs to the side! In the highlighted passage, Drum said that we've made no progress in reducing black-white achievement gaps since the 1980s.
That isn't true on the Grade 4 and Grade 8 Naep, two types of data Drum chose to present. For example, here are some data in Grade 8 math:
Black/white achievement gap, Grade 8 math, NaepOver those 23 years, we've gone from a very large achievement gap to a smaller achievement gap which is still very large.
1996: 40.22 points
2019: 32.25 points
Last year's gap was very large but, in fact, it was smaller. In fairness, the following point must also be made:
The gap persists despite large score gains by black kids. The average score of black eighth-graders rose by 19.93 points over that 23-year span—by almost two academic years, according to a widely used but very rough rule of thumb.
The gap persists because white eighth-graders are also scoring better. This is the type of upbeat point Drum made in 2012, before the gloom took control.
There's good and there's bad in Naep data. Some of the bad is awful. But in our view, it doesn't make sense to call in the gloom on a selective basis.
That said, the gloom has massively prevailed when our upper-end press corps pretends to discuss our public schools. That's why we were so surprised by a December 6 opinion column in the New York Times.
Unheard of! In her column, Emily Hanford reported an accurate fact. Reading scores on the Naep have been on the rise in Mississippi over the past six years!
According to the Times' identity line, Hanford is senior education correspondent for a branch of American Public Media, a major producer of pubic radio programs. In reporting upbeat news from the Naep, she was breaking every rule in the upper-end journalist playbook!
Hanford works for American Public Media; her claims appeared in the New York Times. Needless to say, the journalistic misfeasance turned out to be remarkably vast.
Can you trust anything you read in the New York Times? You're asking a very good question.
We'll start to answer your question tomorrow. In this, our year of thinking anthropologically, our answer—it's anthropologically sound—will turn out to be no.
Tomorrow: The claim
Thursday: The letter