This is lousy reporting: Yesterday, the 2013 NAEP scores were released.
The National Assessment of Educational Progress is widely described as the gold standard of domestic educational testing. It’s hard to argue with that.
In the past decade, the various statewide testing programs have been deeply corrupted. The federally-managed NAEP, more than forty years old, is the only real game in town.
For our money, the score gains achieved since 2011 are quite underwhelming. In this morning’s New York Times, Motoko Rich was able to perform some basic reporting about those gains:
RICH (11/8/13): American fourth and eighth graders showed incremental gains in reading and math this year, but achievement gaps between whites and blacks, whites and Hispanics, and low-income and more affluent students stubbornly persist, data released by the Education Department on Thursday showed.Rich buried the three-point gain in Grade 8 reading. She introduced confusion by saying, “the results represent both public and private schools.”
The results of the tests—administered every two years as the National Assessment of Educational Progress, sometimes called the nation’s report card—continued an upward trend in both areas over the past two decades.
[...]
The average fourth-grade math score this year was 242 on a scale of 500, up from 241 in 2011, the last time the federal assessment results were released. The average eighth-grade math score was 285, up from 284 two years ago.
In reading, the average fourth-grade score edged up slightly to 222 from 221 two years ago, while the average eighth-grade score rose to 268 from 265. About 400,000 fourth graders and 350,000 eighth graders took the exams; the results represent both public and private schools.
In context, do you know what that means? We don’t. It’s pure confusion.
Rich’s instant focus on the persistence of racial achievement gaps is thoroughly standard, but thoroughly pointless. Did someone think those gaps would go away in the course of the last two years?
Meanwhile, what kind of progress did black kids show? That question is never addressed. How large has the progress been down through the years?
That info is also missing.
In our view, Rich did notice one of the most striking aspects of the new scores—the large gains recorded in the D.C. public schools. That said, her technical incompetence obscures the realities:
RICH: Kaya Henderson, the public schools chancellor in Washington, described the increases in the district as “breakthrough gains.” Average scores for public and charter school students in fourth-grade reading increased by five points from 2011, and eighth-grade reading moved up six points, half the total gain in average scores since 1998. Average fourth-grade math scores among public school students rose by seven points, and eighth-grade scores by five points.Just for starters, good God! “Ms. Henderson said?”
New York, by contrast, had smaller gains, and scores in eighth-grade reading were flat compared with 2011.
Although they came from a low base, Washington’s black students, who represent about three-quarters of public school enrollment, raised their average scores in both subjects and grades to their highest levels on record, Ms. Henderson said.
Did a major reporter actually write that? Why didn’t Rich examine the data to see if Henderson’s statement was true?
Rich could and should have fact-checked that claim. For all NAEP data, start here.
How weak is Rich as a journalist? Consider:
When she refers to “New York,” does she mean the city or the state? Everyone gets to guess! Meanwhile, she overstates the size of DC’s score gains by her failure to disaggregate scores. Part of D.C.’s overall gain seems to have been caused by the presence of a higher percentage of white students in 2013. As always in this bifurcated nation, we would have gotten a clearer idea of what happened in DC if Rich had presented the scores for each major part of the student population.
That said, DC’s score gains were large. That remains true even if you look at the data yourself, instead of repeating what Henderson said, and even if you examine each part of the student population.
In our view, one part of Rich’s news report is just egregiously bad. This is heinous reporting:
RICH: In absolute terms, the states with the highest average scores included Massachusetts, Minnesota and New Jersey. In some cases, it appeared that demographics affected the results. These high-performing states have lower proportions of students eligible for free or reduced-price lunches than the national average, and much lower rates than some of the lowest-performing states.That strikes us as awful reporting. Here’s why:
“Absolutely, without question, demographics are a factor,” said Mr. Buckley of the National Center for Education Statistics. But, he added, some states with similar demographics did not perform as well, and some states with high populations of typically underserved students showed gains. “Demography is not destiny,” he said.
In Massachusetts, where average fourth-grade reading scores actually declined between 2011 and 2013, Mr. Buckley said the drop was driven by falling scores among low-income students.
Rich cites three states for achieving the highest average scores “in absolute terms,” whatever that means. She then makes an astoundingly fatuous statement:
“In some cases, it appeared that demographics affected the results.”
In some cases, it appeared that demographics affected the results? Of course demographics affected results; the statement is utterly fatuous. That said, to what sorts of demographics was Rich prepared to refer?
Alas! In discussing those states, Rich refers to demographics of income, ignores demographic of race. If we’re trying to inform New York Times readers about the real world, that decision was very bad.
Who knows? Perhaps Rich’s editors felt that the Times had already engaged in enough race talk. That said, consider the strangeness of Rich’s presentation. Consider one state she singled out and two other states she did not.
Presumably, Rich is citing Massachusetts, Minnesota and New Jersey as the states whose overall student populations scored best. That is a reasonable selection, though Minnesota ranks a clear third best in that group.
But consider—did Minnesota gain an advantage on other states because it has fewer low-income students? Please! Minnesota is on that list because it has fewer minority kids.
Consider the way Minnesota’s three student groups performed:
Minnesota’s rank among the fifty states, Grade 8 math:Minnesota’s white students were third in the nation. Its black and Hispanic students scored in the middle of the pack. Minnesota only scored well overall because it has an unusually large percentage of white kids. (For details, see below.)
White students: 3rd in the nation
Black students: 23rd
Hispanic students: 29th
By way of comparison, this is the way Texas students ranked:
Texas’ rank among the fifty states, Grade 8 math:Say hello to Cordelia! White students scored well in both these states. But all three groups scored well in Texas. In Minnesota, black kids and Hispanic kids recorded average scores.
White students: 4th in the nation
Black students: 3rd
Hispanic students: 3rd
Despite that, Rich cited Minnesota, disappeared Texas, then turned euphemistic about the relevant demographics. We’d say she shortchanged Maryland too:
Maryland’s rank among the fifty states, Grade 8 math:Which state did better overall—Maryland or Minnesota? Maryland ranked high with all three groups of kids, Minnesota with just one.
White students: 6th in the nation
Black students: 7th
Hispanic students: 4th
Are we cherry-picking our subject? Here you see rankings for Grade 8 reading:
Minnesota’s rank among the fifty states, Grade 8 reading:Which of those states did better “in absolute terms?”
White students: 9th
Black students: 24th
Hispanic students: 23rd
Maryland’s rank among the fifty states, Grade 8 reading:
White students: 3rd
Black students: 2nd
Hispanic students: 3rd
We’re going to take a guess. Newspapers like the Times don’t like to talk about race. They’re always willing to feature those racial achievement gaps. That’s a standard part of the package. It lets everyone wring his hands.
After that, they prefer to stop talking race. They won’t discuss the gains by black kids over the years. And if they talk about “demographics,” they’ll tend to go euphemistic.
Politely repeating what Henderson said, Rich broke that rule in discussing D.C. That put her well over her newspaper’s racial quota. As a result, she told us that Minnesota was one of the three best states, then got wobbly about the nature of the “demographics” involved.
This is bad reporting. It’s based on a flight from race, the fear of telling the truth about the results of our brutal history.
Big news orgs need to learn how to talk about black kids. On average, black kids still trail white kids in academic achievement, although the gaps have gotten smaller and will be gone some day.
Big newspapers need to discuss that. Also:
They need to be willing to tell the world, without a sense of embarrassment or assault, how things got this way. The New York Times should learn how to discuss our gruesome American past.
For the record: The new NAEP reports don't seem to show what percentage of kids in each state were low-income. They do show state demographics by race. To wit:
On this year’s Grade 8 math test, 74 percent of Minnesota’s students were white. In Texas, the corresponding figure was 32 percent. In Maryland, 45.
That’s the basic demographic difference here. The New York Times, a big newspaper, should learn how to discuss such matters. We need to learn how to discuss our nation’s great kids and our horrific past.
The Times will not discuss the top performance of Texas students across all races until Texas becomes a blue state. Then, voila!
ReplyDeleteLeave now.
Delete"But still, far less than half of the nation’s students are performing at a level deemed proficient in either math or reading."
ReplyDeleteTidbit from Motoko Rich's article deleted for some reason or another from this post.
There are probably a whole bunch of sentences from the article that were not included in this post. What's your point?
DeleteOMB (Gack)
ReplyDeleteThis was an excellent piece BOB.
"For the record: The new NAEP reports don't seem to show what percentage of kids in each state were low-income." BOB
No, but they do show comparsions between and ranking of states based on student Eligibility for Free and Reduced Lunch, just as they do for race and ethnicity. In that category Massachusettes has one of the consistently highest (and statistically significant) gaps in performance based on that rough measure of income. On that measure, the higher the score the worse a state ranks. Texas doesn't fair as well as they do based on race using BOB's measure. Their GAP seems mostly better than the national average but not significantly so.
KZ
Update: The NAEP statistics on percentage of students Eligible for Free and Reduced Lunch can be found on the individual snapsho section for each state under theheading "State Profiles."
DeleteBut gack, that only measures people around twice the federal poverty level or below.
KZ
On the NEAP "proficient" means above average, so, if less than half are proficient (i.e., outstanding) that makes sense. Half are below average, by definition. Most are average, and some are outstanding.
ReplyDeleteThis is not an international comparison test For that look at the TIMSS scores.