Part 4—Rising scores on the SAT and the ACT: Are seniors in American high schools doing less well in reading and math?
In theory, that’s an important question. In theory, we the people want to know the answer. In theory, our journalists and “educational experts” are trying to help us get answers to questions like that.
In practice, things are quite different. In reality, no one gives a flying fig about questions like that, except to the extent that such questions can be used to advance prevailing narratives and agendas.
On the front page of last Thursday’s Washington Post, we saw the con happen here:
ANDERSON (9/3/15): The steady decline in SAT scores and generally stagnant results from high schools on federal tests and other measures reflect a troubling shortcoming of education-reform efforts. The test results show that gains in reading and math in elementary grades haven’t led to broad improvement in high schools, experts say. That means several hundred thousand teenagers, especially those who grew up poor, are leaving school every year unready for college.Is it really difficult to “pinpoint a reason for the decline in SAT scores?” On the one hand, yes it is. In fact, it’s impossible to “pinpoint” some exact reason for the decline in (average) scores.
“Why is education reform hitting a wall in high school?” asked Michael J. Petrilli, president of the Thomas B. Fordham Institute, a think tank. “You see this in all kinds of evidence. Kids don’t make a whole lot of gains once they’re in high school. It certainly should raise an alarm.”
It is difficult to pinpoint a reason for the decline in SAT scores, but educators cite a host of enduring challenges in the quest to lift high school achievement. Among them are poverty, language barriers, low levels of parental education and social ills that plague many urban neighborhoods.
On the other hand, it isn’t even slightly hard to identify certain factors which have probably helped to lower average national scores. Doing so is the easiest task in the world. Everyone knows how to do it.
On the front page of last Thursday’s Post, Anderson skipped right past those obvious factors. Instead, he played a familiar game, misusing SAT scores as he did:
First, he assumed that the decline in average SAT scores means that high school seniors are doing less well in reading and math. From there, he proceeded to the only factor that entered his head as he tried to explain this decline. He assumed the decline must be related to efforts at “reform.”
Our view? In that passage, we’re looking at journalistic malpractice, of a familiar type.
In fact, the SAT is not designed to facilitate the type of year-to-year comparisons in which Anderson was engaging. The College Board makes no attempt to test representative samples of high school seniors. The tests are given to all who apply—and the demographics of the population taking the tests are changing every year.
Duh! These demographic changes affect average national scores in ways which everyone understands. Those changing demographics create a confounding statistical mess for those who want to use SAT scores in this ham-handed way.
How do demographic changes in the tested population create a statistical mess? Let us count the ways:
More kids taking the test: As the SAT notes in its basic materials, more students are taking the tests every year. Given the history of SAT testing, this tends to lower average national scores, although that won’t always be the case.
A changing racial and ethnic blend: Every year, more black and Hispanic students take the SATs. Given our nation’s brutal racial history—given some ongoing policy practices—this tends to lower average scores.
Everybody knows that, but there’s more! In some large urban districts (see below), all students are now required to take the SATs, whether they plan to go to college or not. This may be good educational practice, but it tends to lower national average scores.
The rise of the ACT: Over the course of the years in question, the ACT has supplanted the SAT as the nation’s predominant college admission test. Many students take both tests, but many students have switched from the SAT to the ACT.
This test-switching adds to the statistical chaos when we look at national average scores on these tests. By the way, average scores have not declined on the ACT, even as it has surged past the SAT as the nation’s predominant program.
Taking the SATs as a junior: A substantial number of students take the SATs as juniors. For whatever reason, the percentage has risen in recent years, to the current 35%. Presumably, this changing percentage adds to the difficulty in making valid year-to-year comparisons.
At this point, might we make the world’s most obvious statement? The SATs were not designed for year-to-year comparisons. In fact, the SATs weren’t designed to measure populations at all. The SATs were designed to measure individuals.
The SATs were not designed to measure populations! The College Board makes no attempt to test a representative sample of any population, whether it’s the entire national high school population or the entire national high school population of some particular group.
No representative samples are involved in this process at all! To understand how the changing demographics of the tested students can create a giant statistical mess, consider a recent news report in the Dallas Morning News.
Uh-oh! Average SAT scores have declined quite a bit in Texas in recent years. Last week, reporter Terrence Stutz asked a state official to explain why that is.
The state official in question may have been overstating. But she offered some obvious observations—the types of observations which were barred from Anderson’s front-page report:
STUTZ (9/3/15): State education officials have attributed the declining SAT scores in Texas to an increase in the number of minority students taking the exam. Minorities generally perform worse than white students on standardized achievement tests like the SAT and ACT, the nation's two leading college entrance exams.Why have average statewide scores in Texas declined? We can’t “pinpoint” that! But Ratcliffe is citing one obvious possible explanation—a type of obvious explanation Anderson disappeared from the front page of the Post.
Debbie Ratcliffe, a spokeswoman for the Texas Education Agency, said the lower SAT scores this year are at least partly the result of testing policies in two dozen school districts—including Dallas and Fort Worth—where all upperclassmen now take the SAT each year.
“The SAT takers in those districts include not only those who are college-bound, but the whole student population [of juniors and seniors],” Ratcliffe said. “That translates in lower average scores because the more test takers you have, the more scores will decline.”
Duh! According to Ratcliffe, two dozen districts are now requiring all their students to take the SATs, not just those students who are college-bound as judged by traditional measures.
This may be good educational policy; we have no view about that. But in large urban districts like Dallas, this will mean that many low-achieving students will now be taking the tests. Previously, they would not have.
This will lower average scores; this fact is blindingly obvious. But the decline in average scores won’t tell us anything about the Texas student population as a whole, or about the role of “education reform” in their schools.
More precisely, this decline in average scores won’t “reflect a troubling shortcoming of education-reform efforts.” It will simply reflect the fact that a larger swath of students were tested. It will tell us nothing else.
To what extent was this year’s decline in nationwide average SAT scores caused by changes in the population tested? It’s very hard to answer that question. The SATs were not designed to let us “pinpoint” an answer.
That said, it’s journalistic malpractice when the Washington Post does what it did last week. When it offers a lengthy, front-page news report and fails to mention this obvious factor. When it moves instead to its favorite corporate theme:
Something has failed in the schools!
Has something failed in American high schools over the past few years? Because the SATs weren’t designed to answer such questions, there’s no way to tell from looking at average scores. But just to add to the fun, this is what some average scores look like on the SAT and the ACT in the last five years after you “disaggregate”—after you break average scores down by demographic group:
Average reading scores, SAT, 2011/2015All of a sudden, we seem to be looking at small gains in average scores rather than small declines. But these comparisons are bogus too. Offering one example, here’s why:
White students: 528/529
Black students: 428/431
Hispanic students: 451/450
Asian-American students: 517/525
Average composite scores, ACT, 2011/2015
White students: 22.4/22.4
Black students: 17.0/17.1
Hispanic students: 18.7/18.9
Asian-American students: 23.6/23.9
Back in 2011, the SAT didn’t test a representative sample of the nation’s black students. It didn’t test a representative sample in 2015 either.
If the SAT had tested representative samples in each of those years, those reading scores would suggest that the nation’s black kids had shown some progress in reading. But the SAT didn’t test any such samples—and given all the statistical chaos involving the SAT and the ACT, you simply can’t make a valid comparison between the two groups of kids who did get tested those years.
Having said that, let us also say this: Between 2011 and 2015, those composite ACT scores held steady (or slightly rose), even as the number of students taking the ACT increased by almost twenty percent.
By normal reckoning, you might expect average national scores to drop as the tested population increased to that extent. In this case, that didn’t happen. Might that suggest that something good occurred in the nation’s high schools?
That wouldn’t be a valid conclusion! Simply put, the SAT and the ACT aren’t designed for the purpose of making such comparisons.
The SAT isn’t designed to do that! But at newspapers like the Washington Post, invalid comparisons will be made—and narrative will prevail. When it comes to education testing, all roads will lead to this:
Something has gone wrong in the schools! Nothing seems to be working!
Last Thursday morning, the Washington Post committed a form of journalistic malpractice. Copying from the Anderson’s paper, Laura Moser quickly made things worse with this gruesome report at Slate.
How did other newspapers do—the New York Times, for example? What did the AP write? What did “educational expert” Petrilli say when he expounded on these matters at greater length? And how in the world did Kevin Drum get dragged into this mess?
Meanwhile, what about the NAEP? Don’t they test representative samples of the nation’s twelfth-graders? What does their testing show or suggest?
We’ll continue our back-to-school series next week. In the meantime, please understand—the mainstream press will almost always do two things when it discusses, or pretends to discuss, educational testing and test scores:
Reporters will reliably bungle their work on a technical basis. Then, in obedience to Hard Pundit Law, they’ll also say that nothing is working in the nation’s schools.
First, they’ll flunk the SATs in their own front-page reporting. After that, they’ll lament the way the nation’s kids don’t know how to read, write and cipher!
Coming next week or even tomorrow: Petrilli and Drum and Slate oh my! Also, what did the New York Times say? What can we learn from the NAEP?
Then, it’s back to our pre-announced back-to-school topics:
Are black kids suspended more often in the South than in other parts of the country? Also, what sort of progress has occurred in the New Orleans schools?
In theory, those are important questions. In practice, it's clear that nobody cares!