EXPERTS, PROFESSORS AND JOURNOS: What kind of “expert” reasons this way?

THURSDAY, SEPTEMBER 17, 2015

Part 4—Truly horrific work:
Are seniors in American high schools doing less well in math?

In theory, it’s an important question. In practice, it’s clear that nobody cares. For various reasons, it can also be a tricky question—a difficult question to answer.

Are seniors doing less well in math? In theory, the question’s important. For that reason, we’ve spent several days trying to puzzle it out—more days than we had planned.

It’s also important to understand something else. It’s important to understand the way such questions get churned at our biggest establishment news orgs.

Routinely, our journalists turn to a stable of “educational experts” as they produce their scripted reports about the allegedly floundering public schools. All too often, these experts seem to serve as conduits of establishment narrative, sometimes in defiance of basic obvious facts.

Michael Petrilli is an establishment educational expert. For the record, he’s head of the Thomas B. Fordham Institute, “an ideologically conservative American nonprofit education policy think tank with offices in Washington, D.C., Columbus, Ohio, and Dayton, Ohio.”

(Unless you read the Washington Post, in which case the Fordham Institute is simply an “educational think tank.” Whatever!)

Are seniors in American high schools doing less well in math? On September 3, Petrilli explored this general question in a blog post bearing this title:

“Why is high school achievement flat?”

In our view, Petrilli’s blog post is horrific—a moral and/or intellectual disgrace. On the brighter side, it helps us see the way our “educational experts” routinely function inside the hall of mirrors we call our “public discourse.”

What makes that blog post such a mess? Yesterday, we started to answer that question. Today, let’s run through the basics.

In fairness to Petrilli, he seems to understand several things about the interpretation of test scores. In particular, he understands some of the problems people encounter when they try to interpret Grade 12 scores—when they try to evaluate progress, over time, at the Grade 12 level.

What does Petrilli understand? Consider three basic points:

The SATs weren’t designed for that purpose: Petrilli seems to understand a basic fact—the SATs were not designed for that purpose!

More specifically, the SATs were not designed to permit comparisons of America’s high school seniors over time. In fact, the SATs weren't designed to measure populations at all.

As we explained last week, the SATs were designed to measure the achievement of individuals. The program doesn’t make any attempt to test representative samples of the Grade 12 student population—not this year, not last year, not ten years ago.

Petrilli almost seems to understand this fact. As we noted yesterday, his blog post starts like this:
PETRILLI (9/3/15): The latest SAT scores are out today, and as I remarked to Nick Anderson at the Washington Post, education reform appears to be hitting a wall in high school.

In truth, we already knew this. The SATs aren’t even the best gauge—not all students take them, and those who do are hardly representative.
Question: What kind of expert reasons that way, in whatever field?

Petrilli understands that the students who take the SATs are “hardly representative” of the Grade 12 student population as a whole. On this basis, he makes a weird statement:

The SATs aren’t the best gauge of that population, our expert weirdly says.

Good God! Because they’re “hardly representative,” the tested students can’t safely be used as a gauge at all! Beyond that, Petrilli surely knows that the demographic blend of the tested students has been changing every year, in ways which tend to lower average scores and doom attempts at comparisons over time.

He knows that, but he doesn’t say so. What kind of “expert” does this?

A statistical complexity involving Grade 12 NAEP: Petrilli is also aware of a statistical complexity involving Grade 12 scores on the NAEP. As we noted yesterday, he explains this statistical problem in the passage shown below.

Never mind what he’s explaining here. His basic point is clear:
PETRILLI: One explanation could be America’s rising graduation rate. Students who would have previously dropped out are now staying in school and remaining in the NAEP sample, thereby dragging down the scores. That sounds plausible to me...
Below, we’ll look at the fuller passage, which we regard as horrific. That said, Petrilli seems to understand a possible statistical complexity which affects the utility of Grade 12 score comparisons over time. To wit:

As our national drop-out rate declines, lower-achieving students who once dropped out are staying in school through Grade 12.

Educationally, that’s a positive trend. But over time, the lower drop-out rate probably tends to “drag down [average NAEP] scores.”

“That sounds plausible to me,” Petrilli says. Below, we’ll marvel at what he says next.

Simpson’s Paradox: Petrilli is even aware of the role played by “Simpson’s Paradox.” This affects analysis of test scores at all grade levels, not just in Grade 12:
PETRILLI: Or maybe it’s Simpson’s Paradox at work. That would suggest that all racial groups are doing better, but because lower-scoring Latinos are replacing whites over time, our overall scores are declining.
Simpson’s Paradox refers to a counterintuitive state of affairs. Within a given population, every group can improve its average performance over time—but the overall average performance may remain unchanged, or even go down.

In the realm of test scores, this will happen if lower-scoring groups constitute a larger portion of the overall group over time, as has been the case with American public school testing.

This helps explain why average SAT scores have dropped in recent years. Petrilli understands this obvious fact, but he didn’t mention it in his blog post.

Petrilli understands these things! At the same time, he’s talking about a nation whose Grade 12 NAEP scores have been on the rise in the most recent period available for review.

According to the NAEP, high school seniors have been doing better in math! As we’ve shown you in the past two days, these are the actual score gains:
Gains in average scores, 2005-2013
Main NAEP, Grade 12 math
National public schools

White students: 4.32 points
Black students: 5.24 points
Hispanic students: 7.67 points
Asian-American students: 11.08 points
American Indian/Native Alaskan students: 9.48 points
Judged by normal measures, those score gains in math are substantial. And uh-oh:

As we’ll show you tomorrow, those score gains are actually larger than the score gains in Grade 4 over the same eight-year period, the most recent period for which we have NAEP data.

(The largest score gains occurred in Grade 8. All data tomorrow.)

The Grade 12 score gains are actually larger than the gains in Grade 4! And yet, Petrilli writes the following, under a headline which declares that high school achievement is flat:
PETRILLI: ...NAEP shows respectable gains for younger students, especially in fourth grade and particularly in math. Yet these early gains seem to evaporate as kids get older.
How did our educational expert come up with that highlighted claim? In his blog post, he made two unfortunate plays, each of which helped him reach the gloomy conclusion which is currently “hot.”

First, Petrilli jumped to a different study! In discussing NAEP scores from the fourth and twelfth grades, he has been talking about the so-called “Main NAEP” study, which tests students in Grades 4, 8 and 12.

For the record, it’s clear that that is the study Petrilli has been discussing. Early on, he offers this gloomy claim:

“Twelfth-grade NAEP: Flat.”

As we noted yesterday, Petrilli doesn’t link to any NAEP scores when he offers this assessment. Instead, he links to a news report by a “staff writer” for the Christian Science Monitor—a largely bungled report about the release of score from the 2013 “Main NAEP.”

Plainly, Petrilli has been discussing scores from the Main NAEP. But uh-oh! Grade 12 math scores on the Main NAEP actually haven’t been flat.

In math, the Grade 12 score gains seem substantial; they’re larger than the gains at Grade 4! That said, how did Petrilli support the claim we’ve posted above?

By switching to a different study! This is his fuller passage:
PETRILLI: ...NAEP shows respectable gains for younger students, especially in fourth grade and particularly in math. Yet these early gains seem to evaporate as kids get older.

Here’s what that looks like using data from the long-term trend NAEP for three recent student cohorts. Progress at ages nine and thirteen hasn’t translated into progress at age seventeen.
In that passage, Petrilli switches—without saying so—from the “Main NAEP” to a different NAEP study, the so-called “Long-Term Trend Assessment.”

The Long-Term Trend Assessment tests 9-year-old students, 13-year-old students and 17-year-old students, without regard to what grade they’re in. (Warning! Some 17-year-old students will be sophomores or juniors.) It uses a different math test than the one employed in the “Main NAEP.”

The Long-Term Trend Assessment is a different, parallel study. That doesn’t mean that it can’t be consulted in a wide array of ways. But it doesn’t specifically test high school seniors, as the Main NAEP specifically does. And for today, we’ll ask you to notice this:

When Petrilli switches to the Long-Term Trend Assessment, he looks at changes in scores over an 18-year period. He looks at score gains from 1994 through 2012, the last year for which data are available.

It’s true! Over that 18-year span, score gains were substantially larger among the two sets of younger students than among the 17-year-old students. Here’s the obvious problem:

Eighteen years is a fairly long time! To what extent might changing drop-out rates have “dragged down” average scores among 17-year-old students during that lengthy period?

For ourselves, we have no idea—and Petrilli doesn’t seem to care! Below, you see the fuller passage in which he describes the possible effect on average scores of changing drop-out rates.

We’ll ask one question at this time. What kind of “expert,” in any field, would ever reason like this:
PETRILLI: One explanation could be America’s rising graduation rate. Students who would have previously dropped out are now staying in school and remaining in the NAEP sample, thereby dragging down the scores. That sounds plausible to me, but to my knowledge, nobody has proved it empirically. Budding education policy scholars out there: Who is game to tackle that methodological challenge?
What kind of “expert,” in any field, would ever reason like that? Here’s what Petrilli does, and fails to do, in that ridiculous passage:

Did our declining drop-out rate “drag down” average scores at the 17-year-old level over that 18-year time span? To Petrilli, that “sounds plausible.”

Having said that, he also says this—to his knowledge, no one has proven that this occurred.

Of course, this seems to mean that no one has proven that it didn’t occur! In other words, Michael Petrilli doesn’t know if the smaller gains among the 17-year-old students, over that lengthy time span, resulted from that “plausible” cause.

Petrilli doesn’t know if that happened; we can’t tell you either. That said, we didn’t run to the Washington Post and use those cherry-picked data to say, in Best Approved Elite Reform Fashion, that progress is stagnant at the Grade 12 level—even as the more recent math scores from the Main NAEP seem to show substantial progress occurring in Grade 12.

Petrilli’s blog post is horrific. How absurd does it get?

At one point, its author—an educational expert—says the SAT is no good for the task at hand. So he tells us to consider the ACT instead!

What kind of “expert,” in any field, would ever produce such ludicrous work? We’ll try to answer that question tomorrow.

We’ll also show you the most recent data from all three grades—Grades 4, 8 and 12—tested in the Main NAEP. The Grade 12 score gains seem substantial—and they’re larger than those at Grade 4!

How would our “expert” explain such a thing? Would he just keep churning script?

Tomorrow: A deeply important disclosure

28 comments:

  1. I think today's post answers the questions raised by KZ yesterday, except for his absurd complaint about minor changes in Somerby's wording over time (excessively literal again, KZ).

    You can complain about the rough "rule of thumb" but here Petrilli calls the same size gains substantial at one grade level and flat at the other. You don't need to worry about how those score gains related to improvement to see that Petrilli is playing games to advance his narrative.

    You also don't have to think about education and test scores at all to see what KZ's narrative is here. That he is now arguing in service of conservative attacks on the achievements of teachers and their students suggests he is another conservative troll, like Cicero, David in CA and not a few anonymous commenters (perhaps redundant).

    ReplyDelete
    Replies
    1. No, my dear friend @ 1:17, today's post does not answer the questions we raised last night.

      Our first question was, "Has anyone BOB has quoted in this series said American high school seniors are doing less well in math?"

      If you think this post answers that question, we sadly must disagree.

      That said we are glad to see Bob ask almost the identical question to start off today's post as he asked yesteday:

      Today: "Are seniors in American high schools doing less well in math?"

      We will repeat our question exactly as written:

      "Has anyone BOB has quoted in this series said American high school seniors are doing less well in math?"

      Nobody bothered to answer that yesterday. BOB certainly added nothing in this post to answer it today.

      So we challenge you directly @ 1:17. Go thorough Bob's series and the articles to which he links. Find someone who said Amercian seniors are doing less well in math.

      While you are looking for the person who suggested seniors are doing less well in math, look for any scores Bob has put forward to answer this question: Are our seniors doing less poorly in reading?

      Finally, you have suggested a narrative for us that says we serve someone else's cause by noting your OTB is a phony. We are not the one who routinely denigrates entire professions, persons of ideological viewpoints or belittles people based on their age. That would be your BOB. We think HE is stupid to do that. We think YOU are stupid to follow. That implies nothing about us or our views about anyone else. It implies nothing about what we seem to feel about BOB and his believers. You don't have to read anything into it. It doesn't seem to be our view. It is our view. Literally.

      Meanwhile we continue to work on Part 2. We hope to answer the question we raised about Bob's constantly evolving questions in Part 1 soon.

      KZ of Doom

      Delete
    2. What do you think it means when an expert says gains in early years are not maintained and expresses concern that reforms are not working? Your excessive literalism is a drag.

      Delete
    3. It means gains in early years are not maintained. It means reforms are not working.

      Do you think that because we answered your question and you won't answer ours means we are saying you are doing less well in answering? Or do you think we literally mean your ability to defend BOB has stagnated?

      KZ

      Delete
    4. KZ, you are dealing with a person fluent in Bobspeak, where words don't mean what they say and say what they mean until interpreted for us by a true Bobfan.

      Until told what Bob really, really meant as opposed to what he actually wrote, we stand accused of "excessive literalism."

      Hence, even Bob and his loyal fans stand confused by even the simplest declarative sentences which quite often to them sound like they were translated from the original Norwegian by native speakers of Urdu.

      Delete
  2. What kind of experts reason this way? Conservative experts.

    ReplyDelete
  3. I don't agree that Cicero and David in CA are trolls. They just disagree with us.

    ReplyDelete
    Replies
    1. I just disagree with you.

      Delete
    2. They are trolls because they are never influenced by anything others say, even when they are proven factually incorrect.

      Delete
    3. cicero is a known paid troll. You naivete is refreshing though.

      Delete
    4. OK, let's compromise. Cicero is a troll; David is sincere but misguided.

      Delete
    5. David is a troll too. He keeps repeating the same tired conservative talking points with no evidence he heard anything said to him. They seem to be generated by the noise machine, not everyday thoughts or concerns of a conservative person.

      Delete
    6. cicero's blowup dollSeptember 17, 2015 at 10:56 PM

      Here's poor, misguided David's latest sincere, misguided support of now-proven liar Carly Fiorina:

      "The are times when one has to trust other people. This isn't one of them. One source supports Fiorina, saying,
      As for Fiorina’s quote, she is likely referring to the entirety of the 10 videos, including the seventh video released by the Center for Medical Progress. Watch the full video for yourself. It does, in fact, show a fully formed fetus, heart beating and legs kicking. And it shows this while Holly O’Donnell, a former organ harvester who worked for StemExpress at a Planned Parenthood affiliate, graphically discuss the harvesting of a brain from a baby whose heart was beating.
      http://thefederalist.com/2015/09/17/watch-the-video-planned-parenthood-and-its-media-allies-deny-exists/
      Bob could find out whether Fiorina's claim is true by watching the videos himself."

      Maybe you can straighten him out for us, Caesar.

      BWAHAHA!

      Delete
    7. blowup, is David responsible for the Federalist?

      Delete
  4. "What kind of “expert,” in any field, would ever reason like that?"

    I would say a very sound mind would withhold judgment on what "sounds plausible" until there is empirical evidence to support it. And Petrilli clearly says there is none, so the oft-repeated line that dumb kids who would have dropped out before are dragging down the test scores at age 17 remains unproven.

    And to give you a classic example. It once not only "sounded plausible" but it also "looked damned plausible" that the entire universe revolved around the earth.

    Then those pesky scientists with their new fangled telescopes but that whole danged notion to the test. And guess what? The once-plausible wasn't so plausible after all.

    ReplyDelete
    Replies
    1. And a further notion worth looking deeper at. Those high school drop outs hadn't dropped out in the fourth grade. So you actually had a stronger cohort of fourth graders taking the test when they became seniors.

      In other words, you now have all fourth graders and all former fourth graders eight years later. You don't have all fourth graders and all former fourth graders minus the dropouts eight years later.

      Which raises the question of why fourth grade gains are lost in the same cohort following eight years.

      Delete
    2. Regression to the mean.

      Delete
  5. "The SATs aren’t the best gauge of that population, our expert weirdly says.

    Good God! Because they’re “hardly representative,” the tested students can’t safely be used as a gauge at all!"

    Bob Somerby 9/17/15

    "SAT's are a limited measure of national achievement"

    Bob Somerby 9/1/2005

    Good God! In just ten years SAT went from limited measure to unsafe!

    ReplyDelete
    Replies
    1. Somerby is talking about national achievement. That is broader than academic performance or student achievement in mathematics. What exactly do you think "limited measure" means. Taken out of context, the phrase could even be understatement. But why are you still fixating on Somerby's writing style and ignoring the point in his essays?

      Delete
    2. No, you sad sack Somerby apologist. Somerby in 2005 was using SAT scores going up to argue there was not crisis in schools suggested by a high school teacher writing to the New York Times in response to a Bob Herbert column.

      Herbert used the term crisis because, in Herbert's words, "By the fourth grade, low-income students read about three grade levels behind non-poor students. Across the nation, only 15 percent of low-income fourth graders achieved proficiency in reading in 2003, compared to 41 percent of nonpoor students." A high school teacher wrote in to agree with Herbert that there was a crisis. She blamed the lack of literacy in homes.

      And you know what educational "expert critic" Bob Somerby used to refuted that concern? Did you guess SAT scores? You know, the test that "can't safely be used as a gauge at all."

      Rising SAT scores from a test given to self selected high schoolers. Wrote Somerby:

      "The teacher suggests that this “crisis” is growing as American homes get too many TVs. But as the Times reported just yesterday, national SAT scores hit an all-time high in math last year, and the verbal score has risen four points in the past decade. SATs are a limited measure of national achievement because only college-bound students take them. But did Herbert describe a generalized (and growing) “crisis?” The claim is hard to square with those numbers, despite what this highly-qualified teacher says.

      When they are up, (four pooints in a decade!) those SAT scores are useful for Bob Somerby. When they are down, (seven points in one year) only a charlatan doing the bidding of billionaires would mention them.

      Somerby taken out of context? No. Somerby displaying his basic character.

      Delete
    3. He stated the warning that they are a limited measure because of selection bias. Should he have started a discussion of NAEP scores? He refuted a broad claim of crisis with a broad and imprecise measure.

      You claim he argues out of two sides of his mouth because he uses evidence whose precision he later criticizes, but his point, that improvement is being ignored, is consistent.

      Petrilli is an education expert making a more technical argument. It is appropriate to hold him to a higher standard than a teacher making a sweeping, inaccurate claim.

      Delete
    4. We find ourselves chuckling at your response @ 11:29, not so much for what it includes, but for what it missed. We went back to the post in question because we were intrigued by the phrase used by BOB "as the Times reported just yesterday."

      Sure enough the OTB linked to a 2005 New York Times article on the release of annual SAT scores. We followed that link. You should too:

      http://www.nytimes.com/2005/08/31/education/sat-math-scores-at-record-high-but-those-on-the-verbal-exam-are-stagnant.html

      SAT Math Scores at Record High, but Those on the Verbal Exam Are Stagnant

      So said the NY Times headline!

      Did BOB devote a whole series to attacking that work a decade ago? No. But you are right about Bob's character.

      Bob was praising the big verbal score gains when they were actually characterized thusly:

      "verbal scores remained flat, well below their historic high, the College Board said yesterday.

      The score on the math test was 520, of a possible 800, up two points from the previous year. The verbal score was 508, the same as in the previous year. This score is the highest on the verbal test since 1986 but far below the record of 543 in 1967, when the College Board began keeping track, and again in 1968."

      In this post, and indeed in this series, you would think verbal skills were unimportant. Reading is not FUNdamental.

      In fact, reading has virtually disappeared. One might say, except for one question in one post, Bob Somerby would rather walk across the Bay Bridge than discuss reading scores.

      Which we plan to discuss in Part 2. We had planned it for today, but it seems BOB is extending the series to Friday. So we'll wait until then.

      KZ

      Delete
    5. Reading depends on childhood language experiences that happen (or don't) before school age. Reading is much harder to improve than math. This isn't Somerby's fault.

      Delete
    6. Yep. Based on that famous Kansas study that got shredded upon peer review for both its methodology and the broad conclusions it reached.

      Certainly it is important for parents who read to their infants and talk to them as their brains get wired and their language skills develop.

      But it is quite something else to say that "Reading DEPENDS (your word, emphasis added) on childhood language experiences that happen (or don't) before school age. . ."

      Quite a convenient way to pass the buck isn't it? And a pretty good excuse to give up on kids pretty quick, even "before school age" since the die is already cast.

      Delete
  6. My husband cheated on me for Almost three years. he ignore me for several months and left me with nothing, but i am happy today that Chief Nwaluta brought my husband back, I am so happy, Now my husband is all mine again. I can now say I'm happy again. Great spell from Chief Nwaluta, Chief Nwaluta is genuine. I truly believe in him and his spells. he is a professional. , My name is Sandra Gault and I live in Bryan Texas, My husband and i got married for more than 11 years and have gotten two kids. thing were going well with us and we are always happy. until one day my husband started to behave in a way i could not understand, i was very confused by the way he treat me and the kids. later that month he did not come home again and he called me that he want a divorce, i asked him what have i done wrong to deserve this from him, all he was saying is that he want a divorce that he hate me and do not want to see me again in his life, i was mad and also frustrated do not know what to do,i was sick for more than 2 weeks because of the divorce. i love him so much he was everything to me without him my life is incomplete. i told my sister and she told me to contact a spell caster, i never believe in all this spell casting of a thing. i just want to try if something will come out of it. i contacted Chief Nwaluta for the return of my husband to me, they told me that my husband have been taken by another woman, that she cast a spell on him that is why he hate me and also want us to divorce. then they told me that they have to cast a spell on him that will make him return to me and the kids, they casted the spell and after 1 week my husband called me and he told me that i should forgive him, he started to apologize on phone and said that he still love me that he did not know what happen to him that he left me. it was the spell that he Chief Nwaluta casted on him that make him come back to me today,me and my family are now happy again today. thank you Chief Nwaluta for what you have done for me i would have been nothing today if not for your great spell. i want you my friends who are passing through all this kind of love problem of getting back their husband, wife , or ex boyfriend and girlfriend to contact Chief Nwaluta,if you need his help you can contact him through his private E-mail:{ Nwalutaspelltemple@gmail.com } or you can contact him through his Web site :http://nwalutaspelltemple.blogspot.com. Thanks you Chief Nwaluta.

    ReplyDelete
  7. I am Hwa Jurong, a Private Money Lender do you need a loan to start up business or to pay your bills and a corporate financial for real estate and any kinds of business financing. I also offer Loans to individuals,Firms and corporate bodies at 2% interest rate. I give out loan to serious minded people that are interested of loan if interested contact this email: hwajurong382@yahoo.com or hwajurong12@gmail.com

    ReplyDelete
  8. i am so happy to share my testimony on how i was able to get back my ex lover around September 2015 with the help of Dr. Ofemo since 5 year ago, I almost gave up trying to get my ex lover back to commit to our relationship fully. I had tried the whole lot I knew, but what I DIDN'T know was that all I had to do was come to you Dr. Ofemo, and with your blessings and extraordinary magic powers, I wouldn't have to do all the work at all. you did your magic, and immediately after 24 hours, my lover came back to me with lots of apologies and flowers.. And now my life is balanced and happy again. Dr. Ofemo is a real and genuine spell caster he help me restore back my broken relationship. You do a great service to people, You are talented with spell to bring back ex lover and many more. Thank you for weaving your magic love spells for me. She is back just the way it was when we first met,his email address:drofemospelltempl@gmail.com it was all like a dream to me, i am so happy now as we are back together again thanks to Dr. Ofemo and i will advice anyone in need of help to contact him via email: drofemospelltemple@yahoo.com or his cell number +2348163387496 his web/site http://drofemospelltemple.wix.com/dr-ofemo

    ReplyDelete
  9. i am so happy to share my testimony on how i was able to get back my ex lover around September 2015 with the help of Dr. Ofemo since 5 year ago, I almost gave up trying to get my ex lover back to commit to our relationship fully. I had tried the whole lot I knew, but what I DIDN'T know was that all I had to do was come to you Dr. Ofemo, and with your blessings and extraordinary magic powers, I wouldn't have to do all the work at all. you did your magic, and immediately after 24 hours, my lover came back to me with lots of apologies and flowers.. And now my life is balanced and happy again. Dr. Ofemo is a real and genuine spell caster he help me restore back my broken relationship. You do a great service to people, You are talented with spell to bring back ex lover and many more. Thank you for weaving your magic love spells for me. She is back just the way it was when we first met,his email address:drofemospelltempl@gmail.com it was all like a dream to me, i am so happy now as we are back together again thanks to Dr. Ofemo and i will advice anyone in need of help to contact him via email: drofemospelltemple@yahoo.com or his cell number +2348163387496 his web/site http://drofemospelltemple.wix.com/dr-ofemo..

    ReplyDelete