OUTSCORING FINLAND: Muddling ahead!

TUESDAY, DECEMBER 3, 2013

Interlude—New PISA scores: Normally, we’re out of the house by the time Morning Joe starts.

This morning, we lingered. For that reason, we heard the news from the Morning Joe gang:

The new PISA test scores have been released, a group of shocked doomsayers told us.

Before the week is through, we’ll look at the various things this clueless gang said. For today, we’ll only say this:

None of the gang showed the slightest sign of knowing what they were talking about. That didn’t stop them from waxing at length—from waxing and rattling script.

Scarborough, Brzezinski, Meacham, Geist? They showed no sign of knowing squat from squadoosh when it came to the subject at hand.

But so what? The past few decades have been a “disaster” in our public schools, they cluelessly advised at one point. They seemed to know nothing about the facts—but they did know their standard scripts.

Later, we were also struck by the news report in today’s New York Times. We still haven’t examined the new PISA data. Just for today, let’s consider what we read in Motoko Rich’s report.

(On-line and in our hard-copy Times, no graphic accompanies this report.)

We’re often puzzled by Rich’s work. Hard-copy headline included, this is the way she started:
RICH (12/3/13): American 15-Year-Olds Lag, Mainly in Math, on International Standardized Tests

Fifteen-year-olds in the United States score in the middle of the developed world in reading and science while lagging in math, according to international standardized test results being released on Tuesday.

While the performance of American students who took the exams last year differed little from the performance of those tested in 2009, the last time the exams were administered, several comparable countries—including Ireland and Poland—pulled ahead this time.
The headline didn’t quite fit the copy, but the copy stated several key points. American students scored “scored in the middle of the developed world in reading and science,” did less well in math.

Their performance “differed little from the performance of those tested in 2009,” the last time the PISA was administered.

Those are very basic claims. Meanwhile, in what way is Ireland (population 6.4 million)a “comparable country” to the U.S.? These are the mysteries which sometimes appear in Times education reporting.

The Irish question is minor. As she continued, Rich reported a few more basic facts about the standing of American students on these tests from last year:
RICH: The Program for International Student Assessment, commonly known as PISA, was administered to 15-year-olds in 65 countries and school systems by the Organization for Economic Cooperation and Development, a Paris-based group that includes the world’s wealthiest nations. Just over 6,100 American students took the exams.

[...]

Some scholars warned that the lagging performance of American students would eventually lead to economic torpor. “Our economy has still been strong because we have a very good economic system that is able to overcome the deficiencies of our education system,” said Eric A. Hanushek, an economist at Stanford University. “But increasingly, we have to rely on the skills of our work force, and if we don’t improve that, we’re going to be slipping.”

The United States’ underperformance was particularly striking in math, where 29 countries or education systems had higher test scores. In science, students in 22 countries did better than Americans, and in reading, 19 countries.
As she started her report, Rich had said that the U.S. was “lagging” in math alone. Now, she seemed to say that the U.S. was “lagging” in general.

Whatever! In this passage, Rich seems to define where the U.S. stood in relationship to other countries and “school systems” (or “education systems”) which took the PISA tests:

According to Rich, 65 “countries and school systems” took part in the testing. This seems to be where the U.S. stands:
U.S. standing in the world, 2012 PISA
In reading: 20th out of 65
In math: 30th out of 65
In science: 23rd out of 65
By that reckoning, our standing is somewhat higher in reading and science. In our worst performance, we’re just above the middle in math.

According to Rich’s report, the U.S. seems to be in the top half in all three subjects. That’s why we were puzzled when she tried to offer a bit of context.

To her credit, Rich cited results from the 2011 TIMSS. That said, we were bollixed by her assessment:
RICH (continuing directly): The results painted a slightly different picture from tests administered to fourth and eighth graders in 2011 through the Trends in International Mathematics and Science Study. Those results indicated that the United States was about on par with international averages.
To her credit, Rich attempted to use the 2011 TIMSS to offer context and contrast. But do you understand what she said?

According to Rich, the new PISA results “painted a slightly different picture” from the results on the TIMSS. But riddle us this:

Based on what Rich has written, did American students do slightly better on the TIMSS? Or did they do slightly worse?

(Note: The U.S. is commonly said to do better on the TIMSS.)

According to Rich, “the United States was about on par with international averages” on the TIMSS. Hadn’t she just painted that same portrait with respect to the PISA?

Truth to tell, Rich is a bit stingy in her description of those TIMSS results. On the TIMSS scale, 500 is set as the international average, with a standard deviation of 100. On all four measures in 2011, the U.S. exceeded that average:
Average scores, American students, 2011 TIMSS
Grade 4 math: 541
Grade 8 math: 509
Grade 4 science: 544
Grade 8 science: 525
Question: In what way would those results differ from the rankings Rich described from the 2012 PISA?

On the TIMSS, the U.S. scored above the international average on all four measures. On the PISA, Rich says the U.S. scored in the top half on all three measures.

In what way is the new performance different? From what Rich wrote, are we supposed to think the U.S. did slightly better on the PISA? Or did we do slightly worse?

Go ahead—puzzle it out. Do you have any idea?

We’re constantly puzzled by Rich’s reporting, especially given her background. She graduated summa cum laude from Yale in the early 1990s.

Today, she writes for our greatest newspaper—and her work is persistently jumbled, unclear.

Perhaps her editor jumbles things up, but her reports are persistently fuzzy. In our view, her work is a striking sign of the times, of our post-journalistic culture.

Still coming: TIMSS results, disaggregated

Regarding Finland: In paragraph 3, Rich dropped a bomb concerning miraculous Finland:
RICH: As in previous years, the scores of students in Shanghai, Hong Kong, Singapore, Japan and South Korea put those school systems at the top of the rankings for math, science and reading. Finland, a darling of educators, slid in all subjects but continued to outperform the averages, and the United States.
We appreciate the touch of snark concerning that twelve-year love affair. That said, by how much did Finland slide? We haven’t looked at the data ourselves. Was the slide worth noting?

Note one more point of confusion:

Finland “outperformed the averages?” At a glance, a reader might reasonably assume that the U.S. outperformed the averages too!

No 800-word news report can settle every factual question about a topic like this. But we’re often puzzled by Rich’s reporting. Given her background and her high station, we’d call it a sign of the times.

8 comments:

  1. OMB

    When you disaggregate Coach BOB, will you please compare how the SEC states did versus the PAC-12?

    KZ

    ReplyDelete
    Replies
    1. is it any wonder I regard KZ as sharing my genius ... so, following his lead and keeping to whats important, I ask how can this stand: "the Irish question is minor" ... could Somerby make his hatred any plainer?

      Delete
    2. tamo&#etc.

      You would think old BOB, being totally obsessed with test scores, might figure that what was comparable about Poland, Ireland, and the U.S. was their test scores on the last round of the tests which are the subject of article.

      We are constantly puzzled by BOB's reporting, especially given his background. He graduated from some school in Boston back when Ho wasn't just a rap put down.

      KZ



      Perhaps reading the Salone article.




      Delete
    3. Hatred? For what -- Irish or minors?

      Delete
  2. OMB (Unlike the SEC, there are no patsies for the USofA to play)

    "Finland “outperformed the averages?” At a glance, a reader might reasonably assume that the U.S. outperformed the averages too! "

    Sadly, only the average BOB reader might reasonably assume that. To the actual reader of real PISA results the US was below the OECD average on two of the three tests. And this testing season we were skunked by Poland and Ireland. Zip outta six.

    I can't wait for BOB to invent a new stat to explain that over the last sumpty-sumpin years, U.S. average score gains combined show the Irish to be nothing more than a bunch of dumb Polacks.

    Geaux Tigers! (Asiatic variety, not the ones from the Cudzu Conference.)

    KZ

    ReplyDelete
    Replies
    1. KZ, Bob was clearly referring to the confusion created by the article, he was not trying to imply that the US was in fact above average on the 2012 PISA. In fact it would be rather easy to to assume that the US outperformed the average because of the unclear way in which the article presents the data (i.e. ordinal rankings). Does the article mention OECD averages anywhere? I don't believe it does, which is an oversight on the part of the article's author, not Bob's.

      Delete
  3. One sign of weak reporting in Ms. Rich's articles that you did not mention was the two magic words used by reporters when they don't actually have facts: the adverb is "increasingly" and the adjective is "growing", as in:

    In the midst of increasingly polarized discussions about...
    An increasingly vocal group of parents, teachers, union leaders and others have also objected ...
    In the midst of increasingly polarized discussions about...
    A growing chorus of critics have said that...
    Amid growing alarm over ...

    These phrases are a way of putting an opinion in a news article without citing the source of the opinion (usually the author, her/his self)

    ReplyDelete
    Replies
    1. There was a nice quote in the original article that said all these international test scores are pretty meaningless and unimportant to discussions about the state of US education.

      Delete