OUTSCORING FINLAND: Muddling ahead!

TUESDAY, DECEMBER 3, 2013

Interlude—New PISA scores: Normally, we’re out of the house by the time Morning Joe starts.

This morning, we lingered. For that reason, we heard the news from the Morning Joe gang:

The new PISA test scores have been released, a group of shocked doomsayers told us.

Before the week is through, we’ll look at the various things this clueless gang said. For today, we’ll only say this:

None of the gang showed the slightest sign of knowing what they were talking about. That didn’t stop them from waxing at length—from waxing and rattling script.

Scarborough, Brzezinski, Meacham, Geist? They showed no sign of knowing squat from squadoosh when it came to the subject at hand.

But so what? The past few decades have been a “disaster” in our public schools, they cluelessly advised at one point. They seemed to know nothing about the facts—but they did know their standard scripts.

Later, we were also struck by the news report in today’s New York Times. We still haven’t examined the new PISA data. Just for today, let’s consider what we read in Motoko Rich’s report.

(On-line and in our hard-copy Times, no graphic accompanies this report.)

We’re often puzzled by Rich’s work. Hard-copy headline included, this is the way she started:

RICH (12/3/13): American 15-Year-Olds Lag, Mainly in Math, on International Standardized Tests

Fifteen-year-olds in the United States score in the middle of the developed world in reading and science while lagging in math, according to international standardized test results being released on Tuesday.

While the performance of American students who took the exams last year differed little from the performance of those tested in 2009, the last time the exams were administered, several comparable countries—including Ireland and Poland—pulled ahead this time.

The headline didn’t quite fit the copy, but the copy stated several key points. American students scored “scored in the middle of the developed world in reading and science,” did less well in math.

Their performance “differed little from the performance of those tested in 2009,” the last time the PISA was administered.

Those are very basic claims. Meanwhile, in what way is Ireland (population 6.4 million)a “comparable country” to the U.S.? These are the mysteries which sometimes appear in Times education reporting.

The Irish question is minor. As she continued, Rich reported a few more basic facts about the standing of American students on these tests from last year:

RICH: The Program for International Student Assessment, commonly known as PISA, was administered to 15-year-olds in 65 countries and school systems by the Organization for Economic Cooperation and Development, a Paris-based group that includes the world’s wealthiest nations. Just over 6,100 American students took the exams.

[...]

Some scholars warned that the lagging performance of American students would eventually lead to economic torpor. “Our economy has still been strong because we have a very good economic system that is able to overcome the deficiencies of our education system,” said Eric A. Hanushek, an economist at Stanford University. “But increasingly, we have to rely on the skills of our work force, and if we don’t improve that, we’re going to be slipping.”

The United States’ underperformance was particularly striking in math, where 29 countries or education systems had higher test scores. In science, students in 22 countries did better than Americans, and in reading, 19 countries.

As she started her report, Rich had said that the U.S. was “lagging” in math alone. Now, she seemed to say that the U.S. was “lagging” in general.

Whatever! In this passage, Rich seems to define where the U.S. stood in relationship to other countries and “school systems” (or “education systems”) which took the PISA tests:

According to Rich, 65 “countries and school systems” took part in the testing. This seems to be where the U.S. stands:

U.S. standing in the world, 2012 PISA
In reading: 20th out of 65
In math: 30th out of 65
In science: 23rd out of 65

By that reckoning, our standing is somewhat higher in reading and science. In our worst performance, we’re just above the middle in math.

According to Rich’s report, the U.S. seems to be in the top half in all three subjects. That’s why we were puzzled when she tried to offer a bit of context.

To her credit, Rich cited results from the 2011 TIMSS. That said, we were bollixed by her assessment:

RICH (continuing directly): The results painted a slightly different picture from tests administered to fourth and eighth graders in 2011 through the Trends in International Mathematics and Science Study. Those results indicated that the United States was about on par with international averages.

To her credit, Rich attempted to use the 2011 TIMSS to offer context and contrast. But do you understand what she said?

According to Rich, the new PISA results “painted a slightly different picture” from the results on the TIMSS. But riddle us this:

Based on what Rich has written, did American students do slightly better on the TIMSS? Or did they do slightly worse?

(Note: The U.S. is commonly said to do better on the TIMSS.)

According to Rich, “the United States was about on par with international averages” on the TIMSS. Hadn’t she just painted that same portrait with respect to the PISA?

Truth to tell, Rich is a bit stingy in her description of those TIMSS results. On the TIMSS scale, 500 is set as the international average, with a standard deviation of 100. On all four measures in 2011, the U.S. exceeded that average:

Average scores, American students, 2011 TIMSS
Grade 4 math: 541
Grade 8 math: 509
Grade 4 science: 544
Grade 8 science: 525

Question: In what way would those results differ from the rankings Rich described from the 2012 PISA?

On the TIMSS, the U.S. scored above the international average on all four measures. On the PISA, Rich says the U.S. scored in the top half on all three measures.

In what way is the new performance different? From what Rich wrote, are we supposed to think the U.S. did slightly better on the PISA? Or did we do slightly worse?

Go ahead—puzzle it out. Do you have any idea?

We’re constantly puzzled by Rich’s reporting, especially given her background. She graduated summa cum laude from Yale in the early 1990s.

Today, she writes for our greatest newspaper—and her work is persistently jumbled, unclear.

Perhaps her editor jumbles things up, but her reports are persistently fuzzy. In our view, her work is a striking sign of the times, of our post-journalistic culture.

Still coming: TIMSS results, disaggregated

Regarding Finland: In paragraph 3, Rich dropped a bomb concerning miraculous Finland:

RICH: As in previous years, the scores of students in Shanghai, Hong Kong, Singapore, Japan and South Korea put those school systems at the top of the rankings for math, science and reading. Finland, a darling of educators, slid in all subjects but continued to outperform the averages, and the United States.

We appreciate the touch of snark concerning that twelve-year love affair. That said, by how much did Finland slide? We haven’t looked at the data ourselves. Was the slide worth noting?

Note one more point of confusion:

Finland “outperformed the averages?” At a glance, a reader might reasonably assume that the U.S. outperformed the averages too!

No 800-word news report can settle every factual question about a topic like this. But we’re often puzzled by Rich’s reporting. Given her background and her high station, we’d call it a sign of the times.