VISIONS OF PISA: All hail our new national Pisa Day!

MONDAY, DECEMBER 9, 2013

Part 1—From Friedman over to Ravitch: In this post-journalistic nation, the PISA is now a cult.

The PISA’s official status became clear just last week. In part, this occurred with the adoption of “PISA Day,” a national day of mourning and false remembrance which was widely observed across the land.

“In the U.S., the lackluster [PISA] results will be marked by great fanfare,” the Huffington Post reported. “On Tuesday, Andreas Schleicher, the OECD researcher who created the exam, will hand the results to Duncan, the education secretary, in a long, glitzy Newseum ceremony known as PISA Day.”

(In its report, the Huffington Post made a series of overstatements and misstatements drawn from the PISA’s official poop sheets. So it goes as the somewhat shaky PISA comes to rule the land.)

Needless to say, PISA Day had to be ratified at the Newseum! That said, the Huffington Post was hardly alone in its adoption of PISA perspectives. Across the press corps, pundit elites recited the scripts which emerged from this Paris-based cult.

Consider the way Thomas L. Friedman began his column in yesterday’s New York Times. We include the column’s title, which can almost be read as a bit of an irony:

FRIEDMAN (12/8/13): Can’t We Do Better?

The latest results in the Program for International Student Assessment, or PISA, which compare how well 15-year-olds in 65 cities and countries can apply math, science and reading skills to solve real-world problems were released last week, and it wasn’t pretty for the home team. Andreas Schleicher, who manages PISA, told the Department of Education: “Three years ago, I came here with a special report benchmarking the U.S. against some of the best performing and rapidly improving education systems. Most of them have pulled further ahead, whether it is Brazil that advanced from the bottom, Germany and Poland that moved from adequate to good, or Shanghai and Singapore that moved from good to great. The math results of top-performer Shanghai are now two-and-a-half school years ahead even of those in Massachusetts—itself a leader within the U.S.”

Three years ago, our Dearest Leader “came here with a special report!”

Friedman’s opening paragraph ran 137 words. Eighty-two of those words were a long quotation from the rather self-confident Chairman Andreas, widely loved leader of this newest cult.

We have never seen any sign that Friedman knows much about public schools. (There’s no particular reason why he should.) He does know who he’s expected to quote, and revere, in his high-profile columns.

In yesterday’s column, Friedman quoted Andreas the Giant at remarkable length. In the following passage, he was even willing to quote Dear Leader as he made one of his more obscure pronouncements—and as he served some genuine marshmallow fluff:

FRIEDMAN: So now let’s look at the latest PISA. It found that the most successful students are those who feel real “ownership” of their education. In all the best performing school systems, said Schleicher, “students feel they personally can make a difference in their own outcomes and that education will make a difference for their future.” The PISA research, said Schleicher, also shows that “students whose parents have high expectations for them tend to have more perseverance, greater intrinsic motivation to learn.” The highest performing PISA schools, he added, all have “ownership” cultures—a high degree of professional autonomy for teachers in the classrooms, where teachers get to participate in shaping standards and curriculum and have ample time for continuous professional development. So teaching is not treated as an industry where teachers just spew out and implement the ideas of others, but rather is “a profession where teachers have ownership of their practice and standards, and hold each other accountable,” said Schleicher.

Let’s see if we were able to follow those varied pronouncements:

If we’re reading that passage correctly, Dear Leader tells us that parents who have high expectations tend to get better results from their kids! Also, students who feel they can make a difference will end up making a difference!

Or something like that. After making these pronouncements, Schleicher swam the Rhine.

As for Friedman, he also quoted Schleicher decreeing that teachers should “have ownership of their practice and standards.” Do you know what that pronouncement meant? Neither does anyone else who read Friedman’s column!

Whatever! All through the column, Friedman keeps quoting Dear Leader as he makes fuzzy or pointless pronouncements. In a column of 885 words, we count at least 268 words which come directly from the mouth of Chairman Andreas.

We’re not sure if we’ve ever seen a column so devoted to quoting the words of one person. That said, Friedman adopts some of the basic approaches of the new cult all by himself.

Most notably—and it ought to be very notable—most notably, Friedman cites the results from the 2012 PISA. But he never so much as mentions the results from the 2011 TIMSS or the 2013 NAEP.

That decision should be very notable.

Has Friedman ever heard of the TIMSS or the NAEP? We wouldn’t place money on that. But what happens when unschooled pundits disappear the TIMSS and the NAEP? A one-sided portrait appears!

In many ways, the NAEP may be our most reliable national testing program—more reliable than the PISA or the TIMSS. And uh-oh! Especially after disaggregation, NAEP scores seem to show substantial academic progress over the course of the past several decades.

TIMSS scores seem to suggest the same general pattern. But so what? As people like Friedman restrict themselves to pronouncements from the new cult, two-thirds of our data get thrown down the well! In the process, we are left in the hands of our newest Dear Leader.

All week long, we’ll examine the ways the cult of the PISA ruled the press corps last week. Here’s the most intriguing point:

We liberals now have our own standard reactions to the scripts which drive the mainstream press concerning international tests. But even in our own standard scripting, we tend to bow low to the PISA!

As a general matter, Diane Ravitch stands in opposition to the cult of the PISA. If we liberals have our own leader in this area, it is clearly Ravitch.

Ravitch made some perfectly decent points in her responses to PISA Day. She also adopted some perspectives which strike us as extremely unhelpful, even borderline uncaring and cruel.

Beyond that, she continued to encourage some claims which are flatly bogus.

Tomorrow, we’ll consider the way Ravitch adopted one standard PISA framework in response to the new scores. Some of her reactions were right on target.

Other reactions were not.

Tomorrow: Can anyone here play this game?

Advanced students may want to read ahead: For Ravitch’s basic reactions, see this blog post, My View of the PISA Scores.

For advanced writing students, undertake this assignment. Rewrite Visions of Johanna using this revised first line:

"Ain’t it just like the PISA to play tricks when we’re trying to be so quiet?"

WHERE THE TEST SCORES ARE: Bay State kids kick Finland's asp!

FRIDAY, SEPTEMBER 9, 2016

Part 3 in this series

Part 4—Journalists, please go to Boston: Last Saturday morning, a viewer in Mississippi placed a phone call to C-Span.

He made a familiar presentation. As so many others have done, he praised the fabulous public schools of the miracle nation, Finland. This Tuesday, in part 1 of this series, we transcribed that call.

The caller identified himself as "a William F. Buckley conservative." In truth, he seemed a bit defensive about his praise for a northern European welfare state. But, he told his "conservative brethren," "I'm looking over the horizon. You know we go with what works."

His statement was very familiar. If we've heard that phone call once, we've heard that that phone call a million times. Over the past dozen years, praise for Finland's miracle schools has become remarkably common across the ideological spectrum.

In her piece for Smithsonian Magazine in 2011, LynNell Hancock explained how this stampede began. We'll present that short passage again:

HANCOCK (9/11): The transformation of the Finns’ education system began some 40 years ago as the key propellent of the country’s economic recovery plan. Educators had little idea it was so successful until 2000, when the first results from the Programme for International Student Assessment (PISA), a standardized test given to 15-year-olds in more than 40 global venues, revealed Finnish youth to be the best young readers in the world. Three years later, they led in math. By 2006, Finland was first out of 57 countries (and a few cities) in science. In the 2009 PISA scores released last year, the nation came in second in science, third in reading and sixth in math among nearly half a million students worldwide.

Hancock's history is basically accurate. Results from the inaugural PISA "revealed Finnish youth to be the best young readers in the world," or something very much like that.

Finland chic started there. American journalists began flocking to Finland to determine how the small Euro nation had done it. By now, belief in the greatness of Finland's schools has spread remarkably widely. People from all sides of every spectrum can be found praising Finland's schools, often picking-and-choosing the factors which, they say, must explain the nation's great test scores.

In truth, it never made obvious sense for reporters to fly off to Finland.

As far as we know, the Finns run excellent schools. (This is a point which Finnish officials are always prepared to make.) That said, Finland is a small, unicultural, middle-class nation. Demographic issues which may complicate schooling in other nations play very little role there.

To its credit, Finland has little poverty. As of the year 2000, it had permitted virtually no immigration, let alone to immigrants from low-literacy backgrounds.

To its credit, Finland didn't spend centuries trying to eliminate literacy from one segment of its population, as our own benighted ancestors attempted to do. Situations which may complicate schooling Over Here were virtually non-existent Over There when Finnish kids seemed to emerge as (being among) "the best young readers in the world."

It never exactly made sense to fly to Finland seeking answers to types of questions which didn't exist Over There. That said, our journalists have been conducting that hajj for the fifteen years.

Their upbeat profiles are so standardized that they practically write themselves. This June, The Atlantic published a similar piece, a piece which asked if Estonia—a nation which is tiny, not small—might be "the new Finland."

Our journalists went where the test scores were! (The tabs for their expensive trips were sometimes paid by interest groups, a point we'll discuss in the weeks ahead.) In the process, a problem has occurred:

They've managed to overlook miracle scores which are found a lot closer to home.

According to Hancock, Finland's kids emerged as the world's best readers on the 2000 PISA. According to The Atlantic's report, Estonia emerged as the possible new Finland on the 2012 PISA.

(The PISA is administered every three years. Last year's results aren't available yet.)

Those two small nations did in fact score well on the PISA. But so did kids in a jurisdiction a great deal closer to home.

Good news! The PISA allows sub-national jurisdictions to participate in the testing as independent entities. Three different American states did so in 2012, testing a large enough sample of students to produce statistically meaningful test scores.

Massachusetts, population 6.8 million, was one of those states. It's a small corner of North America, but it's larger than Finland, population 5.5 million. It's much larger than tiny Estonia, population 1.3 million.

Are Finland's kids the world's best readers? Could Estonia be the new Finland? Apparently, Massachusetts is the new Finland too! Here are the average reading scores from the 2012 PISA:

Average scores, reading, 2012 PISA
Estonia: 516.29
Finland: 524.02
Massachusetts: 527.09

Given the way the PISA scale works, the differences between those average scores aren't all that significant. But three years after those scores were released, Americans have never heard about the miracle schools of Massachusetts—and journalists are still being sent to examine the wonders of schools in two far-away lands.

Massachusetts took the PISA math test that year too. Again, there is little significant difference between these average scores:

Average scores, math, 2012 PISA
Estonia: 520.55
Finland: 518.75
Massachusetts: 513.51

Given the way the PISA scale works, there's little difference between those scores. But for some reason, our journalists will sometimes fly from Logan Airport to gaze on the wonders of Finland.

We're looking here at the PISA, one of our two major international testing programs. The other major international program is the combination of the TIMSS and the PIRLS, whose most recent results date from 2011.

The world's developed nations take part in the PISA and the TIMSS/PIRLS. But for reasons which sometimes seem all too clear, journalists funded by certain interests only discuss the PISA.

As Hancock correctly reported, Finland gained international prominence due to its performance, down through the years, on the PISA. In 2011, the miracle nation took part in the TIMSS for the first time.

(Generally speaking, the PISA is said to stress "critical thinking." The TIMSS is said to stress knowledge of curriculum. The TIMSS tests kids in fourth and eighth grades. The PISA tests 15-year-olds.)

Which testing program is more worthwhile, the PISA or the TIMSS? That is a matter of judgment. That said, the major developed nations, including the United States, routinely participate in both programs. Here's what happened when Finland took the TIMSS, which tests kids in science and math:

Average scores, Grade 8 math, 2011 TIMSS
Estonia: did not participate
Finland: 514.03
Massachusetts: 560.09

Given the way the TIMSS scale works, that is a rather large difference. Massachusetts ate Finland's lunch on the TIMSS math test.

Massachusetts took part as an independent entity in the TIMSS science test too. Once again, it outscored a miracle nation:

Average scores, Grade 8 science, 2011 TIMSS
Estonia: did not participate
Finland: 552.35
Massachusetts: 566.78

The PIRLS tests fourth-graders in reading. Massachusetts didn't take part in the 2011 PIRLS as an independent entity. Florida, the one state which did, outscored Finland by exactly one point.

For today, we'll leave our comparisons here. As our report continues in the next few weeks, we'll be looking at more international scores, and at domestic scores too.

We think the data we've shown you today are enough to raise an obvious question. Our question concerns the way our national news orgs treat that C-Span viewer.

Like citizens all over the country, he's possibly heard it again and again. Again and again, for the past dozen years, we've been told, again and again, about Finland's miracle schools.

Tiny Estonia may be "the new Finland," we have also now been told. Three years ago, in a widely-praised book, Amanda Ripley said the same thing about Poland. She seemed to pick and choose her data to advance this mandated claim.

The story is pleasing; it never gets old. Inevitably, it features an invidious comparison to the lousy, pitiful schools our own hapless teachers are running.

Frankly, readers, how strange! We're constantly told that Finland—a small, unicultural corner of Europe—is running miracle schools. But how strange! Five years later, that C-Span viewer in Mississippi has never heard a similar piece of news—a piece of news about a slightly larger corner of North America:

Five years ago, Massachusetts cleaned Finland's clock on the TIMSS math tests. One year later, it narrowly outscored Finland in reading on the PISA, was narrowly outscored in math.

That said, how strange! When Finland scored well on the 2000 PISA, its kids became "the best readers in the world." But how strange! On the 2012 PISA, Bay State kids outscored Finland in reading. To this day, no one has heard that any such thing occurred!

"Please come to Boston," Dave Loggins sang, way back in 1974. You're supposed to think it's a sappy song. To this day, we think it's fascinating, though that's neither here nor there.

"Please come to Boston," the singer pleads. For unexplained reasons, his girl friend says no. So it has gone, for many years, as a bit of a journalistic con is handed to us the people.

Journalists fly to the ends of the earth looking for those miracle schools and the secrets they surely contain. For the price of a ticket on Amtrak, they could please go to Boston instead—or to Needham, Leominster, Worcestor, Dennisport, Fall River, Ipswich, Woburn.

Instead, they insist on going abroad, to small corners of Europe. We have a larger, high-scoring corner Right Here, and admit it—you've never been told!

Eventually, we'll ask a question about this peculiar state of affairs. Why have our big news orgs so persistently done this?
Why don't they simply get on the train and take themselves where the test scores are? Why can't they be decent enough to admit that those test scores exist?

Next week: Where the NAEP scores are

More information: Nine states took part as independent entities in the Grade 8 TIMSS math test in 2011. Six of those states outscored Finland. Florida came quite close.

VISIONS OF PISA: Weingarten, Ravitch lose the NAEP!

THURSDAY, DECEMBER 12, 2013

Part 4—The liberal world’s rolling incompetence: The misstatements tend to be general when you read Diane Ravitch, who tends to be excessively partisan in support of whichever side she is currently on.

We assume that Ravitch is well intentioned. In certain ways, her new book, Reign of Error, is very useful, at least theoretically.

But Ravitch tends to overstate toward the goal of winning each partisan point. As an example, let’s consider something she said in a post this Tuesday.

The post was aimed at the D.C. schools, one of her favorite targets. She started with a statement which is semi-unknowable in principle, apparently false in fact:

RAVITCH (12/10/13): Despite its recent gains on the 2013 NAEP, the District of Columbia is not a national model.

It remains the lowest performing urban district in the nation.

The post goes on from there, making various points about the D.C. schools. But why would Ravitch make the highlighted claim?

Does D.C. “remain the lowest performing urban district in the nation,” even after its ballyhooed score gains on the 2013 NAEP? Almost surely, it does not. Here’s a bit of background:

In one respect, the answer to that question is semi-unknowable. At present, the NAEP only reports stand-alone scores for about twenty urban districts. There are no such scores for dozens of the nation’s cities, including ten of the twenty biggest.

Full data for those twenty urban districts on the 2013 NAEP are not available yet. (D.C.’s scores have been released because it’s classified as a “state” for certain purposes.)

That said, it’s hard to believe that D.C. is the lowest performing urban district.

Because of its many charter schools; because it’s classified as both a city and as a “state,” with attendant confusion; D.C.’s data can be sliced and diced in many ways. But even on the 2011 NAEP, it’s hard to see how someone could say that D.C. was the lowest performing of the NAEP’s twenty urban districts. (Detroit’s scores were especially horrible.)

Given D.C.’s score gains in 2013, it becomes even less likely that it could be the lowest performing urban district.

Why did Ravitch say what she did? We have no idea, though it helps her drive some favorite points about D.C.’s pursuit of standard “reforms,” whose merits she was overstating just a few years ago.

With Ravitch, this tends to be what you get. That said, it isn’t just Ravitch.

Quite a few liberals have said odd things about the 2012 PISA scores, which were released last week. For our money, it was amazingly strange when Randi Weingarten, head of the AFT, said this in an instant press release:

WEINGARTEN (12/3/13): Today's PISA results drive home what has become abundantly clear: While the intentions may have been good, a decade of top-down, test-based schooling created by No Child Left Behind and Race to the Top—focused on hyper-testing students, sanctioning teachers and closing schools—has failed to improve the quality of American public education. Sadly, our nation has ignored the lessons from the high-performing nations. These countries deeply respect public education, work to ensure that teachers are well-prepared and well-supported, and provide students not just with standards but with tools to meet them—such as ensuring a robust curriculum, addressing equity issues so children with the most needs get the most resources, and increasing parental involvement. None of the top-tier countries, nor any of those that have made great leaps in student performance, like Poland and Germany, has a fixation on testing like the United States does.

Whatever Weingarten may think of “top-down, test-based schooling,” her press release gives the impression that there has been no progress in American schools over the past decade.

From someone who represents the nation’s demonized teachers, that is a very strange post.

Have American schools “failed to improve” in the past decade? That isn’t what Weingarten explicitly said after her words are carefully parsed. But we’d say that impression is clearly conveyed by that remarkable post.

Have American schools “failed to improve?” That’s true if you only consider the PISA; if you assume its results can be trusted; and if you don’t disaggregate. And that’s what our new national cult—the cult of the PISA—wants us all to do!

The cult of the PISA worships two gods; their names are Andreas and PISA. As a result, this jealous new cult wants us to talk about scores from the PISA alone.

In her statement, Weingarten bowed to the wishes of this cult. But so did Ravitch, in her own post about the PISA scores.

As we noted yesterday, Ravitch drew four lessons at the end of her 1400-word post, My View of the PISA Scores. This was her very first lesson:

RAVITCH (12/3/13): From my vantage point as a historian, here is my takeaway from the PISA scores:

Lesson 1: If they mean anything at all, the PISA scores show the failure of the past dozen years of public policy in the United States. The billions invested in testing, test prep, and accountability have not raised test scores or our nation’s relative standing on the league tables. No Child Left Behind and Race to the Top are manifest failures at accomplishing their singular goal of higher test scores.

That statement is simply astounding. In it, Ravitch seems to follow the edicts of the new cult of the PISA.

You can parse those words with care, as with Weingarten's statement. But those words convey a plain impression, just as Weingarten did.

Those words convey the plain impression that U.S. test scores have not improved in the past dozen years. And that claim is perfectly true—but only if you accept the edicts of the new PISA cult.

It’s true! There have been no gains in American scores—if you only consult the PISA, if you don’t disaggregate.

But what if you look at other tests, tests which may be more reliable than the somewhat unconventional PISA? What if you look at the TIMSS? What if you look at the NAEP?

On the NAEP, there have been tremendous score gains over the past dozen years. Her, you see one example:

Average NAEP scores, Grade 8 math, black students
2000: 243
2013: 263

By normal rules of interpretation, that looks like a very large gain. Bowing low to the cult of the PISA, Weingarten and Ravitch act as if those score gains don’t exist.

Since the rest of the “liberal” world doesn’t give a flying fig about what happens to American black kids, this ridiculous conduct by these “intellectual leaders” goes unnoticed, unremarked, unexplored, undiscussed. Because Ravitch is so influential, her ratification of the new cult spreads through the liberal world.

We have no earthly idea why Weingarten says the things she does. But please understand: to her credit, Diane Ravitch knows all about the NAEP!

In Chapter 5 of her new book, Reign of Error, Ravitch discusses the NAEP in detail, noting that she sat on its board of governors for seven years. (She was appointed by President Clinton.)

She goes on and on, sometimes overstating, about the primacy of the NAEP. This is the way the chapter begins:

RAVITCH (page 44): Critics have complained for many years that American students are not learning as much as they used to or that academic performance is flat. But neither of these complaints is accurate.

We have only one authoritative measure of academic performance over time, and that is the National Assessment of Educational Progress, known as NAEP (pronounced “nape”).

“NAEP is central to any discussion of whether American students and the public schools they attend are doing well or badly,” she writes on page 45. Despite the usual puzzling blunders and the standard balls of confusion, she goes on to offer a detailed account of the major score gains on the NAEP, including score gains which occurred between 2000 and 2011 (see page 45).

It’s hard to know how we get from that detailed chapter to Ravitch’s blog post last week, in which she seemed to endorse the idea that no improvement has been recorded in the past dozen years. “No Child Left Behind and Race to the Top are manifest failures at accomplishing their singular goal of higher test scores,” an angry Ravitch wrote.

Carefully parsed, that claim may be true. But it conveys a plain impression, and Ravitch never mentions the NAEP or the TIMSS in her lengthy post.

Why did Ravitch write that unfortunate post about the new PISA scores? We’ll offer two possible guesses:

First, Ravitch is one of the most disorganized thinkers we’ve ever encountered in print. She constantly contradicts herself. (Compare her praise for the NAEP on page 263 of her book to her apparent wholesale condemnation of the NAEP exactly four paragraphs later.)

Second, she may have been unable to resist a partisan urge in last week's post—the urge to overstate against “testing, test prep, and accountability,” and against the two programs (No Child Left Behind and Race to the Top) which have embodied those approaches. When a partisan claim is there to be made, Ravitch rarely misses the chance to overstate it.

Whatever! Whatever her intentions may have been, Ravitch gave the impression in last week’s post that there has been no recent improvement. She never even mentioned the NAEP, or the many NAEP scores gains she details in her book.

Last week, it became quite clear that we have a new cult in this country.

That new cult is the cult of the PISA. Its adepts want you to do these things:

They want you to consult the PISA, and nothing else, when you assess academic performance.

They don’t want you to disaggregate scores. They want you to voice their gloomy conclusions about the lack of progress in our ratty schools, thanks to our ratty teachers.

This is exactly what Ravitch did in her peculiar post. Across the land, that gloomy impression spreads through the liberal world.

What have black kids ever done to have this unfaithful servant as their top liberal advocate? On the NAEP, they recorded that 20-point gain in math.

All across the United States, we liberals refuse to discuss it.

Tomorrow: How to keep pimping a comparison known to be bogus

A further note on PISA scores: For ourselves, we don’t have enormous confidence in the PISA.

That said, we’d like to see solid reporting about the different impressions one might gain from American scores on the PISA as opposed to American scores on the NAEP and the TIMSS.

Still and all, there have been some American gains on the PISA, especially after disaggregation. For example, here are the gains in science scores since 2006 (data for 2000 and 2003 are not available):

Average scores by American students, PISA science, 2006 versus 2012
White students: 523/528
Black students: 409/439
Hispanic students: 439/462
Asian-American students: 499/546
All American students: 489/497

According to Amanda Ripley, 39 points is roughly comparable to one academic year on the PISA scale. On that basis, some of those score gains look fairly substantial, if you trust PISA samples.

We don’t have huge confidence in the PISA, but we’d like to see good solid reporting. Given the way our “press corps” works, that will of course never happen.

Instead, we’ll get what we got last week—the rise of our latest cult. Never mention those scores from the NAEP, its powerful adepts will tell us.

THE LETTERS: Defining the actual shape of our problem!

THURSDAY, JANUARY 2, 2020

American kids punk the world: The letters strike us as a major embarrassment—but they're also highly instructive.

The letters appeared in the New York Times on Sunday, December 22. Did we mention the fact that we think they should be seen as a major embarrassment?

The newspaper published nine letters in all. They explored the reason, or the reasons, for the massive failure of our floundering public schools.

How bad are things in our public schools? Once again, we quote from Letter 4:

"The stagnant results of the international PISA exam have spoken: An extensive overhaul in the American education system is desperately needed."

What was the cause of the stagnant results which signaled the need for this vast overhaul? As we noted yesterday, the letters published by the Times offered a wide array of ideas.

As we noted yesterday, many things were said to be wrong! But no one disputed the basic premise which had emerged from two recent reports in the Times. No one disputed the idea that something is vastly wrong with our public schools, extending across the board.

Are we testing our students too much? Have we foolishly stopped teaching phonics?

Have we failed to listen to teachers as we craft our ballyhooed reforms? Are our drooling kids a mess because we've stopped reading for pleasure?

No one disputed the overall claim—the claim that something is, or seems to be, badly wrong across the board.

But what if we told you that those Pisa scores actually say something quite different? What if large segments of our student population actually punked the entire world on these most recent Pisa tests?

We now return to Letter 4, which recited one treasured script. According to Letter 4, our hapless kids performed so poorly because we aren't like miraculous Finland:

LETTER TO THE NEW YORK TIMES (12/22/19): The stagnant results of the international PISA exam have spoken: An extensive overhaul in the American education system is desperately needed. Although myriad troubles plague American schools—from lack of support for immigrant students to inequalities between schools—part of the solution may lie in one of the countries that outperformed us on the PISA exam: Finland.

Our country has sought to boost test scores by introducing a multitude of standardized tests, essentially forcing teachers to center their class around preparing for these tests rather than teaching their students foundational skills. In Finnish schools, students are subject to almost no standardized tests, yet Finnish students surpassed American students in the PISA exam.

In our desperation to improve academic achievement, our country has fostered a culture obsessed with test results, yet, ironically, this fixation only serves as a detriment to America’s academic performance on the international stage.

Credit where due! This writer said there may be many factors which explain why our kids are so hapless. But she focused on our alleged overtesting—and as is required by Hard Pundit Law, she cited miraculous Finland.

("Where the children are all above average?")

Nothing which follows is meant to be a criticism of Finland—a small, middle-class, single-culture nation which we understand to be extremely well run. How well-run is middle-class Finland? For one example, consider these OECD health care statistics:

Per capita spending, health care, 2018
United States: $10,586
Finland: $4,228

Somehow, Finland provides universal health care while spending roughly 40 percent as much as we do, per person. For reasons the Times will never explore or attempt to explain, we spend well more than twice as much as Finland does, even as many people still don't have coverage, and many more can't afford to receive actual care.

As far as we know, Finland—total population, 5.5 million—is a very well-run nation. That said, it isn't a "peer nation" of ours.

It's a small, middle-class nation with much less demographic complexity. You can assess that demographic fact any way you wish, but Finland is not the U.S.

Still and all, there it was, the mandated praise for Finland's fantastic test scores@! This silly, dimwitted international script took wing roughly twenty years ago, after the very first Pisa tests. On the Sunday morning in question, the Times once again gave it wing.

That said, how dumb are our American kids as compared to the giants of Finland? To what extent do these latest Pisa scores show that we need "an extensive overhaul" of our educational system?

The Pisa tests 15-year-old students; the 2018 Pisa focused on reading while also testing science and math. An initial glance at the data gives us such shame as this:

Average scores, Reading Literacy, 2018 Pisa
Finland: 520
United States: 505

Should those scores be cause for embarrassment? Also, should that 15-point gap in average score be seen as a lot or a little?

We have no rule of thumb which lets us know how large that gap really is. That said, our national shame may be reduced if we include some other average scores:

Average scores, Reading Literacy, 2018 Pisa
Finland: 520
Sweden: 506
United States: 505
United Kingdom: 504
Japan: 504
Denmark: 501
Norway: 499
Germany: 498
France: 493

For all such data, just start here. Then execute one more click.

Say what? Our dunderheads actually outperformed such peer nations as Japan, the U.K., Germany and France? And as compared to Finland's small Nordic neighbors, we scored just one point below Sweden, six points ahead of Norway? With Denmark thrown in for good measure?

Do those numbers make it obvious that we need "an extensive overhaul" of some kind? Offhand, we would say they do not.

That said, the actual shape of our actual problem actually swims into view if we examine a different set of data. Below, you see another way of pondering Finland's performance on the Pisa as compared to our own.

We include the score of tiny Estonia (population, 1.3 million), the world's highest-scoring nation:

Average scores, Reading Literacy, 2018 Pisa
United States, Asian-American kids: 556
United States, white kids: 531
Estonia: 523
Finland: 520

Say what? Can those statistics really be accurate? Did white kids in our floundering public schools punk every nation in the world on the Pisa reading test?

Yes, that's what actually happened! Meanwhile, our nation's Asian-American kids performed as if they go to school on some planet in a galaxy far far away!

There are many different ways to crunch these Pisa data. We'll post one last set of numbers below—numbers which may let us consider the shape of our actual problem.

There are many different ways to crunch these Pisa data. For example:

American kids performed slightly less well well on the Pisa science test, but the basic pattern holds up. American kids performed quite a bit less well on the Pisa math test, which is somewhat unconventional in its design.

Our kids tend to do better on the other major international test in math, the Timss. Perhaps for that reason, upper-class newspapers like the Times have tended to deep-six data from the Timss.

(In 2013, Amanda Ripley's widely-praised and deceptive book, The Smartest Kids in the World, took this disappearing trick to a truly remarkable level.)

Let's return to the scores from the Pisa reading test. For starters, let's make sure we understand what we're talking about.

That rather high score for our "white" kids includes all such kids nationwide. It includes low-income white kids from Appalachia, even from the Deep South. It includes white kids from the Rust Belt who are living with their grandparents because their parents have died from opioids.

That rather high score for our "white" kids isn't a score from Westchester County or Exeter, Miss Porter's and Hotchkiss. It's a score for the whole ragtag band of kids from our nation's dominant demographic group.

Question: When you see their average score—when you see the corresponding score for the Asian-American kids—do you still feel that over-testing has made it impossible for American kids to succeed?

Does it still make sense to wring our hands about the way we no longer read for pleasure? About the way we've (allegedly) stopped teaching phonics?

Does it still make sense to say that our scores reflect the fact that our kids didn't try to do their best when they took the Pisa? Does it make sense to say that our scores are low because we didn't get enough input from teachers in formulating recent plans for "education reform?"

The people whose letters appeared in the Times all seemed to accept the idea that something is grossly wrong across the board in our public schools. But does it make sense to advance that claim when you see that last set of Pisa scores?

For ourselves, we're amazed that those "disaggregated" scores are as high as they are. In our view, it's a minor indictment of Finland and its miracle schools that our nation's ragtag collection of white kids can actually outperform their kids on a major reading test, even as our Asian kids left Finland in the dust.

That's the way those scores strike us, but the scores are what they are. Unless you read the New York Times, which will never report such data to its badly misused readers. Nor will this hapless, upper-class newspaper ever publish this set of scores:

Average scores, Reading Literacy, 2018 Pisa
United States, Asian-American kids: 556
United States, white kids: 531
Estonia: 523
Finland: 520
United States: 505
United States, Hispanic kids: 481
United States, black kids: 448

Our nation's giant "achievement gaps" come into stark relief in those horrible data. Simply put, the New York Times (and the Washington Post) refuse to report or discuss such scores.

Just a guess—people at the New York Times find such data embarrassing. These terrible, useless upper-class people are committed to phantasmagoric tales about the actual size of our gaps and about the best ways to address the gaps, tiny as they are.

Do people at the New York Times even know that those horrible data exist? Given the paper's spectacular incompetence, we wouldn't assume that they do.

That said, we'll guess that people at the Times find those data embarrassing. At any rate, they refuse to report those horrible scores, and they then publish embarrassing letters which help keep script alive.

Tomorrow: Astounding journalistic incompetence (the ninth letter-writers's tale)

WHERE THE PISA GAPS ARE: Amanda Ripley wrote a book!

TUESDAY, OCTOBER 11, 2016

Previous report in this series

Part 1—Half the facts disappeared: In August 2013, a former Time magazine writer published a widely acclaimed, much discussed book about the world's public schools.

Amanda Ripley's ballyhooed book was called The Smartest Kids in the World/And How They Got That Way. One point was clear in Ripley's highly readable presentation:

As it turned out, the smartest kids in the world weren't found in the U.S.! Instead, Ripley focused on three nations whose students, she said, had scored remarkably well on international tests of reading, science and math.

Predictably, some of the smartest kids in the world were found in miraculous Finland. The small, middle-class European country had been a darling of American education writers over the prior decade. In Ripley's book, Finland became one of three countries with "the smartest kids in the world."

Ripley also found the world's smartest kids in South Korea. Again, this was no surprise. South Korea is one of three large Asian nations whose students consistently outscore the rest of the world on the the two international testing programs in which the world's developed nations take part. We refer to the Trends in International Math and Science Studies (the TIMSS) and the Program in International Student Assessment (the PISA).

(The TIMSS has a companion international reading test, the PIRLS.)

Finally, some of the smartest kids in the world were said to be living in Poland. This was perhaps a bit surprising, because this is the way Poland's kids had performed on the latest TIMSS math and science tests, whose results had been released in December 2012:

Average scores, Grade 4 math, 2011 TIMSS
United States: 541
Poland: 481

Average scores, Grade 4 science, 2011 TIMSS
United States: 544
Poland: 505

(Poland didn't participate in the TIMSS on the Grade 8 level.)

Given the way the TIMSS scale works, those were substantial score differentials. Now, Ripley seemed to be describing Poland's students as among the smartest kids in the world, with our pitiful U.S. kids sadly lagging behind.

That said, a funny thing happened to people who read Ripley's widely-praised book. Unless they were education specialists, they had no way of knowing what follows. But as they read Ripley's highly readable book, they learned about results from only one of the developed world's two major testing programs.

Ripley discussed results from the PISA. Results from the TIMSS (and the PIRLS) were, in effect, completely disappeared.

Except in a single endnote, the TIMSS was never mentioned by name at any point in Ripley's book. (The TIMSS was only glancingly named in that lone endnote. The acronym appeared, but not the name. The endnote didn't explain what the acronym meant, or what the TIMSS even is.)

Ripley did cite results from the TIMSS at one point in her book (see page 73). She referred to the TIMSS, without citing its name, as "a major international math test." But she did so only to praise the state of Minnesota for its students' high scores on this unnamed "major international test." She went on to attribute Minnesota's success on this major test to certain types of "education reform."

Ripley never mentioned the high scores achieved by American kids in general on that "major international test." In the course of her entire book, she never said its name.

Readers of Ripley's book were never told that there are two major international testing programs. They learned about results from the PISA. Except for that one targeted use, results from the TIMSS were ignored.

A funny thing happened to readers of Ripley's book. They read a book from which exactly half the international data had in effect been banned. Ripley never told her readers why she had sifted the data this way. Indeed, those readers had no way to know that half the international data were being shielded from view.

On its face, Ripley's presentation of data was strangely selective. On its face, it would be strange to sift the available data this way, without any attempt at explanation, in a 1200-word Sunday newspaper essay.

On its face, this selective sifting of data was only that much more peculiar in a 238-page book.

That said, Ripley's selective presentation had become rather common by the time her book appeared. Especially among advocates of certain types of "education reform," it was now common to do what Ripley had done—to cite results from the PISA, while skipping results from the TIMSS (and the PIRLS).

The United States, and other developed nations, participated in both testing programs. Increasingly, though, only results from the PISA were mentioned. Why in the world was that?

Two possible explanations exist. First, it's commonly said that the PISA focuses on "critical thinking" while the TIMSS focuses on "knowledge of the curriculum." On that basis, a person might judge that the PISA is the more valuable program.

That might be a perfectly valid judgment, though it isn't universally shared. That said, Ripley never explicitly made that case in her widely-praised book. Instead, the TIMSS was simply disappeared, except when its results could be used to praise "education reform."

Is that why Ripley dumped the TIMSS? Did she disappear the TIMSS because she feels the PISA is the only valid measure? Does she believe that the PISA is a substantially better measure?

We can't answer those questions. But there's another possible reason why advocates of "reform" stress the PISA while largely ignoring the TIMSS:

American students have scored less well on the PISA!

When compared to their peers in the rest of the world, American students have scored less well on the PISA. They've scored better on the TIMSS. As such, PISA scores can more fruitfully be used to portray American schools as an embarrassing mess—to argue the case for certain types of "education reform."

Is that why Ripley skipped the TIMSS? We can't answer that. What we can do—what we'll do this week—is offer an overview of the way American students have scored on the PISA.

We'll look at "aggregate" average scores on the PISA—at the average scores achieved by American students as a whole. We'll also "disaggregate" those average scores to look at the performance of major groups within our student population.

How have white American kids performed as compared to the rest of the world? How about black kids and Hispanic kids? How about Asian American students? How have these different groups of kids performed as compared to the rest of the world?

We'll explore an additional question. How have America's immigrant kids performed on the PISA? In one of the most deceptive passages we've ever seen in such a major book, Ripley gives the clear impression that Finland was achieving its usual miracles with its (very small number of) immigrant kids.

We'll show you the data which Ripley withheld. Those data will come directly from the source Ripley cited in one of the most deceptive passages we've ever seen in such a book.

How have American students performed on the PISA? It's an important question. Without forgetting results from the TIMSS, we'll explore the answers (plural) all week.

In many ways—not all—the answers may seem depressing. Within the American student population, very substantial "achievement gaps" exist on both the TIMSS and the PISA.

Those very large gaps may speak to our brutal racial history. They may speak to current educational and social practices. But however daunting those gaps may seem, they help define our ongoing educational challenges. They lead us to next week's final set of reports in this five-week series, a set of reports in which we'll examine this topic:

"Where the Challenges Are."

A final word. Most of our educational challenges are found in our classrooms and homes. But some of our educational challenges are found in the lazy, failing-grade work performed by the American press corps.

Ripley's widely ballyhooed book was highly readable. It many respects, it was highly informative.

It also involved journalistic sleights of hand which should have embarrassed a serious people. But alas! When it comes to the way they discuss our schools, it's hard to say that America's press corps can be described that way. As a group, American "education experts" aren't a whole lot more impressive.

Next week, we'll examine where the challenges are. All this week, we'll be taking a look at the PISA gaps.

Tomorrow: The aggregate scores

WHERE THE ACHIEVEMENT GAPS ARE: On to the PISA!

TUESDAY, OCTOBER 4, 2016

On to the PISA next week: Are U.S. public schools an embarrassing mess as compared to schools in the rest of the world?

How about as compared to schools in the rest of the developed world? Are American schools an embarrassing mess as compared to them?

That impression is constantly given within the American discourse. It's done through comparisons to the miracle schools of miraculous Finland, a small, middle-class, unicultural nation whose kids tend to outscore their American peers on international tests.

It's also done through comparisons to the world's other nations in general. That said, it's hard to conclude that American schools are an embarrassing mess by looking at scores from the TIMSS and the PIRLS, which constitute one of the two international testing programs from which all our data flow.

On the TIMSS (math and science) and the PIRLS (reading), American students tend to score rather well, except when compared to a handful of Asian nations and city-states which tend to outscore everyone else in the world. If we look at results from the TIMSS and the PIRLS, American students and American schools don't seem to be a horrid, embarrassing mess.

Perhaps for that reason, we rarely hear about results from the TIMSS and the PIRLS. When American "opinion leaders" talk about the public schools, they tend to discuss the wonders of Finland—and they tend to discuss test scores from the PISA.

(Note: By all accounts, Finland is an admirable nation which functions extremely well. We would only criticize the misleading way its education ministry tends to promote itself.)

The PISA (the Program for International Student Assessment) is the other major international testing program. On the one hand, we have the TIMSS and the PIRLS. On the other hand, we have the PISA, which conducts tests in reading, math and science.

Compared to kids from other nations, American students haven't ranked as well on the PISA as they have on the TIMSS/PIRLS. Perhaps for that reason, advocates of certain types of education reform tend to discuss results from the PISA while completely ignoring results from the TIMSS and the PIRLS.

We'd planned to discuss "achievement gaps" on the PISA today, but we've decided it can't be done in a single report. We want to show the size of the gaps between the U.S. and other nations. But we also want to "disaggregate" those gaps. That is, we want to show what the gaps look like on the PISA (and on the TIMSS) when you report the divergent average scores of American students from our major population groups.

In Part 3 of our current report, we recorded the large achievement gaps of that type which obtain on our major domestic testing program, the NAEP. Despite large score gains by all major groups, large achievement gaps persist on the NAEP.

In our view, you can't fully understand American scores on international tests unless you see what scores look like for those different major groups. In our view, the data are painful, but instructive. That "disaggregation" makes our nation's educational challenges that much more stark and more clear.

It doesn't make sense to try to do all that in one post. Instead, we'll reshape our ongoing series, which was originally planned to last four weeks in all.

Next Monday, we'll start a new four-part series, Where the PISA Gaps Are. we'll look at the gaps between the world's nations, and between different groups of American kids.

We could try to cram it all in today. We could do it, but it would be wrong.

For today, you might want to look at Kevin Drum's capsule summary of these matters. In this new post, Drum offers an overview of some of the data we've been discussing.

That said, we do have a question about this highlighted point:

DRUM (10/3/16): [A]s Somerby points out, one of the striking things about these tests is that a small clutch of Asian countries do far better than us. In fact, they do far better than everyone, something they accomplish through a combination of cherry picking the students who take the test and a monomaniac culture of test prep. So let's take that as given, and just look at the rankings outside of the Asian tigers...

We'll have occasion to discuss that "monomaniac culture of test prep" again next week. That said, we don't think we've ever seen anyone claim that the high-scoring Asian nations "cherry-pick the students who take" the TIMSS and the PISA.

Drum may be referring to the limited case of Shanghai, which took the PISA as an independent entity in 2009, producing extremely high scores which set off paroxysms in the American press. It was almost like Shanghai's kids had put a new Sputnik in orbit.

As it turned out, Shanghai's student population, and its public schools, are highly unrepresentative of the Chinese system in general. It was a bit like testing kids from Groton and Choate and acting like you'd tested a representative sample of American students.

Once the gorilla dust had settled, the PISA was criticized for letting China participate in the program in such a misleading way. That may be the situation to which Drum referred. Aside from that, we're not aware of claims that South Korea, Japan or Taiwan arrange for an unrepresentative sample of students to be tested. Perhaps some such allegation exists, but we don't think we've ever heard about it.

Asian tigers to the side, American students score pretty well on the TIMSS and the PIRLS. They seem to stack up less well on the PISA.

Next week, we'll do a four-part report about American scores on the PISA. Along the way, we'll break American scores down by groups, and we'll look at how certain states performed on the PISA as independent entities.

In our view, these practices give us a clearer picture of the educational challenges we face. Aside from standard assertions of script and cant, these topics are very rarely discussed. We want to end up with a detailed report which makes lots of basic information available all in one place.

We're sorry for the delay. But we've come to believe that the PISA's gaps deserve a week of their own.

WHERE THE PISA GAPS ARE: Large gaps within the American scores!

THURSDAY, OCTOBER 13, 2016

Part 2 in this series

Part 3—The shape of our demographics: In December 2012, results were released from the latest quadrennial administration of the TIMSS and the PIRLS.

This latest testing had occurred in 2011. As we noted yesterday, American students had performed rather well as compared to their peers around the world. Unless you read the New York Times, where Motoko Rich's news report started like this, gloomy headline included:

RICH (12/11/12): U.S. Students Still Lag Globally In Math and Science, Tests Show

Fourth- and eighth-grade students in the United States continue to lag behind students in several East Asian countries and some European nations in math and science, although American fourth graders are closer to the top performers in reading, according to test results released on Tuesday.

Fretting about how American schools compare with those in other countries has become a regular pastime in education circles. Results from two new reports, the Trends in International Mathematics and Science Study and the Progress in International Reading Literacy Study, are likely to fuel further debate.

South Korea and Singapore led the international rankings in math and fourth-grade science, while Singapore and Taiwan had the top-performing students in eighth-grade science. The United States ranked 11th in fourth-grade math, 9th in eighth-grade math, 7th in fourth-grade science and 10th in eighth-grade science.

Although the average scores among American students were not significantly lower than the top performers, several nations far outstripped the United States in the proportion of students who scored at the highest levels on the math and science tests.

"Several nations" did that! The New York Times was finding ways to see the glass slightly empty.

Technically, nothing in that passage is flatly false. Everything in that passage can perhaps be defended as technically accurate, although the claim about "lagging behind some European nations" was a substantial stretch.

Yesterday, we showed you how USA Today reported those same test results. The more plebeian, multi-hued newspaper saw the glass 90 percent full as it reviewed these newest international scores. Hewing to establishment narrative, the New York Times seemed to see the glass 10 percent empty.

Gloom-and-doom frameworks routinely prevail when the American press reports on American schools. In our view, the New York Times went a bit overboard in its gloomy assessment of those TIMSS scores. That said, when the latest PISA scores were released—those from the 2012 testing—American students performed less well as compared to the rest of the world.

Yesterday, we reviewed the achievement gaps which obtained on the 2012 PISA. That is to say, we looked at the international achievement gaps—the gaps which obtained between American kids (in the aggregate) and their peers from other countries.

On the PISA math test, American students performed less well than their counterparts in almost all comparable countries. On this second major testing program, American students stacked up less well than they had on the TIMSS and the PIRLS.

What explains the difference performances on these two testing programs? That question lies beyond the scope of this series. We'll only the note the heavier weight given by the American press to results from the PISA.

As in Amanda Ripley's high-profile book, The Smartest Kids in the World, American journalists tend to discuss the less impressive American results from the PISA while disappearing the more impressive results from the TIMSS and the PIRLS. This tends to advance a gloomy portrait of the way our schools stack up as compared to the rest of the world.

That said, substantial gaps obtained on the PISA between American students and some of their peers in the rest of the world. That was especially true on the PISA math test. But those are the international gaps. For today, we'll review the domestic achievement gaps which obtained on the 2012 PISA and on the 2011 TIMSS.

We refer to the gaps which obtained between different groups of American students on these international tests. It's painful to review these gaps, but they help us see the shape of our ongoing educational challenges.

Let's start with the PISA reading test, where the average score of American students basically matched the OECD average. Because Finland is so often used as a basis for international comparisons, we'll include that nation's scores in all the presentations which follow.

How did different groups of American students score on that test? When we "disaggregate" the American data, painful domestic gaps appear, although one other gap narrows:

Average scores, reading literacy, 2012 PISA
Finland: 524
United States, white students: 519
United States: 498
United States, Hispanic students: 478
United States, black students: 443

Over the past dozen years, Finland has been relentlessly praised, within the press, for its miraculous public schools. Routinely, journalists have flown to Finland for a week of brainwashing by Finland's education ministry, whose love of propaganda is the one negative trait we know about in that small, distant middle-class land.

Finland has been endlessly praised for its superlative schools. That said, white students in the U.S. essentially matched their peers in Finland on this particular test. In her PISA-friendly book, Ripley seems to say that thirty points on the PISA scale is roughly equal to one academic year. A five-point gap on the PISA scale is a rather modest amount.

On this test, a rather small gap obtained between white American students and their Finnish peers. That said, large gaps obtained between those white American students and their black and Hispanic American peers.

Those painful domestic achievement gaps help define our ongoing educational challenge. What explains their existence?

For ourselves, we'd start with American history. Our benighted ancestors spent several centuries trying to eliminate literacy from the black population. Their horrible efforts failed, but a nation can't spend centuries engaged in such demented practices without reaping the effects of what it so brutally sowed.

For ourselves, we'd start with American history. Other people will suggest other explanations for those large domestic achievement gaps—explanations involving current educational policies and current social practices.

The attempt to explain those punishing gaps lies beyond the scope of this series. We'll continue to explore the domestic and the international gaps in the remaining reports in this series, including such factors as poverty rates and immigration practices.

That said, equally painful domestic gaps obtained on the other PISA tests in 2012. White students were at least within hailing distance of their peers in Finland, where almost every student hails from the majority ethnicity and culture. Black and Hispanic students lagged far behind:

Average scores, math literacy, 2012 PISA
Finland: 519
United States, white students: 506
United States: 481
United States, Hispanic students: 455
United States, black students: 421

Average scores, science literacy, 2012 PISA
Finland: 545
United States, white students: 528
United States: 497
United States, Hispanic students: 462
United States, black students: 439

Those domestic gaps are painful and large. To the extent that anyone cares, they define a very large part of our ongoing educational challenge.

In Finland, almost every student comes from the majority culture. There's very little child poverty. As we'll note in detail tomorrow, there were very few immigrant kids in Finland's student population when these PISA tests occurred.

In the United States, white students—those from the "majority culture"—lagged behind their Finnish peers on the PISA, but not by gigantic amounts. The larger gaps obtained between different groups of American kids, those from our largest population groups.

So far, we've been looking at the gaps which obtained on the 2012 PISA. That said, the United States and Finland had participated, one year before, on the TIMSS and PIRLS.

In our view, what's sauce for the PISA is sauce for the TIMSS. Here's the way the results broke down on the TIMSS math tests:

Average scores, Grade 4 math, 2011 TIMSS
United States, white students: 559
Finland: 545
United States: 541
United States, Hispanic students: 520
United States, black students: 489

Average scores, Grade 8 math, 2011 TIMSS
United States, white students: 530
Finland: 514
United States: 509
United States, Hispanic students: 485
United States, black students: 465

At both grade levels, white students outperformed Finland on the TIMSS math tests. Black and Hispanic students lagged far behind, defining our ongoing challenge to the extent that we care.

Roughly similar patterns obtained on the TIMSS science tests, and on the PIRLS reading test, which is conducted in Grade 4 only.

By the way, those average scores for white students include all American white students. It includes white kids living in poverty in Appalachia. It includes students in certain states whose traditional stress on education may have lagged behind that in some other states. Even with those demographic challenges, white kids in the U.S. outscored Finland on the TIMSS, came fairly close on the PISA.

In our view, these "disaggregated" test results speak to two basic issues. For starters, they help us see the foolishness of the past dozen years of Finland public school chic.

Let's assume that Finland has wonderful schools. There may be lessons that someone can learn from such schools. But the heart of our educational challenge is built in the populations which don't really exist in Finland—populations affected by our brutal racial history, by our higher incidence of poverty, by the low-income immigration practices which don't exist in Finland.

When American journalists fly to Finland, they can't see such challenges addressed. By and large, such challenges don't exist there.

In our view, these data start to point to the foolishness of Finland chic. Beyond that, the punishing domestic achievement gaps found in these basic data helps us see where our challenges lie.

Can we learn how to address those challenges by having our journalists party in Finland? Tomorrow, we'll look at what Amanda Ripley wrote in her widely-praised book about Finland's alleged success with its immigrant kids.

We're not sure we've ever seen a bigger journalistic con in such a high-profile text. Truly, there's nothing elites won't say and do to advance their narrative about our sad embarrassing schools and the horrible terrible pitiful children and adults within them.

Tomorrow: Immigration here and there

VISIONS OF PISA: Ravitch buys into the cult!

WEDNESDAY, DECEMBER 11, 2013

Part 3—A deeply flawed trumpeter: Routinely, reading Diane Ravitch is an extremely large challenge.

Yesterday afternoon, then again this morning, we disappeared to favorite nooks to reread parts of her new book, Reign of Error.

Along with Amanda Ripley’s much more ballyhooed groaner, Reign of Error is one of our two current “big education books.” But as a writer, Ravitch tends to be extremely jumbled. For a world-class example, see below.

That said, Ravitch also tends to be excessively partisan, whichever side of whatever fight she happens to be on:

At the start of the last decade, Ravitch was a leading voice in support of standard “education reform”—in support of standards, testing and accountability. Today, she is the leading liberal voice against the side she was recently on.

We certainly don’t doubt Ravitch’s good intentions. Much of what she writes in her new book is at least theoretically useful.

But good intentions can produce bad results, especially when those good intentions are very strongly felt. This brings us back to Ravitch’s reactions to the new PISA scores.

Yesterday, in Part 2 of this series, we saw Ravitch weirdly denying that American scores were “flat” on the 2012 PISA tests. When she gets her dander up, Ravitch tends to say such things.

Today, let’s consider the four basic lessons she says she drew from the new PISA scores. She stated these lessons at the end of a 1400-word post—a post in which she almost seems to accept the reign of the new cult of the PISA.

What lessons did Ravitch draw from the new PISA scores? We’ll quote in full from the end of her post (all emphases are ours):

RAVITCH (12/3/13): From my vantage point as a historian, here is my takeaway from the PISA scores:

Lesson 1: If they mean anything at all, the PISA scores show the failure of the past dozen years of public policy in the United States. The billions invested in testing, test prep, and accountability have not raised test scores or our nation’s relative standing on the league tables. No Child Left Behind and Race to the Top are manifest failures at accomplishing their singular goal of higher test scores.

Lesson 2: The PISA scores burst the bubble of the alleged “Florida miracle” touted by Jeb Bush. Florida was one of three states–Massachusetts, Connecticut, and Florida–that participated in the PISA testing. Massachusetts did very well, typically scoring above the OECD average and the US average, as you might expect of the nation’s highest performing state on NAEP. Connecticut also did well. But Florida did not do well at all. It turns out that the highly touted “Florida model” of testing, accountability, and choice was not competitive, if you are inclined to take the scores seriously. In math, Florida performed below the OECD average and below the U.S. average. In science, Florida performed below the OECD average and at the U.S. average. In reading, Massachusetts and Connecticut performed above both the OECD and U.S. average, but Florida performed at average for both.

Lesson 3: Improving the quality of life for the nearly one-quarter of students who live in poverty would improve their academic performance.

Lesson 4: We measure only what can be measured. We measure whether students can pick the right answer to a test question. But what we cannot measure matters more. The scores tell us nothing about students’ imagination, their drive, their ability to ask good questions, their insight, their inventiveness, their creativity. If we continue the policies of the Bush and Obama administrations in education, we will not only NOT get higher scores (the Asian nations are so much better at this than we are), but we will crush the very qualities that have given our nation its edge as a cultivator of new talent and new ideas for many years.

Let others have the higher test scores. I prefer to bet on the creative, can-do spirit of the American people, on its character, persistence, ambition, hard work, and big dreams, none of which are ever measured or can be measured by standardized tests like PISA.

For all her virtues, Ravitch is a very aggressive partisan. Consider the second lesson she draws—the one in which she declare victory over Jeb Bush and “the Florida miracle.”

It’s true! Of the three states which participated in the PISA as independent entities, Florida produced the worst aggregate scores. Below, you see some of the relevant scores from the PISA math exam.

We’ll throw in a few other jurisdictions to widen the range of comparisons:

Average scores, 2012 PISA, math
Taiwan: 560
Korea: 554
Finland: 519
Massachusetts: 514
Connecticut: 506
United Kingdom: 494
OECD average: 494
United States: 481
Florida: 467

For all scores, click here, then scroll to page 12.

Florida scored quite poorly in math, below even the U.S. average. It scored well below Massachusetts and Connecticut.

Did Massachusetts “do very well?” That’s pretty much as you like it. Ravitch likes it the way she states it, with glorious Massachusetts leaving Jeb Bush for dead.

It’s also true that Massachusetts trailed Taiwan and Korea by perhaps a bit more than one school year. (Just for the record: Massachusetts has engaged in standards and testing too.)

That said, Ravitch understands, but chose to ignore, one basic set of reasons for the aggregate scores produced by those three states:

Florida’s student population is much poorer than that in Massachusetts. Beyond that, Massachusetts has a much higher percentage of white and Asian-American kids. Those groups remain our nation’s highest scorers, although the gaps have been getting smaller.

Demographically, how do Florida and Massachusetts compare? On the 2011 NAEP, 55 percent of Florida’s eighth-graders qualified for free or reduced price lunch. In Massachusetts, the figure was 33 percent. (This is a measure of lower income. It is not a measure of poverty.)

In Massachusetts, 77 percent of eighth-graders were white or Asian-American. In Florida, the corresponding figure was 48 percent.

(In Connecticut, the numbers resembled Massachusetts: 33 percent qualified for free or reduced price lunch, 70 percent were white or Asian-American. For all these demographic data, click here, scroll to page 85.

One year later, samples of those eighth-grade student populations were taking the 2012 PISA. As everyone knows, those differences in demographics help explain the differences in those states’ PISA scores.

Ravitch knows that it doesn’t make sense to compare those aggregate scores and say nothing else. She may even know this:

On the 2013 NAEP, Florida’s black students outscored their counterparts in Connecticut in Grade 8 math. So did Florida’s Hispanic students.

Readers can’t begin to imagine such facts from those aggregate PISA scores, or from reading what Ravitch wrote. But Ravitch tends to be a somewhat unbalanced partisan, whichever side she is currently on.

Not long ago, Ravitch was overstating on behalf of “reform.” Now, she tends to overstate in the other direction.

For our money, the passage called “Lesson 2” is vintage Ravitch. Beyond that, it isn’t especially helpful, except from a partisan standpoint.

But in some ways, Ravitch’s “Lesson 1” is even more striking.

Gack! In Lesson 1, Ravitch almost seems to buy into the primacy of the new cult of the PISA. Here’s why we say that:

At the start of her lengthy post, Ravitch seems to deny that American scores on the PISA were stagnant or flat. But by the time she hits Lesson 1, she seems to be saying something quite different.

In Lesson 1, Ravitch seems to say that our lousy PISA scores “show the failure of the past dozen years of public policy in the United States.” According to Ravitch, our investment in testing, test prep, and accountability “have not raised test scores or our nation’s relative standing on the league tables.”

According to Ravitch’s new position, our test scores have stayed the same for the past dozen years! But that gloomy pronouncement is only true if we do what cult leaders want—if we only consider aggregate scores, and we only consider the PISA.

Elsewhere, American test scores have been rising. This has been happening on tests which may be more reliable than the slightly unconventional and strongly crusading PISA.

In that deeply unfortunate passage, Ravitch seems to say that American test scores have been frozen in place for the past dozen years. Her desire to damn the side she hates seems to make her adopt the frameworks of a new, unfortunate cult.

In the past week, Ravitch’s reactions have been widely recited in liberal circles. What can it mean when we in the liberal world accept such a deeply flawed trumpeter as our leading voice?

Tomorrow: Disappearing the NAEP (and the TIMSS)

A paragraph for the ages: Tomorrow, we may consider the following paragraph from Ravitch’s book. We can’t be certain, but it may be the most confusing paragraph ever composed:

RAVITCH (page 47): The film Waiting for Superman misinterpreted the NAEP achievement levels. David Guggenheim, the film’s director and narrator, used the NAEP achievement levels to argue that American students were woefully undereducated. The film claimed that 70 percent of eighth-grade students could not read at grade level. That would be dreadful if it were true, but it is not. NAEP does not report grade levels (grade level describes a midpoint on the grading scale where half are above and half are below). Guggenheim assumed that students who were not “proficient” on the NAEP were “below grade level.” That is wrong. Actually, 76 percent on NAEP are basic or above, and 24 percent are below basic. It would be good to reduce the proportion who are “below basic,” but it is 24 percent, not the 70 percent Guggenheim claimed.

That paragraph is a marvel. We’ve never been able to read it without feeling forced to take out pencil and paper. At that point, we attempt to diagram the claims it contains.

It’s hard to believe that a person can create so much confusion merely by stating three percentages, while throwing in one bogus definition. (Since when does the term “grade level” automatically “describe a midpoint on the grading scale where half are above and half are below?”)

Reading Ravitch is a challenge. What can it mean when we in the liberal world accept such a puzzling trumpeter as our leading voice?

How well did American kids do in reading?

WEDNESDAY, DECEMBER 27, 2023

PISA, 2022: As we noted yesterday, the Programme for International Student Assessment (PISA) is one of two (2) high-profile international public school testing programs. 

Here's the overview offered by the leading authority on the PISA:

The Programme for International Student Assessment (PISA) is a worldwide study by the OECD in member and non-member nations intended to evaluate educational systems by measuring 15-year-old school pupils' scholastic performance on mathematics, science, and reading. It was first performed in 2000 and then repeated every three years. Its aim is to provide comparable data with a view to enabling countries to improve their education policies and outcomes. It measures problem solving and cognition.

The results of the 2022 data collection were released in December 2023.

The other such program is the TIMSS. At the risk of sounding extremely jaundiced, the PISA became the international program of choice in this country because of the fact that, for whatever reason, Americans kids score extremely poorly on its math component.

The fact that American kids scored poorly in math on the PISA made the PISA a popular press corps choice. This was back in the day when everyone had bought into a certain, somewhat selective program for public school "reform."

Our journalists are still strongly inclined to adopt the gloomiest possible posture concerning our public schools. In the next few days, we're going to show you results from the 2022 administration of the PISA—results which were released at the start of this month.

Today, we'll start with reading. We're going to start with some good news, and with some news that's not so good.

For starters, riddle us this! In the face of persistent claims about the failure of our public schools, here's the way U.S. pupils scored on last year's PISA reading test. We're including nations of substantial size. You can see a complete listing here:

Average scores, Reading Literacy, 2022 PISA:
Japan: 516
Korea: 515
Taiwan: 515
Canada: 507
United States: 504
Australia: 498
U.K.: 494
Germany: 480
France: 474
Spain: 474

American students outscored their counterparts from the U.K., Germany and France. They scored three points below Canada. (We can't offer you a rule of thumb by which to assess the size of those score gaps.)

You're right! Our public school kids weren't the basketcase of the world on the PISA reading test. They even outscored miraculous Finland, the perennial press corps darling, as you'll see in the data below.

Now we're going to bring introduce "the eternal note of sadness." Here's a slightly larger list of reading scores, including the average scores recorded by the four major American demographic groups:

Average scores, Reading Literacy, 2022 PISA:
U.S. Asian-American kids: 579
Singapore: 543
U.S. white kids: 537
Ireland: 516
Japan: 516
Korea: 515
Taiwan: 515
Canada: 507
United States: 504
Hong Kong: 500
Australia: 498
U.K.: 494
Finland: 490
U.S. Hispanic kids: 481
Germany: 480
France: 474
Spain: 474
U.S. black kids: 459

There are certain kinds of problems involved in comparing scores this way. That said, American white kids scored at what would generally be regarded as a very high level. Asian-American kids basically scored off the planet's charts.

Having said that, alas! In those data, you see the enormous "achievement gaps" which obtain between our four largest student groups. As we'll see in the next few days, our major newspapers continue to work extremely hard to keep their readers from ever having to think about those very large gaps.

(To verify those subgroup scores, you can just click this, though you'll have to click again.)

Our Asian-American kids scored off the charts. Our black kids scored much less well. What explains those giant gaps? Also, what explains the mediocre score recorded by Finland, the former press corps darling?

We'll offer some possible answers in the next few days.  Before the week is done, we'll also look at the PISA scores in science and math—and we'll show you how test scores like these get reported (and get disappeared) in our nation's major newspapers.

Simple story! For reasons at which we're forced to guess, the Washington Post and the New York Times never expose us the people to basic data like these. 

Simply put, it isn't done! As you'll see in the next few days, it wasn't done when these new PISA scores were released at the start of the month.

Amanda Ripley's believe it by law!

FRIDAY, DECEMBER 9, 2016

Truly ridiculous work: As you know, there are two major international testing programs for public school students—the Timss and the Pisa.

The United States participates in both programs. So do the Asian tigers. So does miraculous Finland. So does the vast bulk of the countries in the developed world.

Which program is more valuable—the Pisa or the Timss? We can't tell you that. In our view, the Pisa has developed a slightly cultish feel, but the developed nations seem to see value in both programs. Absent further discussion, we're disinclined to attribute more value to one than to the other.

Check what we just said! If you read the New York Times, it seems the Timss no longer exists. Did we say there are two major testing programs? In the Times, there may now be just one.

Within the past two weeks, each of these testing programs has released its most recent results, from 2015. (The Timss is administered every four years. The Pisa runs on a three-year cycle.)

But in its hard-copy editions, the Times hasn't even mentioned the Timss. Yesterday, it reported the Pisa through a ridiculous, long report by the increasingly cultish Amanda Ripley.

Ripley's 1400-word report is amazingly bad. To show you what cultish behavior looks like, this is the way she began:

RIPLEY (12/8/16): Every three years, half a million 15-year-olds in 69 countries take a two-hour test designed to gauge their ability to think. Unlike other exams, the PISA, as it is known, does not assess what teenagers have memorized. Instead, it asks them to solve problems they haven't seen before, to identify patterns that are not obvious and to make compelling written arguments. It tests the skills, in other words, that machines have not yet mastered.

The snark you see represents the Timss being disappeared. Apparently, it "assesses what [students] have memorized."

That makes it unlike the magnificent Pisa, which is "designed to gauge their ability to think."

Does the Pisa somehow "gauge students' ability to think?" We don't know, and such high-blown claims tend to make us suspicious.

That said, the world's developed nations all find value in the Pisa. Of course, the same is true of the Timss, despite the shade Ripley instantly threw at the Timss without even stating its name.

Ripley played this same strange game in her widely-praised, largely ridiculous book, The Smartest Kids in the World. In its several hundred pages, she never mentioned the Timss by name, though she referred to it as "a major international math test" when she wanted to cherry-pick some Timss results to establish a favored point about the reason for alleged improvement in Minnesota's schools.

It's astounding that Ripley would write an entire book about the world's public schools without mentioning one of the two international testing programs in which the world's nations take part. Before telling us which nations had the best schools, she discarded exactly half the data, without explaining why she did so or acknowledging the fact that she had!

That was a strange way to write a book. Now, the New York Times seems to have adopted this same approach. They're even running Ripley's cultish report under their Upshot brand! That's their most brainiac work!

Ripley's report is horrible in many ways. For today, we'll suggest that you observe its wonderfully cultish shadings. In particular, note the fawning treatment extended to The Founder, Andreas Schleicher, in paragraphs 4-15, which constitutes almost half of Ripley's report.

In that lengthy chunk of her report, Ripley praises the ability of the Pisa staff to predict which countries will show improved test scores, based simply on the extent to which they've adopted certain favored policies. In this passage, Ripley steps aside and lets The Founder heap praise on himself:

RIPLEY: In the end, the PISA team had called virtually every country correctly. Colombia and Singapore had indeed improved. And France had done a bit worse in science and math while improving ever so slightly in reading. ''It's hard to surprise us when it comes to these things,'' Mr. Schleicher said.

That's wonderful clownistry all by itself. Here's the problem:

Based on Ripley's reporting, it doesn't sound like the Pisa team called any country correctly, except to the extent that their predictions were clownishly broad.

Ripley puts her thumb on the scale to pretend that they called the U.S. correctly. Did they somehow call France correctly? According to Ripley, this was the ridiculous "prediction" the Pisa team had made:

"Nobody predicted France would be a star performer." Thus spake Schleicherthustra!

Ripley started out as a standard Time magazine blitherball type. Somehow, she transformed herself into an international "education expert." In the course of executing this move, she signed on to the almost unbearable greatness of the Pisa.

That said, her widely praised book seemed to be baldly dishonest is several remarkable ways. Yesterday's report in the Times was a peculiar gong show.

We'll offer more examples next week. Having said that, let us also say this: Bless their hearts, but the New York Times is just extremely strange.

According to Kevin Drum, lead exposure was very high in the not too distant past. We often think of this notable fact when we read the glorious Times, our nation's greatest newspaper.

VISIONS OF PISA: The liberal world fails!

SATURDAY, DECEMBER 14, 2013

Epilogue—What Zakaria said: For our money, liberal reaction to the new PISA scores was often remarkably clueless.

The most remarkable liberal error involved the apparent acceptance of our nation’s new elite cult.

According to the new cult of the PISA, we should consider the PISA, nothing else, when we talk about test scores.

We shouldn’t discuss American scores on the TIMSS. Most important, those large score gains on the National Assessment of Educational Progress (the NAEP) must never be discussed.

In her reactions to the new PISA scores, Diane Ravitch seemed to buy this premise. So did Randi Weingarten, whose puzzling conduct belies her role as spokesperson for the nation’s much-maligned teacher corps.

Elsewhere, mainstream pundits advanced the key scripts of the new PISA cult. Consider what Fareed Zakaria said in the Washington Post.

Zakaria isn’t an education specialist. There is no sign that he knows very much about international testing, domestic testing or American schools in general.

There’s no reason why Zakaria should know about public schools. That said, lack of knowledge rarely stops elite pundits from offering scripted opinions.

In his column about the PISA, Zakaria made some sensible observations about the American economy. But at one point, he painted a slightly shaky portrait of American standing on international tests:

ZAKARIA (12/5/13): Diane Ravitch, a critic of educational reform, has pointed out that the United States has never done very well on international tests, and yet, the U.S. economy has done better than many higher-scoring countries. Why? Well, the United States benefits from an amazingly flexible free-market economy, a tradition of invention and entrepreneurship, a dynamic society, talented immigrants and a strong work ethic. Those strengths might outweigh poor test scores, on average.

In addition, there’s increasing evidence that it takes a small number of high-achievers to generate a great deal of economic vitality…

[...]

The real story of these tests has been “the rise of the rest.” The United States has muddled along over the past few decades, showing little improvement or decline. Meanwhile, countries including South Korea and Singapore have skyrocketed to the top, and now China, Vietnam and Poland are doing astonishingly well. These countries have workers whose productivity levels have been rising in tandem with their educational achievements.

Zakaria makes several claims in that last paragraph. On the whole, he is stating the Standard Elite Pundit Narrative, the views which have been created and sanctioned within the PISA cult.

That said, how accurate are his various statements? We’d say his portrait isn’t especially accurate. but the liberal world doesn’t seem to know how to challenge this standard portrait.

Let’s examine several standard statements:

Has the United States “shown little improvement or decline over the past few decades?” This is largely true on the PISA, which only dates to the year 2000 and strikes us as a bit shaky in some of its procedures.

On the TIMSS, which dates to 1995, more progress seems to occur. Example:

In Grade 8 math, black students in the U.S. gained 46 points between 1995 and 2011. Hispanic students gained 42 points.

When nations record those kinds of gains, scripted pundits like Zakaria say those nations are “skyrocketing.” Trust us: Zakaria has never heard about those gains by American students, and “liberal leaders” don’t seem inclined to tell him.

Here's our final, most significant point about the claim that the United States is recording no progress:

On the NAEP, our widely-praised federal testing program, large score gains have been recorded over the past few decades. There’s a very good chance that Zakaria has never heard that either.

Ravitch discusses those score gains in some detail in her new book, Reign of Error. In her response to the new PISA scores, those gains on the NAEP went weirdly unmentioned.

It represents educational malpractice when Ravitch and Weingarten discuss the PISA without mentioning those large score gains on the NAEP. It represents journalistic malpractice when newspapers like the Washington Post don’t attempt to analyze the various impressions one might gain from these three major test batteries.

How about Zakaria’s claims about foreign countries? For starters, is it true? “Over the past few decades,” have countries including South Korea and Singapore “skyrocketed to the top?”

Not exactly! That statement may give the impression that Singapore and Korea have been rapidly gaining ground in the past few decades, even as the United States is said to be standing still.

In fact, Singapore and Korea have always been at the top of the pile, right from the first TIMSS testing. These countries outscore the rest of the world, but that has been true all along.

On the TIMSS, the U.S. gained 15 points on Singapore between 1995 and 2011 in Grade 8 math. Would readers imagine any such thing from Zakaria’s description, in which Singapore is skyrocketing to the top as the U.S. stands still?

Final question: Are China, Vietnam and Poland “doing astonishingly well?” Not necessarily, no.

As Zakaria surely must know, we have no nationwide scores for China. No one has any idea what nationwide scores would look like.

Is Poland doing astonishingly well? On the 2012 PISA, Poland scored almost exactly the way Canada did. We haven’t heard why that should be shocking, although it means that Poland scored better than the United States.

That's how Poland scored on the PISA. On the 2011 TIMSS, this is the way Poland scored (Poland didn’t participate at the Grade 8 level):

Grade 4 math, 2011 TIMSS
United States: 541
Poland: 481

Grade 4 science, 2011 TIMSS
United States: 544
Poland: 505

What explains the different outcomes and rankings? Our “press corps” may examine such questions by the end of the century. In the meantime, elite pundits will recite the scripts of their cults, including the cult of the PISA.

As we’ve long told you, everybody praises the NAEP. No one reports what the NAEP data show.

A new cult says we should only consider the PISA when we talk about test scores. Score gains from the NAEP get thrown down a deep, empty well.

In their reactions to the PISA, Ravitch and Weingarten failed to challenge this cult last week. Questions:

What explains the wayward behavior of our big liberal stars? What explains the way the liberal world accepts such hapless behavior?