Limning Minnesota: Rocking the TIMSS!

WEDNESDAY, OCTOBER 30, 2013

Ripley’s first round of embellishments: Did public school students in Minnesota “rock a major international test” in 2007?

That’s what Amanda Ripley says in her widely praised book, The Smartest Kids in the World. Just to refresh you, here’s the passage in which she makes that assessment:
RIPLEY (page 72): Of the three American students I followed, Eric was the only one who did not loathe math. Coincidence or not, Eric’s home state of Minnesota was one of only two states that came close to achieving world-class math performance. Roughly speaking, Minnesota ranked below just a dozen other countries (including Canada, Korea and Finland) in math proficiency; only Massachusetts did better in the United States.

When Eric arrived in Korea, he had a solid math background. There were lots of reasons for this: One might have been that his timing was good. Had he been born earlier, things might have turned out different.

In 1995, Minnesota fourth graders placed below average for the United States on an international math test. Despite being a mostly white, middle-class state, Minnesota was not doing well in math. When Eric started kindergarten two years later, however, the state had smarter and more focused math standards. When he was eleven, Minnesota updated those standards again, with an eye toward international benchmarks. By the time he went to high school, his peers were scoring well above average for the United States and much of the world. In 2007, Minnesota elementary students rocked a major international math test, performing at about the same level as kids in Japan.

What was Minnesota doing that other states were not? The answer was not mystical. Minnesota had started with a relatively strong education system. Then they’d made a few pragmatic changes, the kind of common sense repairs you would make if you believed math was really, truly important—and that all kids were capable of learning it.
In that passage, Ripley says Minnesota “was one of only two states that came close to achieving world-class math performance” at the time in question. She stresses the improvement Minnesota made between 1995 and 2007, when she says it rocked that major international test.

The test in question is the TIMSS, the Trends in International Mathematics and Science Study. Did Minnesota really rock that test?

We wouldn’t necessarily say so. Let’s start by noting one major qualification:

The TIMSS tests students in Grade 4 and Grade 8. If you read carefully, you will see that Ripley only claims that Minnesota rocked the TIMSS at the Grade 4 level.

Did the state actually do that? On the TIMSS scale, 500 is set as the average score, with a standard deviation of 100.

These are the Grade 4 scores in 2007. Not all the world’s countries took part. We are including every country which scored above 510:
Grade 4 math, 2007 TIMSS:
Hong Kong 607
Singapore 599
Taiwan 576
Japan 568
Minnesota 554
Kazakhstan 549
Russia 544
England 541
Latvia 537
Netherlands 535
Lithuania 530
United States 529
Germany 525
Denmark 523
Australia 516
[...]
International average 500
Did Minnesota rock that Grade 4 test? It’s pretty much as you like it! The state scored behind all the Asian tigers, ahead of all the countries in Europe which took part.

(Finland didn’t show up.)

In Grade 8, Minnesota scored 532. The state still trailed all the Asian nations, in this case by larger margins. (Korea took part in Grade 8, scoring 597.) It still outscored all participating European countries.

We wouldn’t say that Minnesota rocked that Grade 4 test. Still, the state outscored the United States and all European nations. This brings us to Ripley’s second claim, concerning Minnesota’s alleged large improvement over the course of twelve years.

In her book, Ripley goes on for several pages, describing the efforts which explain why Minnesota improved so much from 1995 to 2007. But how much did the state improve in its math performance?

In some ways, not that much! Or at least, no more than many states did.

Below, you see Minnesota’s scores (and those of the United States as a whole) on the TIMSS in the years in question. Minnesota’s score improved by 38 points at Grade 4, but by only 14 points at Grade 8:
Minnesota, Grade 4 TIMSS, math:
1995: 516 (U.S. 518)
2007: 554 (U.S. 529)

Minnesota, Grade 8 TIMSS, math:
1995: 518 (U.S. 492)
2007: 532 (U.S. 508)
In our view, Ripley’s portrait falters a bit as we look at these numbers.

According to Ripley, “Minnesota was not doing well in math” in 1995. But in both grades tested on the TIMSS, Minnesota outscored the international average that year, which is always set at 500.

At Grade 8, Minnesota also outscored the national average, by a healthy 26 points.

Twelve years later, Minnesota’s eighth graders produced a higher score on the TIMSS—but so did the U.S. as a whole. (This means that quite a few other states had also shown improvement.) If we rely on TIMSS scores alone, Minnesota’s fourth-graders gained ground on the nation during this period, but the state’s eighth-graders did not.

Why then does Ripley imply, in the fourth paragraph quoted above, that Minnesota showed much more improvement than other states during this twelve-year period? She writes:

“What was Minnesota doing that other states were not?” We’re not sure! But at the eighth-grade level, the fifty states gained an average of 16 points in math while Minnesota was gaining 14!

Ripley also writes this about Eric: “By the time he went to high school, his peers were scoring well above average for the United States and much of the world.” In fact, the state’s eighth-graders were always well above average for the U.S., by a consistent amount. Minnesota’s relative standing hadn’t changed during the twelve years in question.

As a general matter, we don’t think Ripley’s portrait is justified by those TIMSS scores. We don’t see Minnesota doing all that poorly in 1995. Over the next twelve years, we don’t see Minnesota making the type of unparalleled progress that Ripley’s portrait suggests.

Then too, there’s one last point. Did Minnesota’s fourth-graders really “place below average for the United States” in 1995?

Technically yes, they did! The United States scored 518 that year; Minnesota scored 516. But on the TIMSS scale, that is an utterly trivial difference. It’s silly to say that Minnesota “placed below average” for the U.S., except to set up a good story.

That two-point difference is utterly trivial. If you doubt that, look at the way Ripley treats the twelve-point difference between Japan and Minnesota in 2007.

In 2007, Japan outscored Minnesota by 12 points. According to Ripley, this meant that Minnesota was “performing at about the same level as kids in Japan.”

She took a different approach in 1995. In that year, the U.S. outscored Minnesota by only two points. This meant that Minnesota “placed below average for the United States.”

Alas! All through this widely praised book, thumbs appear on many scales as Ripley constructs her stories. In this case, we see no clear sign that Minnesota improved much more than other states did, a key part of Ripley’s portrait. We even find ourselves daring to wonder if the TIMSS might perhaps have gotten a bad sample of Minnesota’s fourth graders in 1995!

Luckily, other data exist; as usual, Ripley makes no attempt to consult them. Why bother? She gets a good story from the TIMSS, Grade 4 level only.

Additional data do exist. They help us answer the questions Ripley poses:

Is Minnesota really our number two state in the teaching of math? Did Minnesota really show more improvement down through the years than the other states did?

It’s stunningly silly to make such claims without consulting the full range of data. Tomorrow, we’ll start to look at NAEP scores for the period in question. When we do, we’ll see Ripley’s embellished claims continue to break down.

Here's the good news: Many states seemed to improve in math during that twelve-year period! As is almost always the case, the actual story seems a bit brighter than the tale Ripley tells.

One final, extremely basic point: To the extent that Minnesota can be ranked number two in math, it’s only because the state has lots of white kids. Throughout her book, Ripley seems completely oblivious to such basic factors.

So do the tools who reviewed her book. You live in a post-journalistic world, a fact the reviews make clear.

16 comments:

  1. an american who happens to have irish-catholic heritageOctober 30, 2013 at 1:33 PM

    is ripley an irish name?

    ReplyDelete
  2. Ripley is a locational name denoting a place in Yorkshire, England. Insofar as before the Reformation Yorkshire (like all the rest of England) was Roman Catholic, it is a Catholic name. Today, York is still notable as a cathedral town. - E
    Google is your friend: https://www.surnamedb.com/Surname/Ripley

    ReplyDelete
  3. You can use the standard deviation as a way of seeing whether a difference between two scores is large or small. When the standard deviation is 100, a difference of 2 points is meaningless. Even differences of 12 or 18 points are small compared to the variability in the whole set of scores.

    Was the point of emphasizing the scores for Minnesota to show that Eric was poorly prepared for math in Korea? Did he struggle? Why would group scores say anything about one student's performance?

    ReplyDelete
  4. OMB (Silly Season and Lots of White Kids!)

    1) "It’s silly to say that Minnesota “placed below average” for the U.S., except to set up a good story." BOB 10/30/13

    2) "Ripley told us how much Poland improved from 2000 to 2006. But how odd! The most recent PISA test results come from 2009. Why didn’t Ripley give us the full enjoyment which would inevitably result from making a full nine-year comparison?" Bob 9/26/13

    "In the talented Ripley’s scam-ridden book, “the Polish miracle” (Ripley’s phrase) is said to be a miracle of improvement. But from 2003 to 2009, American students showed more improvement on the PISA’s three tests than Polish students did." BOB 9/28/13

    It is silly to suggest BOB doesn't know squat about analyzing educational data.

    In quote #1 he says it is silly for Ripley to state the truth. According to BOB the score of Minnesota was only 2 points lower than the US average.

    In quote # 2 he tells us you should use 9 years worth of data when you have it.

    In quote # 3 he uses 6 years of data instead of the 9 he called for only two days earlier in quote #2. Why, you ask? Silly reader. TRIX are for BOB. Full enjoyment of 9 years worth of data would have made quote # 3 a bald faced lie.

    In quote # 3, if it were true, the difference in gain by the US over the Poles is only 3 points. Silly? According to BOB today it is. But it set up a good story then. His.

    In quote # 3 BOB committed a small factual error in presenting a silly statistic to set up his good story. US gains were not higher in the cited period on the PISA's three tests. They were higher on two of the three tests. They were lower on a third. But by silly amounts according to today's standard. BOB was called on this error by your Zarkonship.
    So BOB came up with this:

    4) "American scores improved slightly more on the PISA from 2003 to 2009 than Polish scores did" BOB 10/9/13

    Quote # 4 is true. By dropping reference to the three separate tests that comprise PISA, and combining the total score gains on tests given during these selected years BOB has created a statistic that is true. It is used by no other person in the whole wide world but BOB. And its truth still hangs by the silliest margin of three points on three different tests given eight times in six years. But it's true! Of course with full enjoyment of nine years of data is is false. In fact using a standard statistic with nine years of full enjoyment data, from 2000 to 2009 US students declined while Poland gained in math and reading. We could use a BOBomagic statistic and add the science test. It allows the US to register a total gain over the period because it was only given twice and the US scores went up the second time. Silly? Yes, but nothing is too silly when the lives of tall tales are at stake.

    Coming attractions - "To the extent that Minnesota can be ranked number two in math, it’s only because the state has lots of white kids." You can't imagine how much our planetary palace analysts are licking their chops over this one.

    KZ ( Hoping the sixth game last six hours)

    KZ

    ReplyDelete
  5. Not to mention that "rocked" is a poor choice to describe performance on a test.

    ReplyDelete
  6. KZ is conflating the TIMSS test scores with the PISA. Minnesota didn't take the PISA test as an independent country, as far as I can ascertain. Why KZ would do this, except to troll by throwing sand in everyone's faces, I can't say.

    There are three separate PISA tests, but they were not given in all years, so one cannot average them out over nine years. In 2000, for example, math was not tested.

    Ripley's narrative is that "Eric" was well prepared for math in Korea because he came from Minnesota and Minnesota's fourth graders had scored particularly well in math when he went to school there. For the purposes of this narrative she switches from talking about the PISA to talking about the TIMSS test, only identifying it as "an international test". She omits to say that not only Minnesotans but all Americans students happened to do better on the TIMSS test for some reason than on the PISA tests (though they were not all that bad on the PISA tests either) because that would spoil her narrative about the peculiar degradation of American education with respect to that of Asia and Finland. So, for the purposes of her narrative that all American students are inferior, she has to ascribe an exceptional (and fictional) superiority to Minnesota over the rest of the USA. -E

    Thus is pure spin accomplished.


    ReplyDelete
    Replies
    1. Zarkon conflated TIMSS with nothing, my dear Ellen. Zarkon did not mention TIMSS. Zarkon did not mention Minnesota except to gleefully anticipate the lots of white kids promise to come from BOB.
      So feel free to call me a troll if I am in return free to point out your idiocy in this particular comment.

      But that pales in comparison to this:

      "There are three separate PISA tests, but they were not given in all years, so one cannot average them out over nine years. In 2000, for example, math was not tested."

      PISA Math was given in 2000. The US posted a score of 493. Poland 470. That is why BOB chose to delete 2000 from his two day earlier proclamation that nine years of data should be used. Including it would mean the US students did not outgain Poland in subsequent years. In fact they lost ground on themselves. In 2009 they scored 487. Poland's student posted 495. The same thing happened in Reading.

      But nobody is averaging anything. BOB is attempting to debunk Ripley's selection of Poland by using aggregate gains on the three tests from 2003 to 2009 then comparing the net gain of US students to Poland. On two of the three tests from 2000-2009, the US actually lost ground. On the third, science, which was only given in 2006 and 2009, the US outgained Poland by 3 points, an amount Bob has deemed "silly" when comparing Minnesota math scores to the US average on a test given in the same year.

      Allow me to say your comment is proof positive why BOB thinks he can get away with changing his standards for measuring data in the course of only a few days and thinks he can invent statistics without his readers noticing.

      Also allow me to say all this data, including Ripley's, BOB's, and ours is about as meaningful in life as the box score BOB posted from a baseball game in the 1950's.

      KZ

      Delete
  7. Or maybe Ripley copied from Diane Ravitch without giving credit:

    http://www.forbes.com/2008/12/10/timss-nclb-minnesota-oped-cx_dr_1211ravitch.html

    ReplyDelete
  8. "If we want to see genuine improvement, we should pay attention to Minnesota’s dramatic ascent over the past decade. That state adopted a coherent, focused, grade-by-grade math curriculum developed by a team of Michigan State University scholars and led by Professor William Schmidt. Minnesota competed in the TIMSS study and saw its scores jump from mediocre to world-class.

    While the U.S. continues to rank well below the top-performing nations, Minnesota now ranks fifth in the world, behind only Hong Kong, Singapore, Chinese Taipei and Japan. While U.S. fourth-graders saw a gain of 11 points, Minnesota’s students had a gain of 38 points. Schmidt commented, “Minnesota had more than three times the gain indicated for the United States as a whole. They have left the U.S. behind.”

    Any state could do what Minnesota did. All it requires is implementing a well-designed, coherent curriculum in mathematics and science. Teachers need to know what is expected and should have the appropriate training and resources to enable their students to reach world standards." Diane Ravitch, "American Children Left Behind", Forbes, 2/11/2008

    Unlike Ripley, Ravitch tells you where she got her figures and puts them in some sort of context.
    http://www.forbes.com/2008/12/10/timss-nclb-minnesota-oped-cx_dr_1211ravitch.html

    Minnesota's successful curriculum was put together by teachers, BTW, not corporate testing companies and researchers paid for by billionaires with a right-wing agenda.

    ReplyDelete