Update | Be sure to read this post on the truth behind the numbers — how teachers’ unions do improve student performance, and why. Seriously. Read it. If you don’t, you’re only getting half the story.

Also, if you’d like to follow me on Twitter or Facebook, feel free.

There’s a tweet circulating widely that says that Wisconsin ranks second in the nation in its students’ SAT/ACT scores, while five states that bar teachers from participating in collective bargaining all rank near the bottom of the pile. Here are a few examples:

@WisDems Only 5 states don’t have collective bargaining for educators. Their ACT/SAT rankings: SC-50th/NC-49th/GA-48th/TX-47th/VA-44th.

@PTraeder: 5 states have no collective bargaining & deemed it illegal. States & Their ranking on ACT/SAT scores: VA-44 TX-47 GA-48 NC-49 SC-50 #wiunion

@trek: 5 states forbid collective bargaining for educators: SC, NC, GA, TX, & VA. Their national rank in ACT scores: 50th, 49th, 48th, 47th, 44th.

The information presented here on which states bar collective bargaining in education is correct (although it only reflects the five states where teachers’ unions are illegal, leaving out the other five states where they’re legal but essentially don’t exist). What it doesn’t include, however, is any supporting documentation on the SAT and ACT rankings themselves.

So I’ve taken a look at the data.

State scores on the SAT and ACT are hard to compare directly, because there’s so much variation in how many students take the tests. In addition, I haven’t yet found a source that combines SAT and ACT scores into a composite ranking like the one in the tweet. Looking at charts for SAT and ACT results separately, however, here’s what I found:

Wisconsin ranks 3rd in the nation in SAT scores, but with a participation rate of just 4%. On the ACT, with a much more representative partcipation rate of 69%, it was tied for 17th. In comparison…

  • Virginia was 34th on the SAT with 67% participation, 13th on the ACT with 22% participation.
  • Texas was 45th on the SAT with 53% participation, 33rd on the ACT with 33% participation.
  • Georgia was 48th on the SAT with 74% participation, 34th on the ACT with 44% participation.
  • North Carolina was 38th on the SAT with 63% participation, 20th on the ACT with 16% participation.
  • South Carolina was 49th on the SAT with 66% participation, 44th on the ACT with 52% participation.

Wisconsin is clearly above the other five states in both SAT and ACT scores, but the gap isn’t anywhere near as big as the pro-union tweets suggest. Among high ACT participation states, Wisconsin ranks something like 4th in the nation. But among high SAT participation states, Virginia ranks about 5th in the nation — almost all the states with better SAT scores than Virginia have far smaller participation rates, drawing on a far more elite test-taking group.

I’m open to seeing new data on this, but for now I’m going to mark this claim down as highly exaggerated.

Update | Several readers have posted links to the original source for the tweeted claim, which can be found here. Thanks!

Now, about that source. First, it’s from an analysis conducted in 1999, apparently by a University of Missouri law professor named Douglas O. Linder. Linder doesn’t say specifically what year the SAT/ACT numbers come from, but they’re obviously more than a decade old.

Second, the ranking methodology is really weird, and completely unreliable. What the author did was take each state’s ranking on the SAT, add it to each state’s ranking on the ACT, add those two numbers together, and then put them in order. In other words, Wisconsin scored 5th in the country on the SATs and 4th in the country on the ACTs, giving it a total of 9, and only Iowa had a lower total, so Wisconsin was 2nd in the country overall.

This is just silly. As I noted above, almost nobody in Wisconsin takes the SATs — the state has only a 4% participation rate on that test, with the only students taking it being those who are applying to competitive out-of-state colleges that don’t accept the ACTs. When you compare Wisconsin’s SAT average to Georgia’s, you’re comparing the performance of a tiny elite in one state with that of 74% of the graduating class of the other. And on top of that, this chart gives Wisconsin’s SAT score equal weight with its ACT score in determining which state is “better.”

It’s nonsense. It’s meaningless.

Second Update | The scores don’t seem to match the data, either. Linder gives Wisconsin’s average combined Math and Verbal SAT score as 1073, but Wisconsin’s average SAT scores in the late 1990s ranged from 1169 in 1996-97 to 1181 in 1999-2000. Something’s screwy with Linder’s numbers.

Third Update | As I said at the top of the post, I’ve put up a new essay this morning discussing exactly how teachers’ unions do improve student performance, and why. If you’ve read this far, you really should keep reading.