As I posted yesterday, the tweets going around comparing Wisconsin’s SAT/ACT scores to five states where teachers have no right to unionize are based on bad data — it’s not true that Wisconsin’s SAT/ACT ranking is second in the nation, and that Texas, Virginia, Georgia, North Carolina, and South Carolina are all clustered at the bottom of the pile. The chart that says otherwise is based on outdated statistics and improper statistical analysis.

So what’s the truth? What would good data tell us about this question?

Well, it turns out that that’s kind of a complicated question. I can answer it, but you’ll have to bear with me for more than 140 characters.

It’s hard to measure SAT/ACT performance, because different numbers of students take the tests in every state, and comparing the strongest students from one state with a much bigger sample from another doesn’t tell us much that’s interesting. A 2000 study in the Harvard Educational Review, in fact, found that 85% of the difference in states’ performance on those tests is due to variation in participation rates.

Having said that, though, it’s clear from the numbers in my last post that once you’ve controlled for participation Wisconsin remains near the top of the country on SAT/ACT scores, Virginia is near the middle, and the rest of the no-union states from the tweet are near the bottom. High school graduation rates — the subject of another popular Wisconsin tweet meme in recent days — tell a similar story. It’s not as dramatic as best vs. worst, but it’s still dramatic.

Wisconsin does well on a third measure of student performance, too. Its scores on the National Assessment of Educational Progress (NAEP) in 2009 were above the national average in three of four measures (fourth grade math and eighth grade math and reading) and at the national average in the other (fourth grade reading). Of the ten states in the US without teachers’ unions, only one — Virginia — had NAEP results above the national average, and four — Arizona, Alabama, Louisiana, and Mississippi — were in the bottom quintile. (One scholar, in fact, found that the states with the strongest teachers unions tended to out-perform states with weaker unions too.)

But there’s another problem. Most of the states that don’t have teachers’ unions are poorer than Wisconsin, and have more English Language Learners in their schools, and rank higher for other demographic factors that make strong academic performance less likely. Rich kids in a school with a teacher’s union will do better than poor kids in a school without one, generally, but that doesn’t have much to do with the union itself. States with teachers’ unions do better, on average, than states without, but is this because of the unions, or state demographics?

There’s only been one scholarly effort to tackle this problem that I’m aware of. Back in 2000, three professors writing in the Harvard Educational Review did a statistical analysis of state SAT/ACT scores, controlling for factors like race, median income, and parental education. They found that the presence of teachers unions in a state did have a measurable and significant correlation with increased test scores — that going to school in a union state would, for instance, raise average SATs by about 50 points.

Two other findings leap out from the Harvard Educational Review study. First, they concluded that Southern states’ poor academic performance could be explained almost entirely by that region’s lack of unionization, even when you didn’t take socioeconomic differences into account.

And second, and to my mind far more interesting, they found that concrete improvements in the educational environment associated with teachers’ unions — lower class sizes, higher state spending on education, bigger teacher salaries — accounted for very little of the union/non-union variation. Teachers’ unions, in other words, don’t just help students by reducing class sizes or increasing educational spending. In their conclusion, they stated that

“other mechanism(s) (ie, better working conditions; greater worker autonomy, security, and dignity; improved administration; better training of teachers; greater levels of faculty professionalism) must be at work here.”

To sum up:

Yes, Wisconsin has great schools, with great outcomes. Yes, states without teachers’ unions lag behind. Yes, that lag persists even when you control for demographic variables. Yes, that difference seems to rest less on the quantifiable resources that unions fight to bring to the classroom than on the professionalism, positive working environment, and effective school administration that unions foster.

And yes, Virginia, (and Texas, Georgia, and North and South Carolina) unions do work.


Update | Matthew Di Carlo of the Shanker Blog has a new post up reviewing the state of scholarship on the relationship between teacher unionization and student performance, looking at (and linking to) several articles I missed. Di Carlo leans a little more heavily on an “on the one hand/on the other hand” approach than I would, but it’s definitely a worthwhile read for both his data and his analysis.