Earlier this week a group called the Center for Equal Opportunity released a 21-page analysis of undergraduate admissions data from the University of Wisconsin at Madison, charging what they call “severe discrimination based on race and ethnicity.” Wisconsin students protested at a press conference announcing the findings, while one Republican state legislator is calling for a formal investigation of the university’s selection process.
Wisconsin is already a political tinderbox, of course, and this is likely to add fuel to the fire. It’s legal under binding Supreme Court precedent to consider race as a factor in college admissions, but CEO claims that UW has gone way overboard, admitting manifestly unqualified black and Latino students ahead of more-deserving whites.
I’ve spent a good chunk of the last two days examining the CEO study, and I’ve found that it’s riddled with serious flaws. UW admissions data don’t show what CEO’s report says they do, and the group’s most dramatic claims are its most poorly sourced. CEO is, to put it plainly, misrepresenting the Wisconsin admissions process in multiple serious ways.
At the top of both the CEO study of UW admissions and their press release touting it, the group makes a breathtaking claim about black and Latino students’ chances of admission to Wisconsin’s flagship campus. “The odds ratio favoring African Americans and Hispanics over whites” in UW Madison undergraduate admissions, they say, “was 576-to-1 and 504-to-1, respectively, using the SAT and class rank while controlling for other factors.”
576-to-1. Wow. Those are some pretty steep odds. But what does the claim actually mean?
Well, let’s start with what it doesn’t mean.
It doesn’t mean that the average black applicant to the University of Wisconsin has 576 times the chance of getting in than the average white applicant. Using CEO’s own numbers, the actual figure is about 1.2-to-1. (And as we’ll see in my next post, those numbers are highly problematic — UW’s own publicly available statistics show that black applicants actually have a significantly lower admission rate than whites.)
It also doesn’t mean that a black or Latino applicant to the University of Wisconsin with grades and test scores similar to the average UW applicant has a chance 576 or 504 times greater of winning admission than a white applicant with identical test scores.
The truth is that the CEO report doesn’t ever actually say what they intend to suggest by the 576/504 figures. The statistics’ meaning, they say, “may be difficult to grasp.” The pertinent equations, they say, “are complex and hard to explain.”
So if the meaning of an odds ratio is so obscure, why use it? Why make it the centerpiece of your media campaign?
It’s a good question. And it has a simple answer:
Because any more sensible way of constructing the question wouldn’t make UW’s black and Latino students look stupid.
The odds ratio is an arcane and obscure statistical concept. (I myself misstated it in the first version of this post, as a glance at the early comments shows.) Put as simply as possible, if P is the likelihood of one thing happening and Q is the likelihood of another thing happening, then P/Q is the way most of us would express the ratio of one thing happening versus the other. If P is 95% likely, and Q is 85% likely, then P/Q is 1.12, meaning that P is 1.12 times as likely to happen as Q. That’s what most of us think of when we think of odds, and it’s what most of us think of when we think of an odds ratio.
But it’s not what the term “odds ratio” means to a statistician.
To a statistician, the odds ratio of P to Q is represented by the following equation:
To put that in slightly plainer English, the odds ratio of P to Q is P multiplied by 1 minus Q divided by Q multiplied by 1 minus P. I am told that this is a useful concept for statisticians.
But however useful it may be for statisticians, it’s not useful for us laypeople, because it means something wholly different from what we expect it to mean. Let’s see what happens when we plug the numbers from my original example into this new formula.
(.95*.15)/(.85*.05) = 3.35
So the chances of P happening are 1.12 times greater than the chances of Q happening, but the odds ratio of P and Q is 3.35. And that gap isn’t consistent between samples — in some situations the two statistics are quite similar, while in others they’re very different. Change P to 99% while leaving Q at 85% and the relative chance of P inches up to 1.16 times the chance of Q while the odds ratio of P and Q soars to 17.47.
I want to underscore that. When P has a 99% chance of happening, and Q has an 85% chance of happening, the odds ratio of P to Q is 17.47. Obviously P isn’t seventeen times as likely to happen — P isn’t even anywhere near twice as likely to happen. (Twice as likely as 85% is 170%, and when you’re talking likelihoods, 170% is a meaningless concept.) So if I tell you that the odds ratio between P and Q is 17.47 to 1, and you’re not a statistician, you’re not going to be more informed than you were before. You’re going to be less informed. You’re going to be misinformed.
And that’s exactly what CEO is counting on.
Do a Google search for “odds ratio misleading” and you’ll find scholarly articles, journalists’ websites, statistical papers, all sorts of documents all saying the same thing — it’s scholarly malpractice to highlight odds ratios in materials intended for public consumption, because the risk of confusion is so high.
And it’s not only the public who gets confused. Look how Linda Chavez, the Chairman of CEO, summarized the group’s odds ratio findings in the Wisconsin Daily Cardinal this morning:
“The studies show that a black or Hispanic undergraduate applicant was more than 500 times likelier to be admitted to Wisconsin-Madison than a similarly qualified white or Asian applicant.”
See that? “More than 500 times likelier.” This isn’t true. It isn’t what CEO claims. An odds ratio is NOT an expression of the relative likelihood of two events. But here’s the head of CEO pretending otherwise in the student newspaper of the very university under discussion.