Earlier this week a group called the Center for Equal Opportunity released a 21-page analysis of undergraduate admissions data from the University of Wisconsin at Madison, charging what they call “severe discrimination based on race and ethnicity.” Wisconsin students protested at a press conference announcing the findings, while one Republican state legislator is calling for a formal investigation of the university’s selection process.
Wisconsin is already a political tinderbox, of course, and this is likely to add fuel to the fire. It’s legal under binding Supreme Court precedent to consider race as a factor in college admissions, but CEO claims that UW has gone way overboard, admitting manifestly unqualified black and Latino students ahead of more-deserving whites.
I’ve spent a good chunk of the last two days examining the CEO study, and I’ve found that it’s riddled with serious flaws. UW admissions data don’t show what CEO’s report says they do, and the group’s most dramatic claims are its most poorly sourced. CEO is, to put it plainly, misrepresenting the Wisconsin admissions process in multiple serious ways.
At the top of both the CEO study of UW admissions and their press release touting it, the group makes a breathtaking claim about black and Latino students’ chances of admission to Wisconsin’s flagship campus. “The odds ratio favoring African Americans and Hispanics over whites” in UW Madison undergraduate admissions, they say, “was 576-to-1 and 504-to-1, respectively, using the SAT and class rank while controlling for other factors.”
576-to-1. Wow. Those are some pretty steep odds. But what does the claim actually mean?
Well, let’s start with what it doesn’t mean.
It doesn’t mean that the average black applicant to the University of Wisconsin has 576 times the chance of getting in than the average white applicant. Using CEO’s own numbers, the actual figure is about 1.2-to-1. (And as we’ll see in my next post, those numbers are highly problematic — UW’s own publicly available statistics show that black applicants actually have a significantly lower admission rate than whites.)
It also doesn’t mean that a black or Latino applicant to the University of Wisconsin with grades and test scores similar to the average UW applicant has a chance 576 or 504 times greater of winning admission than a white applicant with identical test scores.
The truth is that the CEO report doesn’t ever actually say what they intend to suggest by the 576/504 figures. The statistics’ meaning, they say, “may be difficult to grasp.” The pertinent equations, they say, “are complex and hard to explain.”
So if the meaning of an odds ratio is so obscure, why use it? Why make it the centerpiece of your media campaign?
It’s a good question. And it has a simple answer:
Because any more sensible way of constructing the question wouldn’t make UW’s black and Latino students look stupid.
The odds ratio is an arcane and obscure statistical concept. (I myself misstated it in the first version of this post, as a glance at the early comments shows.) Put as simply as possible, if P is the likelihood of one thing happening and Q is the likelihood of another thing happening, then P/Q is the way most of us would express the ratio of one thing happening versus the other. If P is 95% likely, and Q is 85% likely, then P/Q is 1.12, meaning that P is 1.12 times as likely to happen as Q. That’s what most of us think of when we think of odds, and it’s what most of us think of when we think of an odds ratio.
But it’s not what the term “odds ratio” means to a statistician.
To a statistician, the odds ratio of P to Q is represented by the following equation:
P(1-Q)/Q(1-P)
To put that in slightly plainer English, the odds ratio of P to Q is P multiplied by 1 minus Q divided by Q multiplied by 1 minus P. I am told that this is a useful concept for statisticians.
But however useful it may be for statisticians, it’s not useful for us laypeople, because it means something wholly different from what we expect it to mean. Let’s see what happens when we plug the numbers from my original example into this new formula.
(.95*.15)/(.85*.05) = 3.35
So the chances of P happening are 1.12 times greater than the chances of Q happening, but the odds ratio of P and Q is 3.35. And that gap isn’t consistent between samples — in some situations the two statistics are quite similar, while in others they’re very different. Change P to 99% while leaving Q at 85% and the relative chance of P inches up to 1.16 times the chance of Q while the odds ratio of P and Q soars to 17.47.
I want to underscore that. When P has a 99% chance of happening, and Q has an 85% chance of happening, the odds ratio of P to Q is 17.47. Obviously P isn’t seventeen times as likely to happen — P isn’t even anywhere near twice as likely to happen. (Twice as likely as 85% is 170%, and when you’re talking likelihoods, 170% is a meaningless concept.) So if I tell you that the odds ratio between P and Q is 17.47 to 1, and you’re not a statistician, you’re not going to be more informed than you were before. You’re going to be less informed. You’re going to be misinformed.
And that’s exactly what CEO is counting on.
Do a Google search for “odds ratio misleading” and you’ll find scholarly articles, journalists’ websites, statistical papers, all sorts of documents all saying the same thing — it’s scholarly malpractice to highlight odds ratios in materials intended for public consumption, because the risk of confusion is so high.
And it’s not only the public who gets confused. Look how Linda Chavez, the Chairman of CEO, summarized the group’s odds ratio findings in the Wisconsin Daily Cardinal this morning:
“The studies show that a black or Hispanic undergraduate applicant was more than 500 times likelier to be admitted to Wisconsin-Madison than a similarly qualified white or Asian applicant.”
See that? “More than 500 times likelier.” This isn’t true. It isn’t what CEO claims. An odds ratio is NOT an expression of the relative likelihood of two events. But here’s the head of CEO pretending otherwise in the student newspaper of the very university under discussion.
studentactivism.net 
15 comments
Comments feed for this article
September 15, 2011 at 10:06 pm
Anthony Eastham
Thanks for the explanation. I was at a loss to understand what the hell their numbers meant, but your essay clears it up nicely.
September 16, 2011 at 12:51 am
joshtk76
I am not defending COE nor their report. I would not be surprised if it turned out that there was malfeasance in the report. But Johnston is not the person to uncover it.
He clearly does not understand logistic regression, and he misinterprets what the COE report was doing. In so doing, he grasps at straws to desperately try to invalidate the COE report.
The COE report presents predicted probabilities to compare whites’, blacks’ Asians’, and Hispanics’ chances of admission to UoW. The predicted probabilities are conditioned on the applicant being a nonlegacy male with the ACT score and HS class rank of the median black admittee.
Johnston conflates these predicted probabilities with the huge odds ratios (OR) in favor of black and Hispanic students. He argues that the 576:1 and 504:1 OR are flawed because they “exclude” women and legacies, and because they set the median black admitee as “the standard”. This is absolutely not true. While the differences in predicted probabilities depend on the levels of the predictors, the odds ratios do NOT (side note: there are NO interactions in the COE’s regression models). Thus, you would still see the same huge odds ratios in favor of blacks and Hispanics among women, legacies, people with high ACT scores, people with low ACT scores, people with average ACT scores etc.
Bottom line: anyone familiar with logistic regression will know that Johnston’s attempt to dismiss the huge odds ratios by arguing that “choosing the black admittees’ median as your starting point will produce stranger numbers than using the median of all applicants” is nonsensical.
September 16, 2011 at 7:43 am
Angus Johnston
Josh, it’s just not true that “you would see the same huge odds ratios in favor of blacks and Hispanics among … people with high ACT scores.”
In fact, it CAN’T be true. Because at a certain (not particularly high) level, the odds of a white applicant gaining admission to UW are higher than 1-in-576. And if the odds of a white applicant being admitted are higher than 1 in 576, then the odds of a similarly situated black applicant CAN’T be 576 times greater, because a student’s odds of admission can never be higher than 100%.
And this is my point. The odds favoring black and Latino students over white students (or vice versa) vary depending on which students you’re referring to. At SAT/rank levels where all students of any race are admitted, the odds are even — a black student and a white student each have 100% chances of admission, so their odds of admission are the same. At an SAT/rank level at which only one student of one race is admitted, the disparity is infinite. The ratios are NOT the same across all contingents of students. They never will be.
It is of course possible that I’ve mangled some part of this argument. I’m not a statistician by trade, and as I noted, CEO’s explanation of its methodology is opaque. But in my analysis I relied heavily on the expert testimony of Stephen Raudenbush in the Grutter v. Bollinger case, in which he discussed the use of odds ratios in just this kind of statistical project. Raudenbush IS a statistician by trade, as he’s one of the nation’s foremost scholars of education research methodology. You can find his Grutter testimony here — the most pertinent section begins on about page 65:
http://www.vpcomm.umich.edu/admissions/legal/grutter/gru.trans/gru1.19.01.html
If there are specific flaws in his analysis of the use of odds ratios in analyzing racial trends in university admissions, or in my own application of his analysis to the Wisconsin report, please bring them to my attention and I’ll be happy to correct the post.
September 16, 2011 at 7:59 am
joshtk76
Angus:
* probabilities are bounded from 0 to 1 (or 0 to 100%). But odds are a different animal. Odds = probability / (1-probability) . So odds range from 0 to infinity–the closer the probability is to 1, the denominator becomes an infinitely small number. Moreover, the 576-1 is not odds, but an odds ratio, taking the ratio of a black applicant’s odds of admission to a white applicant’s odds.
* I don’t have time to look over the Raudenbush testimony. You are right that Raudenbush is definitely an authority on statistics, and one I’d trust. You have good taste in statistical experts. I am pretty certain he would not endorse the line of thinking outlined in this post. My suspicion is that Raudenbush outlined a critique similar to yours, but was applying it to differences in probabilities, not odds ratios.
September 16, 2011 at 8:39 am
Angus Johnston
As I said, Josh, I’m not a professional statistician, and I welcome your technical corrections. But the fact is that “the ratio of a black applicant’s odds of admission to a white applicant’s odds” is, in the case of the University of Wisconsin at Madison, not 576-to-1 by any plain English reading of the meaning of that English-language sentence. It’s just not true that a black applicant, similarly situated to a white applicant, has a 576 times better chance of winning admission. It’s not true, and a moment’s attention to the actual admissions numbers will show that it can’t be true. CEO’s own data show broadly similar admissions rates for the two groups, and broadly similar profiles among admitted applicants. If there are divergences of the magnitude they claim, it is only at the margins.
I noted in my original post that I was deriving their methodology by inference, since they declined to state it clearly in their report. If I’ve inferred incorrectly (and I’m not yet convinced that I have), I apologize. But the core argument of my post — that the 576-to-1 stat is, by any common-sense reading of it, meaningless — remains in place.
Edit: I’ve amended my previous comment to reflect your terminological corrections. Thanks again.
September 16, 2011 at 10:55 am
Brad
Thank you Josh for pointing out some of the flaws here.From the outset, I knew there was a COE agenda at work, but even so, I am fairly confident that their numbers are correct (or at least in the ballpark).
Many of the arguments in this blog fail to understand the premises of logistic regression. For example, the thought experiment suggests an arrangement with extremely low sample sizes when in fact, it’s pretty apparent that The University of Wisconsin had to turn over a representative sample of their data if not the entire set for the entire population of an entering class. Then, there’s this statement:
“There’s a basic principle in statistics that the farther away from the middle you get, the weirder your numbers are going to turn out. If you compare the chances of two students near the middle of the pack, you’re going to get stats on their odds of admission that reflect the fact that they’re similarly situated. But if you go looking for outliers, things start to get wacky.”
This would obviously be true for any normally distributed dependent variable and would set up some clear hurdles for any Gaussian analysis. But logistic regression does not require a normally distributed dependent variable, indeed it’s only possible when the dependent variable is binary, in this case admit or deny (1,0).
But let’s think for a second on a different, less statistical path. If the University was required by legal proceedings to turn over the data to an organization which has conducted the exact same experiments in the past on other campuses, wouldn’t they run the same analysis in an attempt to debunk any data manipulation on COE’s part? Wouldn’t that be the FIRST press release that would surface? They have defended their rights to build a diverse class and maintained their legally-sound admissions process, but so far, I haven’t heard a peep about the data validity.
I think that it’s important that we realize that just because we might not agree with the tone, statements, or agenda of this organization, they probably got the data basically right. After all, it would not be in their best interest to put forth a report with easily refutable statistical claims — especially up against a powerhouse like The University of Wisconsin which has access to the same data and many, many excellent researchers and statisticians at its disposal. It’s also worth noting that if they had found nothing, COE certainly would unlikely document that in their report let alone blasting out press releases. This is why organizations like COE are so nefarious: they disguise advocacy in research’s clothing.
What’s going on here is an old-fashioned scare tactic based on the numbers game. Presumably those who read the COE report and the ensuing media blizzard, saw that 576/504 numbers and thought that was high. But this is only because it was pitched out as being high (for more on this read about anchoring effect). But is it really high? Is a 576/504 converted log-odds advantage less than than the disadvantages stacked against those same students regarding college attendance or graduation? In other words, have African American or Latno students overcome even greater odds to have graduated from high school, performed comparatively well to their white peers on the SAT, and be predicted to be successful at one of the world’s great academic institutions?
If so, let’s be proud of our academic institutions’ ongoing public interest in building diverse classes and providing access to those who have been traditionally marginalized or disadvantaged. The Supreme Court of the United States has affirmed that using race in admissions decisions is legal (assuming no quotas/added points etc.) Colleges and universities have a responsibility to maximize their opportunities to build diverse classes under this legal framework.
*The views expressed here are my own*
September 16, 2011 at 11:07 am
Angus Johnston
Thanks to all for the comments. I’ve gone back and done further research, and realized that I inappropriately conflated two different bits of CEO chicanery, as Josh suggested. I’ll be rewriting this post to address the odds ratio issue more accurately, and putting up a new post on CEO’s misleading use of the black applicants’ median SAT scores as their benchmark for comparison.
More soon.
September 16, 2011 at 11:30 am
joshtk76
Brad: “From the outset, I knew there was a COE agenda at work, but even so, I am fairly confident that their numbers are correct (or at least in the ballpark).”
I agree. I don’t want to rule out chicanery on the part of COE, but the reality is their numbers are comparable to those of another study: Espenshade and Radford’s analysis of admissions to 8 selective public and private colleges, which is presented in their book “No Longer Separate, Not Yet Equal”. Espenshade and Radford are highly acclaimed researchers, and this book just won an award given by the Sociology of Education section of the American Sociological Association. We’re not talking hacks at COE.
If you look at Table 3.4 (page 85) in their book–which you can view at Google Books–they show the OR for blacks, Hispanics, and Asians in gaining acceptance at public and private institutions. When you just control for class, the black OR for public institutions is 1.21, meaning a black applicant’s odds of admission are 1.2 times that of a white applicant’s, assuming both applicants are of the same social class (an edge that is is not statistically significant)
Link: http://books.google.com/books?id=47rORpFmuBwC&lpg=PP1&dq=espenshade%20and%20radford&pg=PA85#v=onepage&q&f=false
When you control for measures of academic ability, the black OR for public universities shoots up to 219.85. That is, a black applicant’s odds of admission are 219.85 times that of a white applicant’s assuming both applicants are of the same social class, same SAT scores, same ACT scores, and have taken the same number of AP tests (as well as being equal on other factors such as gender and nativity).
The major disagreement between COE and ER is that ER shows no significant advantage for Hispanic applicants for admission to public universities.
Maybe chicanary on the part of COE led to them exaggerating the admissions edge for black and Hispanic students, but I think we need to realize that there is a substantial edge for these students. We defenders of affirmative action policies are not going to win any battles by arguing the black edge is an OR of 220 rather than 576.
September 16, 2011 at 11:42 am
Angus Johnston
I’ve updated (essentially rewritten) the post to accurately represent the problems with COE’s use of odds ratios.
September 16, 2011 at 2:28 pm
joshtk76
I don’t mean to be overly critical here, but the gist I am taking from these blog posts is that you are trying to undermine the notion that minorities get a sizable benefit in their chances of admission to UoW. I could sign on with the argument that the huge odds ratios that the COE found are misleading to a lay audience, but I think you overstate this point. And the fact is, that is such a huge effect, and if you presented it in any other sort of way that met your approval (say, by comparing black and whites’ probabilities of admission, using the average characteristics of the applicants) you would still see a sizable effect.
Espenshade and Radford–who are not out trying to dismantle affirmative action policies–did this analysis on page 97 of their book (figure 3.7, which you can’t see on Google Books but you can on Amazon.com) and they show that holding all else constant, an average applicant who is black has around a.80 probability of gaining admittance to one of their selective public schools, compared to around .50 for an average applicant who is white.
And remember, Espensahde and Radford’s odds ratio for black kids relative to white kids (219.85) was substantially lower than COE’s (576). So the comparable effect for COE’s analysis would probably be an even larger gap in predicted probabilities.
September 16, 2011 at 2:48 pm
Angus Johnston
I’m not at all trying to minimize the effects here. Statistical arcana may not be my thing, but educational policy is, and I’m well aware of the stats in this field.
But look at what you just posted. The cite you provided was to a study that found black students had a probability of admission 1.6 times higher than whites, other factors held constant, which corresponded to a 220:1 OR. So someone who misstated that odds ratio as a likelihood of admission — as CEO chairman Linda Chavez did yesterday wrt the current study — would be overstating the impact of race by more than 13,000%.
That doesn’t seem like an error worth correcting to you?
September 16, 2011 at 7:01 pm
HerringRed
As a statistician and medical provider who frequently translates research for patients, I’d like to jump in here in attempt to return us to the meaning—the significance, if you will—of these analyses and expose some of their main assumptions. The CEO report represents the worst kind of statistical malfeasance and not because exploits odds ratios but because it strategically detaches the statistical model from the circumstances it purports to represent. Brad pointed to this in his comment above (a great read), and I’d just like to apply his critique to the statistical methods, as deconstructed by Josh.
Josh points out that COE’s model shows that on average, a black student who applies to UW has the same chances of getting accepted as a white student *of the same economic class*. Now that kind of sounds reasonably fair, right? (I mean, ideally, class wouldn’t make a difference, but we can measure poverty and know it makes us do worse academically. Fine.) Poor white Joe and equally poor black John both apply to UW and have equal chances of being accepted.
The results of the model only grow wildly inflammatory when we try to throw in measures of intelligence. Why? Because every so-called objective measure of intelligence is highly confounded by racism. Rigorous studies show that tests like the SAT are designed to give white students an advantage over black students of equal academic prowess (http://www.insidehighered.com/news/2010/06/21/sat, for example). On these exams, “white students have an edge based not on education or study skills or aptitude, but because they are most likely growing up around white people.” There isn’t just one problematic variable entrenched in this model, but 3: As listed by Josh, SAT scores, ACT scores, and AP tests. Each of these variables introduces more error into the results. There is a fair amount of evidence (not to mention common sense) that suggests that these test scores are better measures of racism than they are intelligence. Let’s not forget that UW considers other qualitative variables in its admissions process that may be much better indicators of intelligence. A more meaningful interpretation of the results, then, would be that on average, black and Latino students who are accepted to UW are more likely to have had to overcome a significant amount of racism (as signaled by deflated test scores) in order to get into college than their white counterparts. COE’s complex regression model was supposed to liberate the simple statistics of their racial bias by controlling for these extra variables, but instead produced even more racial bias through them. It is the very complexity of the statistical model is what allows us to neglect the racism integrated into it.
In short, the CEO model is statistically unsound and designed to reinforce the status quo. All the quibbling over the minor details beyond this (probabilities vs. odds ratios, etc.) only elevates the elitist double-talk of our profession, which in turn justifies and reinforces racism. As a statistician myself, I am shamed that the methods employed incorporated racism into the equation, without even so much as a disclaimer. An unethically slippery statistical move.
I am also ashamed by the tone of the comments posted by self-identified statisticians. If the public thinks there’s something wrong with or doesn’t get our math, *we* are accountable to correct, critique or make it clearer. That is our job and we are in conversation with and accountable to the public. Thank you for holding us to it so articulately, Angus Johnston. I apologize for the at times sometimes patronizing tone of my colleagues—we learn it in the best schools.
If a maneuver equivalent to that of the COE analysis were committed in medicine, it would be considered obvious malpractice. It would pose significant added risks to the patient, but ones that the doctor might be able to gloss over if the patient doesn’t have enough privilege to catch it. It’s like a doctor going in to remove an appendicitis patient’s appendix but then also removes a kidney, just because that doctor had the tools, expertise, and medical-speak to do something a little more complicated and might even make more profit off it. Notably, it does not advance the science and hurts the patient. In the current climate of austerity measures and economic downturn, that kind of malpractice is pricier than ever. We cannot afford to condone such statistical waywardness, not the least because it threatens to further dismantle affirmative action when it is most needed. If the numbers tell us anything, it is that institutionalized racism is alive and thriving. We statisticians should be correcting racial and educational disparities, not erasing them by adjusting for them in our models.
September 17, 2011 at 1:44 am
Frank Youell (@fyouell)
joshtk76,
Given that you state your respect for Raudenbush, please take the time to at least skim his testimony. Basically, he freely admits that race is huge factor in admissions. I quote
Page 123
“Q. And you have concluded, have you not as a general matter, that race is important causal effect with respect to admissions decisions that are made at the Michigan Law School, correct?
A. It’s not quite that simple, if I may explain. That the magnitude of the affect is quite large for minority applicants on average, but not for majority applicants.”
“Q. But you have concluded, have you not, that there would be very important consequence for the racial composition of the Michigan Law School if race were not a factor in the admissions process, correct?
A. That’s correct.”
Read it all.
September 17, 2011 at 7:06 pm
joshtk76
@Angus, FWIW, I just saw your twitter account, and that you are engaging with conservatives misinterpreting the odds ratio, and I have a better appreciation for this revised posting now. While I have reservations about using this odds ratio issue to accuse CEO of duplicity, they are outweighed by the benefits of taking down the “576x chance” line parroted by affirmative action opponents.
September 18, 2011 at 7:03 am
Sunday Reading « zunguzungu
[…] to show that minorities are stealing all the college slots from good hard working white people: Do Black Students Really Have a 576-to-1 Advantage in University of Wisconsin Admissions?, How Did CEO Arrive at Their Admission Rate Numbers?, and Final Thoughts, For Now, on the CEO Study […]