How Criticism Actually Strengthens Rankings

Why do college rankings endure? In a paper published last month, two German sociologists, Julian Hamann and Leopold Ringel, put forth a provocative hypothesis: that criticism ultimately strengthens rankers, who respond by modifying their work rather than abandoning it, which ends up reinforcing their legitimacy. It’s like having the same argument with your spouse over who should do the dishes. In the end, it just reaffirms each of your places in the relationship.

Hamann and Ringel’s idea is especially relevant now. Over the past three months, dozens of law and medical schools have announced they will no longer cooperate with U.S. News & World Report on its rankings. The collective exodus gave law- and medical-school deans an opportunity to air their grievances with ranked lists in a high-profile way, and make it harder for U.S. News to do the rankings, which rely on the schools’ self-reported data. (U.S. News analysts have said they will continue to produce their lists with publicly available numbers.)

It’s been academe’s biggest protest of rankings in years. Yet could the ultimate critique — leaving the rankings — end up cementing rankers’ place in higher education? The Chronicle spoke with Hamann, a junior professor at Humboldt University of Berlin, and Ringel, a postdoctoral researcher at Bielefeld University, about how they interpret the current discontent with rankings, higher ed’s role in the rankings process, and how change could really happen. This conversation has been edited for length and clarity.

You describe this cycle of rankings producing criticism, which then leads to a stronger ranking. How does that work?

Leopold Ringel: If you say that the methodology is flawed, then by implication you’re saying: “We need a methodology that is less flawed, or not flawed.” That gives the rankers the opportunity to say: “Tell us how to make it better. Next year, we’re going to improve it. Then we can improve it again and again and again.”

Courtesy of Julian Hamann

Whether it’s intentional or not, the critique inevitably calls on a better ranking and not, no rankings.

Julian Hamann: One outcome of this kind of critique is rankings that are even more rigorous and even more robust against criticism.

Some critics would like to get rid of the rankings. Should those people then not critique the ranked lists? Is there a way to get out of this mutually reinforcing cycle?

JH: If the goal of higher-education institutions is to get out of this cycle, then boycotting rankings is an efficient way to do so. Just stop submitting data to the ranking organizations.

LR: And not talk about it so much.

A second, almost opposite, option would be to engage in a way that you change them effectively from a ranking into something else. That’s what happened in Germany. There was one major university ranking, it started in the ’90s and it’s still called rankings, but it’s not a ranking anymore. It’s just a rating. One of the reasons for that is apparently they listened so much to academic critics who always said: “You shouldn’t make a ranking. It simplifies too much. We need this traffic light.” It’s now red, green, and yellow, these are the colors that are used to rate the universities.

JH: But one main difference between the U.S. and Germany is that we do not have this long history of a stratified and hierarchical education system. When I was in the U.S., I was always surprised that colleagues have in their head this sort of landscape. They can tell you exactly where to position which school and which university is better. I always wondered, where do they get these precise ideas from?

Did that German ranking system become less popular after it became a rating?

LR: I don’t think we have any data. It’s really hard to quantify that.

JH: I mean, the picture that ratings convey is probably more complex than the picture ranking conveys. So it’s more difficult to sell the story because you cannot say: “These are the three top institutions.” You have to approach it in a more nuanced way. So my intuition would be that ratings are more difficult to popularize.

Boycotting the ranking, that’s what law and medical schools in the U.S. did to U.S. News. Would you say that was an effective way of weakening the rankings?

JH: I would say it’s not an effective way of weakening rankings. I find it striking that most law-school deans criticize the rankings for their methodological shortcomings. They have concerns about whether the methodology can capture this complex experience of the exceptional learning environment and so on. Yes, they boycotted the rankings, but by issuing this methodological critique, they also gave clues on how to improve the ranking.

Courtesy of Leopold Ringel

LR: They say: “We call upon the ranking to be better.” Then the rankers can say: “Well, it is better.” Then it puts the onus back on you. You have to deal with that now. That’s how they create difficulties for themselves in the future. That’s what eventually could lead these law schools to join back in.

[Indeed, after the law-school protest, U.S. News analysts scheduled calls with deans to hear their complaints and promised to change their methodology to take into account the feedback. In November, Erwin Chemerinsky, dean of law at the University of California at Berkeley, told The Chronicle he would be open to rejoining the rankings “if they were to change their formula in a way that doesn’t conflict with our mission.”]

JH: Another reason why it is likely that they will rejoin the ranking is that rankings and elite institutions need each other. Rankings need the Harvards and Yales and Stanfords, just as any football league needs the top teams, because this is what draws the attention and the interest from the public, but also the elite institutions need rankings because someone has to certify their elite status. This is another reason why I find it likely that this sort of productive relation between elite institutions and rankings will survive.

Does your work, of writing sociology papers about rankings, contribute to this cycle?

JH: We cannot exempt ourselves from this whole dynamic. We ourselves could be considered critics of rankings. This means that rankers can co-opt our critique and use it as a means to improve the rankings. There is nothing we can do about this.

This is not about rankers deceiving well-meaning, honest researchers, and it’s not about naïve researchers that are corrupted by the ranking industry. It’s about more general discursive dynamics. This constructive conversation between rankings and critics unfolds on its own, whether those involved want it or not. And this applies to us as well.

LR: We would like to draw the attention of our colleagues to the fact that if we say something in public, that can, in and of itself, already be an intervention. Maybe it’s one that we didn’t wish for, which doesn’t mean that you shouldn’t make it, but you should be aware of what you are doing.