More and more professors are coming forward to express their skepticism of university ranking systems. This time, they’re speaking up about the rankings of business programs.
Bloomberg Businessweek, which publishes an annual MBA ranking, has for years distorted submitted data, according to Anjani Jain, deputy dean at the Yale School of Management. Jain says he’s not the only one with concerns and other business school deans and faculty have doubts over the reliability of widely-read MBA rankings.
In February, math professor Michael Thaddeus exposed faulty data provided by Columbia University to U.S. News & World Report’s top universities ranking. The revelation caused Columbia, ranked no. 2 in 2021, to fall to the no. 18 spot. Meanwhile, Temple University’s former business school dean was convicted of wire fraud last year, after submitting false data which ranked the university’s online MBA program No. 1 by U.S. News from 2015 to 2018.
MBA rankings have profound impacts on business school applications and the size of programs. The top 10 programs have gotten larger, with more applicants, money and faculty, because of the rankings, according to Kenneth Brown, a professor at the University of Iowa’s Tippie College of Business. “There are very real world consequences for these rankings, and they’re probably more dramatic for business schools—faculty have always been concerned that these aren’t really helping students, but it’s become a reality we have to face,” said Brown.
Yale’s Jain, who considers himself an admirer of Thaddeus, first raised issues with Businessweek’s MBA ranking back in 2021. “I somewhat accidentally discovered a problem with it last year,” he said.
Factors like ‘compensation’ given a disproportionate weight
Businessweek gathers data from around 84 different U.S. business schools on five different components: compensation, learning, networking, entrepreneurship and diversity. “The unique feature of this ranking is that they don’t impose their own views on how much each of these factors should be rated,” said Jain. Businessweek instead asks respondents what the weights should be, using stakeholder-generated weights for each component. However, Jain ran into problems when he looked into public data posted by Businessweek alongside its 2021 ranking. When he deduced what the weights for each component would have to be in order to replicate the ranking, he found different results from the stakeholder-generated weights. “You couldn’t replicate the rank order they published using the data they posted and the methodology they described,” he said.
With Businessweek’s 2022 rankings, Jain found the same issue once again, explaining the data distortion in an article in Poets & Quants, a website that focuses on business schools. Compensation, given a 37 percent stakeholder-generated weight, was given a 69 percent weight by Businessweek, while the weight given to learning decreased from 25 percent to 6 percent. “It’s disingenuous. They keep making claims about the virtue of stakeholder-generated weights, when their calculations distort the weights very substantially,” said Jain.
Jain believes that applying the stakeholder-generated weights properly would create a very different ranking — although the top five schools would remain more or less unchanged, the rest of the list would be substantially reshuffled. However, he said Businessweek likely doesn’t want to change its methodology, as a significantly different-looking list could raise questions about the credibility of its past rankings. “It reminds me of the Chinese saying, ‘when you ride a tiger, you can’t easily dismount,” said Jain.
In a statement provided to the Observer, a Bloomberg Businessweek spokesperson said the publisher stands by its rankings.
Beyond Businessweek, Jain believes MBA rankings in general are flawed, reducing multifaceted student experiences into simple linear orders. “But if you’re going to undertake this, at least do it with some integrity,” he said, adding that Businessweek is not the only MBA ranking with issues.
Self-reported data and peer evaluations raise questions of validity
Challenges have previously been raised about the validity of data provided to ranking publishers, said Homer Erekson, a professor at Texas Christian University’s Neely School of Business. Erekson believes the aspects of peer evaluation and self-reported data, common across MBA rankings, are not reliable. And while Businessweek and the Financial Times, another popular compiler of MBA rankings, publish their methodologies, Erekson argues that even those that take the time to double-check rankings with methodologies are still relying upon the validity of data provided by universities themselves.
The Financial Times did not respond to requests for comment.
Compensation, which measures salaries after graduation, is often disproportionately prioritized in MBA rankings, said Brown. While an index like diversity typically won’t vary significantly across universities, compensation differs widely across region and industry, he said. And schools that place students in cities with high costs of living, such as New York and San Francisco, will often reflect higher salaries. The variability of a factor like compensation will disproportionately emphasize its importance in rankings.
“Where schools show the greatest differences, that will be what’s reflected in any type of ranking,” said Brown. “They’re giving credit to schools that place more people in higher paying industries and cities with higher costs of living. It’s difficult to adjust for that.” While high-paying salaries are a goal for many business students, Brown worries that rankings place more emphasis on compensation than other factors concerning the quality of education.
After surprising rankings, some publishers have halted their lists altogether
Recently, some prominent MBA ranking publishers have halted their annual ranking lists. In August, Poets & Quants reported Forbes would also discontinue its MBA rankings for 2022, citing the loss of a senior editor in charge of compiling the list. Meanwhile, The Economist in 2021 published a ranking which numerous business schools declined to take part in, citing data disruptions due to the Covid-19 pandemic. Fifteen of the top 25 schools skipped the 2021 ranking, which resulted in a list that many criticized as uneven. The Economist in July announced that its 2022 MBA ranking would be its last.
The Economist and Forbes did not respond to requests for comment.
According to Brown, this type of backlash reveals how ranking publishers aren’t open to change despite inherent issues within their rankings. “If you end up with something that doesn’t reinforce people’s existing perceptions of status, people will question and not read your publication,” he said. “But if that’s the case, don’t these just reinforce standing status perceptions?”
We get it: you like to have control of your own internet experience.
But advertising revenue helps support our journalism.
To read our full stories, please turn off your ad blocker.
We’d really appreciate it.
Below are steps you can take in order to whitelist Observer.com on your browser:
Click the AdBlock button on your browser and select Don’t run on pages on this domain.
Click the AdBlock Plus button on your browser and select Enabled on this site.
Click the AdBlock Plus button on your browser and select Disable on Observer.com.