US News & World Report has removed Columbia University from its “best colleges” ranking after one of the school’s own professors uncovered discrepancies in the data Columbia submitted to US News. After a follow-up from the magazine, the university failed to substantiate its numbers on class sizes, educational expenditures, and other self-reported data points it had sent to US News. Columbia was duly demoted to “unranked” status.
The university’s conduct is reprehensible, and US News is well-advised to delist it. But the scandal is not an isolated incident. Rather, it is the latest in a series of proven or alleged data-falsification scandals at other elite universities like Emory, Temple, Rutgers, and USC. And those are just the incidents that have come to light. Misreporting appears widespread enough to make the rankings untrustworthy. US News should therefore suspend its college rankings until their journalists can develop a methodology less vulnerable to manipulation.
US News’ current rankings are mostly based on self-reported data from schools. The statistics included in the ranking mostly reflect internal university characteristics, for which there is rarely an external administrative dataset against which to check the numbers. The potential for deliberate misreporting alone makes this a weak methodology. But US News’ formula for undergraduate schools also fails to consider what students look for most in a college: whether it will lead to a good job.
About 40% of US News’ formula for undergraduate college rankings relies on education “inputs,” including class sizes, faculty salaries, expenditures per student, and selectivity. This segment of the formula supplies much of the potential for data manipulation. Columbia University submitted questionable data on class sizes, faculty characteristics, and expenditures; Emory knowingly misreported data on its students’ standardized test scores and class ranks.
But the rankings’ emphasis on inputs is also problematic because students are most concerned with outcomes. Will a college degree from this school help me land a well-paying job? While small classes, full-time faculty and flush endowments are nice to have in a college, those perks will ring hollow for students if their degree doesn’t give them a leg up in the labor market. Yet graduate earnings — despite being easily available through government databases — figure nowhere in the US News undergraduate colleges ranking.
US News does consider some student outcomes, mainly graduation and retention rates. But these are still self-reported by colleges and thus vulnerable to gaming (Columbia also manipulated its graduation rate data by excluding transfer students, according to professor Michael Thaddeus). But even setting aside the misreporting issue, graduation rates are less and less reliable as an indicator of school quality. Many schools goose their graduation rates simply by lowering standards; one paper found that much of the increase in college graduation rates since the 1990s is due to grade inflation. While graduation rates are still an important data point, they are no substitute for solid results like graduate earnings and employment rates.
All these problems should prompt US News to suspend its rankings for a year or two and consider how to make the formula both less vulnerable to manipulation and more relevant to students’ interests. Most critical is to put more emphasis on economic outcomes. US News should leverage graduate earnings data available through the U.S. Department of Education’s College Scorecard, an administrative dataset based on tax returns rather than school self-reporting.
Some will raise the concern that emphasis on outcomes might penalize schools which enroll many disadvantaged students with lower earnings potential. But US News could address this by giving schools more credit when graduate earnings are high relative to what earnings are expected to be based on student demographics. Methodologies already exist to calculate the “value-added” of a college degree in this way, and US News currently makes a similar adjustment in its assessment of schools’ graduation rates.
Education “inputs” like spending, class sizes, and selectivity should no longer enter the formula. After all, inputs are intended to produce outcomes, so rankings should simply measure the outcomes. Moreover, rewarding schools for higher spending and greater selectivity encourages schools to pursue bloat and prestige, contributing to skyrocketing student debt and the socioeconomic stratification of higher education. Institutions which try to serve their students through lower tuition, leaner administration, inclusive admissions, career orientation, and good old-fashioned honesty get little credit. That should change.
Overhauling the US News rankings to prioritize outcomes over inputs would be a radical change, but a necessary one. Too many college rankings scandals have piled up to let the status quo continue. US News should take a year or two off in order to steel its methodology against manipulation and increase its utility to students.
Preston Cooper is a research fellow in higher education at the Foundation for Research on Equal Opportunity.
