Incorrectly reported employment data affecting U.S. News ranking?

[Anne]

Here's a little tidbit that I found more than a little scary (and angering) in U.S. News' "Morse Code" blog, which deals with the USN academic rankings:

Updates to One Law School's Ranking Data
April 29, 2010
Source: http://www.usnews.com/blogs/college-rankings-blog/2010/04/29/updates-to-one-law-schools-ranking-data.html

Officials at just one school to date, University of Alabama's law school, have told U.S. News that they made errors reporting some of their data that were used in the newly published America's Best Graduate Schools 2011 rankings. U.S. News won't recalculate the rankings because of this or any mistake made by the schools, but we are soon going to correct the data on our website.

University of Alabama officials were too late in reporting the information used to calculate the percentage of their 2008 graduating class that was employed, which affected the new law school rankings. U.S. News received the correct data from University of Alabama's law school after the rankings were completed and published. University of Alabama's correct rate for 2008 law graduates employed at graduation is 92.1 percent. The University of Alabama law school received an estimate in the law school ranking model for this value and it was published as N/A or not available.

University of Alabama's law schools would have ranked higher, had its actual at graduation employment rate data had been used in the rankings instead of an estimate.

If other schools have made errors in their rankings data and they notify U.S. News via E-mail at official@usnews.com, U.S. News will explain the error in this blog and then correct the data online.

******

Here's my issue with this:

1. Why is USN okay with schools reporting "estimated" data? In how many other sectors are they accepting "estimated" data? Bar passage rate? Attrition rates? Student-teacher ratios? GPA? LSAT?

2. For how many schools is "estimated" data being used? We don't know. And, because we don't know, we don't really know how accurate these rankings are. This is a huge problem. As I've discussed before (perhaps not on this forum, but anyone that knows me can tell you that I get pretty vocal about this), rankings dictate a number of things, chief among them what a school believes a reasonable tuition cost is. It also determines how employers view graduates of a certain school, which then affect employment chances. If this is all based on "estimated" data then, excuse my French, what a CROCK.

3. This is a pretty critical area to be messing up and "estimating" on. Granted, in this case the employment numbers were "estimated" at lower than they actually were, but how many schools submitted estimates that were higher than the actual figures? In an economic climate like the one we're currently in, fudging the figured when it comes to employment data seems like a pretty shoddy deal. If nothing, else, this should be one area where USN is MORE vigilant than usual.

Of course, I keep harping and harping on how rankings shouldn't be the deciding factor for anyone, that it should only be, at best, one of many, many factors someone considers when choosing a law school--but who am I kidding? Everyone uses rankings as their primary (although hopefully not only) method of discernment: applicants, schools, employers, administrators. The fact that these rankings are based on estimated data (even if only for a few of the calculating elements) is shockingly irresponsible. Even more irresponsible is that this item was simply a blip on the USN radar, getting no more than a brief blog post. Worst still is that no one seems to have picked it up. Scary, scary, scary.

[Anne]

No one's replied to this (which is kinda sad...rankings are such an essential part of the admissions process ), but I thought I would add a little more info to this academic fire:

Paul Caron, a professor at the University of Cincinnati College of Law, has a pretty awesome blog called the TaxProf Blog. He routinely posts some very interesting information and insights on it. This morning's post was no different: "Did 16 Law Schools Commit Rankings Malpractice?" LINK: http://taxprof.typepad.com/taxprof_blog/2010/05/did-16-law-schools.html The article itself is short and sweet, but it leaves you with plenty to think about.

Excerpt: "...because U.S. News uses round numbers in determining a school's overall score, the 4% weighting of the employed at graduation data easily could impact a school's overall ranking -- i.e. a school whose overall score ended in .49 would move up to the next grouping with an increase in its overall score of merely .01.  Ted also explains that an increase of 22 percentage points in the employed at graduation figure would have improved a school's overall score by one full point (in the 2007 rankings) -- which Arkansas-Little Rock and John Marshall (Atlanta) could have achieved by declining to disclose their 58.3% and 52.2% figures, respectively, and instead allowing U.S. News to assign them 65.8% and 63.3% figures."

Take a mo' to read it. Even if you're not interested in Tier 3 and 4 schools, this whole "guesstimating of the rankings" is a good read.