As the Graduate School progresses in its review of academic programs, scholars are questioning the validity of a new set of national rankings released last fall.

Yale’s directors of graduate studies recently used the National Research Council’s latest review of graduate programs to compare their departments to others across the country — but that data has recently attracted criticism from a committee of American university sociologists. The sociologists released a report in February asserting that the NRC rankings of sociology graduate programs are flawed and inaccurate, and in interviews with the News, five Yale directors of graduate studies also expressed discontent with the methodology of the rankings.

Frances Rosenbluth, deputy provost for the social sciences and faculty development, said goals of this year’s analysis of graduate programs include making sure programs have sufficient support and the appropriate number of students given the job market. Administrators use various rankings as rough indicators to make sure departments are performing well, Rosenbluth said, but are careful to avoid accepting them at face value.

“We keep an eye on them just to make sure our departments are doing well,” she said, “but we take them all with a grain of salt.”

The NRC is part of a private, nonprofit institution known as the National Acadamies that operates under a Congressional charter to provide policy advice.

As part of Yale departments’ self-studies, departmental administrators analyzed the accuracy of the NRC rankings themselves, and professors in fields other than sociology also questioned their reliability. David Post, director of graduate studies for ecology and evolutionary biology, pointed out that the NRC collected data between 2006 and 2008, so the rankings did not help his analysis since his department has grown significantly since then.

Maurice Samuels, director of graduate studies for the French Department, and Sociology Department chairwoman Julia Adams said the NRC used statistics for their departments that were incorrect. Adams said the number of Yale sociology faculty reported was “wildly off-base.”

“Many scholars are saying that what matters is not this round of NRC rankings, which are already past and widely discredited, but making sure that this is the last round ever conducted in such a poorly thought-out fashion — particularly for the humanities and social sciences,” Adams said.

Andrew Perrin, a sociologist on the university committee that criticized the NRC rankings and a professor at the University of North Carolina at Chapel Hill, said the NRC study — which yielded a multitude of rankings based on schools’ reputations and performance in about 20 areas, as well as illustrative rankings tailored to the interests of individual users — contained biases in favor of certain kinds of academic programs.

For example, he said, in the rankings for social sciences and sciences, the NRC did not account for how many books a professor wrote or how often that professor’s books were cited, instead counting only articles as academic output. Perrin said this penalized sociological programs with an emphasis on theory over data analysis.

Charlotte Kuh GRD ’76, deputy executive director of the Policy and Global Affairs Division at the NRC, acknowledged that the absence of a measure for books written is a “valid criticism.” Kuh said the NRC decided against counting books for the social sciences because it was was prohibitively expensive.

“We decided that the way we did it was the best way to go,” she said. “If they would rather do rankings really specific to sociology, then they should do this study themselves.”

Kuh said she hopes universities will use the individual sets of data — such as time to degree and doctoral completion rates — that the NRC provides instead of focusing solely on overall rankings.

Graduate School Dean Thomas Pollard said that while “finding flaws in the NRC process is easy,” the council collected data that may prove useful for some schools.

Pollard said the directors of graduate study used only select NRC data in their reports to compare Yale’s programs to “higher-ranked programs.” Pollard said he has refrained from using NRC data in his separate, overarching review of Graduate School programs.

Though faculty and administrators said they are aware of the limitations of NRC data, Perrin said he fears the study could negatively impact universities if administrators decide to alter their programs in order to improve their rankings. Though encouraging professors to write more articles and less books would improve a program’s NRC rank, he said, it could stifle creative ideas.

“The reality is that provosts and so on are watching rankings very closely,” he said. “The concern is that university administrators will see this as a critique that requires fixing.”

The 2010 NRC report ranked 4,838 doctoral programs in 62 fields at 222 American universities.