Clarifying Review Criteria

Revisions to the two criteria the National Science Foundation uses to evaluate every grant proposal it receives were issued last week by the National Science Board (NSB), the agency’s oversight body. The clarifications are intended to better guide grant applicants in addressing the “intellectual merit” and “broader impacts” criteria as well as to assuage members of Congress about the value to society of NSF’s basic research funding.

The revisions are part of a report issued by the NSB Task Force on Merit Review, after a nearly two-year study of the grant review process. NSB approved the report—which also includes principles and guidance on how the agency should use the criteria—at its December 2011 meeting.

“NSF is now charged to implement the enhanced Merit Review Criteria, which affects every aspect of NSF’s business,” NSB Chairman Ray M. Bowen writes in a memorandum to the report.

According to the report, the intellectual merit criterion encompasses the potential to advance knowledge. “All NSF projects should be of the highest quality and have potential to advance, if not transform, the frontiers of knowledge,” the report says.

It attempts to clarify the more contentious broader impacts criterion. The report defines this criterion as encompassing the potential of a given research project to benefit society and contribute to the achievement of specific, desired societal outcomes. The report cites broadening participation in the science enterprise by underrepresented groups and describes the societal benefits of new scientific knowledge and technologies as activities that meet this criterion.

One of the problems with the broader impacts criterion, which was introduced in 1997, is assessment, says Alan I. Leshner, an NSB member and cochair of the task force that produced the report. “We really don’t know how effective these broader impacts were in the past,” he says. Leshner is the CEO and executive director of the American Association for the Advancement of Science.

To try to address the assessment problem and other shortcomings, the report provides in-depth discussion of the two criteria and makes recommendations as to how NSF can guide principal investigators (PIs) to ensure that both elements are part of grant proposal submissions. For example, the task force recommends that NSF include a broader impacts section on grant application documents.

Many scientists, especially older scientists, have complained for years that the broader impacts criterion is a superfluous distraction—a box to check—or that it is unreasonable to request that PIs spend time and money on activities tangential to basic research.

“But even in the pursuit of fundamental knowledge it is important that we engage in research that will be of benefit to society,” says ACS Immediate Past-President Nancy B. Jackson, a Sandia National Laboratories executive who supports strengthening this concept and who has testified to Congress about its importance in the merit review process. “We, scientists and researchers, are supported by society, and we should recognize that support by ensuring our research benefits society.”

The report, however, “doesn’t really advance or demystify the concept of broader impacts,’’ says former NSF Chemistry Division director Luis A. Echegoyen, now a chemistry professor at the University of Texas, El Paso. He has high praise for the task force’s work—and strongly supports the concept of broader impacts—but he says the report lacks teeth in terms of giving NSF an enforcement mechanism.

These changes come as Congress is demanding to see the benefits of NSF-funded research, Jackson says. “I don’t see NSF changing the intellectual quality of the projects it chooses,” she says, “but this is a change in the culture that surrounds how we go about funding science.”