Revisiting Grant Reviews

With federal research dollars on the decline, Congress is asking whether projects being funded by government agencies such as the National Science Foundation represent the best possible science.

During a hearing last month, members of the House of Representatives Science, Space & Technology Subcommittee on Research & Science Education sought to better understand the merit review process and called on witnesses from large research universities, the American Chemical Society, and NSF to testify on the process’s strengths and weaknesses.

“We want to know if the current process spurs or stifles innovation, how award decisions are actually made after receiving peer review, and if there are flaws in the system that may be providing precious federal funds to lower-rated proposals over more highly rated proposals,” subcommittee Chairman Rep. Mo Brooks (R-Ala.) said. “In exercising its oversight role, this subcommittee must ensure that federal dollars are being spent on the best science.”

“I see this hearing as an opportunity to examine the real challenges that do exist in a very strong review system and to discuss current and novel approaches to overcoming those challenges,” Rep. Daniel W. Lipinski (D-Ill.) added.

The subcommittee called on NSF to testify because approximately 96% of the grant proposals it receives are evaluated under the merit review process. From these reviews, the agency determines which projects to fund. To be awarded a grant, a project not only has to demonstrate intellectual merit, but also must show potential for broader impacts on society.

Cora B. Marrett, deputy director of NSF, acknowledged at the hearing that the merit review system has room for improvement and noted that NSF has taken steps in this direction. She pointed out that the National Science Board, which governs NSF, will make recommendations later this year on whether the criteria NSF uses to award grants need revision. Proposed new criteria, for example, extend the broader-impacts criteria to include specific national goals, rather than a general description of how the research could benefit society.

But other witnesses cautioned against placing too many criteria on researchers. “We should not limit the imagination of investigators and researchers by having a large number of restrictions about how the money is going to be awarded,” said Jorge José, vice president for research at Indiana University. “We are going to constrain the freedom” that can lead to, for example, the creation of an industry like biotechnology, he explained.

Keith R. Yamamoto, vice chancellor for research at the University of California, San Francisco, said reviewers should be able to focus on the scientific merit of proposals and not be asked to also ensure that national goals are being met. “One of the great things about the merit review process is that there is not a lot of guesswork there, and we shouldn’t be asking reviewers to step outside of their expertise,” he said. Instead, he suggested, “agencies individually and collectively need to be paying attention to whether national goals are being addressed.”

Rep. Larry D. Bucshon (R-Ind.) asked whether the merit review process is fair. “As you know, there is a disparity in percentage of grants made to new investigators versus established [ones], as well as universities that traditionally get federal funding versus those that may be new to the process,” he noted. “How do we guard against the politics involved in this process?”

Safeguards exist, Marrett said. For example, reviewers must sign conflict-of-interest forms, program officers must undergo ethics training, and institutions must ensure there aren’t ethical violations. “We’ve put in place checks and balances to reduce the likelihood that there will be the kind of personal considerations that I know would always be a concern,” she said.

The subcommittee asked the witnesses for suggestions to make the merit review process more efficient. “We’re absolutely intrigued by anything that could help reduce the burden on program officers, on reviewers,” Marrett said. “The number of proposals we’re getting continues to climb.”

She cautioned, however, that efficiency should not come at the cost of helping researchers, especially new investigators, improve their science. Part of the process involves providing feedback to scientists about their proposals, Marrett said. “Program officers are hesitant to simply say, ‘Let’s try to move this as quickly as possible,’ because I think there is a responsibility to help people understand where the problems might have been, to help cultivate the ways in which they may develop stronger proposals.”

Nancy B. Jackson, president of ACS, noted that ACS’s Petroleum Research Fund separates the review process from the process of helping researchers improve their grants. “We recognize that [certain proposals] are just not going to get funded, so we spend a limited amount of time with them and try to do our education separately,” she said.

Lipinski asked the witnesses how effective they think virtual reviews could be, something Marrett noted NSF is exploring. “This would be a way to bring more people into the process,” she said.

“The idea of having virtual meetings is very exciting to me and to my colleagues because we are more likely to be able to be involved and participate in these kinds of activities,” Jackson said. “I think the kind of expertise they could tap into will only become greater and richer.”