NSF Clarifies Its Broader Impacts Grant Requirement

The National Science Foundation’s controversial broader impacts criterion is getting an overhaul. The action comes after years of confusion about what is required and complaints that the criterion is inconsistently applied.

The revision was prompted by an analysis of grant review criteria that was released earlier this year by the National Science Board (NSB), an oversight body that sets policy for NSF. The board affirmed its support for the two fundamental NSF review criteria—broader impacts and intellectual merit—and made recommendations for how to interpret them.

NSF has incorporated the NSB findings into its annual update of its grantee handbook. This year’s changes—the most significant of which is the reworking of broader impacts evaluations—go into effect for grants due on or after Jan. 14, 2013.

The biggest change will be for grant reviewers, who will be asked to use one set of five aspirational questions to evaluate both broader impacts and intellectual merit.

Grant seekers will also have to change how they write proposals to more clearly delineate their projects’ broader impacts, in response to both the review changes and additional reporting requirements laid out by NSF. Under the new rules, proposals will include a specific section on how a grant meets the broader impacts criterion instead of including this information in the body of the grant as is now done. And successful grantees will have to gauge their progress on their broader impacts goals in annual reports to the agency, something that hasn’t previously been required.

“We care that proposals are of the highest quality, that they have a broader impact on societal goals, and that we can assess how effective they are,” says Joanne S. Tornow, deputy assistant director in NSF’s Social, Behavioral & Economic Sciences Directorate and executive secretary for the NSB task force.

NSF is also asking universities to provide more support for the broader impacts criterion.

In the past few weeks, NSF has been holding a series of meetings to get the word out to the scientific community about how the changes will impact grantees. “By and large, there has been a positive response,” Tornow says.

Seth M. Cohen, a professor and chair of the department of chemistry and biochemistry at the University of California, San Diego, says most of his colleagues understand that broader impacts goals are part of NSF grants and have learned how to work within the system. “I’m not sure if the change is going to make things better or just make people more nervous about it,” he says.

Intellectual merit has been at the heart of NSF’s mission—and its grants review process—since the agency first started awarding grants in the 1950s. It has had few fundamental changes over time.

Broader impacts became an explicit part of the merit review picture in 1997. And almost immediately, scientists began complaining that the criterion wasn’t clear. The agency had set out examples of broader impacts, which could be met through the research project itself or by working toward several societal goals defined by the agency, such as improving retention of women and minorities in science. But many scientists weren’t sure whether or how their projects had to meet any or all of those goals.

After several years, it also became obvious that different NSF divisions and peer reviewers were applying the broader impacts criterion differently.

In 2010, NSB decided to reevaluate its merit review criteria, in large part because of the ongoing confusion about broader impacts. At the time, NSF was also working on a strategic plan for the agency, and NSB wanted to make sure its review criteria for grants matched the agency’s larger goals.

“It was time to take a whole-cloth look at the review criteria and rationalize their descriptions and their use,” explains Alan I. Leshner, chief executive officer of the American Association for the Advancement of Science and cochair of the NSB Task Force on Merit Review.

To do that, the task force undertook an extensive evaluation of the intellectual merit and broader impacts criteria and how they were applied throughout NSF.

The task force gathered opinions from NSF employees, grantees, and other stakeholders, including the public. It examined regularly scheduled divisional reviews done by outside evaluators. It hired evaluators to examine thousands of grant applications to see how the broader impacts criterion was being used by scientists.

In the middle of the NSB review, Congress passed the America Competes Re­authorization Act, which includes language that requires, among other things, NSF to keep broader impacts and lays out several societal goals that the agency should be striving to achieve through this criterion.

Although the NSB task force had considered eliminating the broader impacts criterion altogether, ultimately it decided to keep both criteria—intellectual merit and broader impacts—because they exemplify the goals for the agency, Leshner says.

But the task force did aim to clear up confusion about broader impacts by outlining specific questions reviewers should ask—the same questions that they use to evaluate intellectual merit. “Most of the surrounding principles are the same,” Leshner says. “The report speaks to the importance of quality and a commitment to broader impacts.”

Although NSF is clarifying the broader impacts criterion, Tornow acknowledges that scientists may still be unsure how they should evaluate their work’s broader impacts, which is one of the questions reviewers will use to examine their grants. Tornow suggests that grantees think of evaluating broader impacts in the same way they think of evaluating their own research.

For example, NSF is not asking a researcher to know whether the curriculum he or she is developing leads students to win Nobel Prizes 20 years from now, she explained. But there should be some way to conduct a basic evaluation of how the students did.

UCSD’s Cohen is an advocate of broader impacts in general, and he has worked extensively on the broader impacts portion of his own NSF grants. But he wonders if the additional evaluation burden will make it too difficult to apply for NSF grants. “If you are going to add a much higher standard for broader impacts and make it a more onerous project, you need to increase the size of the awards,” he says.

One solution to make it simpler for scientists to meet the new broader impacts criterion, Cohen suggests, might be to allow principal investigators (PIs) to tie this piece of their grants to effective, already proven outreach programs.

NSF is implementing another change that may also help scientists more easily meet the revised criterion. The change, which is the result of a requirement in the America Competes Act, directs universities to certify that for any NSF grant their faculty members are applying for, the school will provide adequate support to help the scientists meet stated broader impacts objectives. Universities already have to affirm their commitment to providing laboratory support for scientists, such as animal care facilities, electron microscopes, or housekeeping services.

As a result of the change, universities may need to provide more resources to support broader-impacts-related activities, Tornow says. Those services could include hiring outreach or education staff who can assist in sculpting proposals, making connections in the K–12 community, or even designing projects that scientists could participate in.

“I think this clarifies what PIs should be thinking about: Do they have the resources on campus that they need to do their project, or do they need to ask for money to support it?” Tornow says.

NSB’s description of broader impacts sets lofty societal goals for NSF, ranging from increasing the public’s science literacy to developing a globally competitive science workforce. But Tornow emphasizes that those goals are for NSF as a whole. No one project is expected to meet all of them.

This clear explanation that broader impacts is an agency-wide portfolio objective is particularly good for scientists, says Howard H. Garrison, director of public affairs for the Federation of American Societies for Experimental Biology.

In the past, his organization has been ambivalent about the broader impacts criterion. The criterion is important for society, but it is difficult for individual scientists to meet such lofty goals in individual projects, given their short duration and limited amount of funding, he says.

“The change to measure broader impacts on a portfolio level makes a lot of sense,” he says. “It recognizes the value of those goals without tying it to each individual application.”

For now, Tornow is continuing to help scientists understand the changes. “We just have to watch how it plays out once people start submitting proposals.”