Brouhaha over Peer Review

Peer review--it's virtually sacrosanct in science. Peer review of a completed scientific report by other specialists in a particular field serves as a "gatekeeper" determining which studies are sound enough to print. And it involves a collaborative process through which reports, and even the science itself, are revisited--and, with hope, improved--in response to reviewer comments. Peer review is generally held in high esteem across the disciplines of the natural and social sciences.

The federal government, too, embraces peer review. Science agencies use peer review as they determine which applicants will get research grants. And increasingly, regulatory agencies are peer reviewing scientific documents, such as risk assessments, that guide establishment of rules and standards. The Environmental Protection Agency, in particular, has strengthened its peer review program in recent years.

Now, the White House is proposing government-wide standards for peer review of science used for federal regulation (C&EN, Sept. 8, 2003, page 13). This plan would define when and how government documents would undergo peer review.

Many in academe and industry endorse the idea of peer review in the federal government. But whether a centralized, prescriptive peer review system, emanating from the Office of Management & Budget, will actually improve the quality of regulatory science--and at a reasonable cost--is the subject of intense debate.

Critics of the proposed system say the plan will seriously hamper the government's ability to regulate and that it paves the way for industry-dominated peer review panels. Of particular concern is a provision that would prevent any scientist receiving a federal grant from serving as a peer reviewer for the agency that provided the funding. Critics also worry that, under the planned policy, regulatory agencies such as the Food & Drug Administration and EPA won't be able to respond quickly to urgent public health concerns unless they get special dispensation from OMB to waive peer review. Those critics also say the White House has failed to demonstrate that there is a problem with regulatory science that could be remedied with peer review. They suggest that OMB scrap its plan, then start afresh through consultations with the scientific community on how best to apply peer review to regulatory documents.

Supporters of the plan, who come mainly from industry, praise the proposal and say it will improve the scientific basis for regulation. Plus, they want OMB to make the peer review system even more prescriptive than proposed. They want the public to have input into the "charge"--the statement from a regulatory agency that describes the focus of a particular review--and the selection of the peer reviewers. And they think the policy should cover more types of government information than OMB has proposed.

OMB RECEIVED 187 responses to its Aug. 29, 2003, proposal on peer review. Groups filing comments ranged from the National Academy of Sciences (NAS) and large scientific organizations, including the American Association for the Advancement of Science (AAAS) and the Ecological Society of America, to the National Association of Funeral Home Directors. Also offering responses were industry groups, including the American Chemistry Council (ACC), the American Petroleum Institute (API), and the Pharmaceutical Research & Manufacturers of America; public interest groups; and individual scientists. The American Chemical Society did not file comments on the plan.

Bruce Alberts, president of NAS and chair of the National Research Council, endorses the idea of strengthening the federal government's peer review processes but raises questions about the OMB plan. "The highly prescriptive type of peer review that OMB is proposing differs from accepted practices of peer review in the scientific community and, if enacted in its present form, is likely to be counterproductive," Alberts says. "We suggest that the proposed bulletin be framed in terms of general principles and objectives rather than rigid procedures." He offered NAS consultations to OMB and the White House Office of Science & Technology Policy on a revised version of the guidance.

The proposal could hinder the federal government from seeking advice from NAS, Alberts says. "It has been suggested to us that some agencies may be restricted from seeking our reports reviewing scientific information because of the prescriptive nature of the draft OMB guidance."

Henry Kelly, president of the Federation of American Scientists, says the planned guidelines "are likely to provide new targets for litigation by anyone unhappy with regulatory decision." And Ronn G. Smith, who chairs the Wyoming Air Quality Advisory Board, says: "This initiative could raise the barrier to incorporating scientific judgment in the regulations. If too high, this barrier could lead to deficient public policy."

Much of the discussion about the OMB proposal focuses on a provision in the draft guidelines saying information that was subject to peer review by "a respected scientific journal" would not need further peer review if those data were used to support a federal regulation. However, the presumption that journal peer review is adequate for federal regulatory purposes could be rebutted.

Many industry groups argued against this provision, saying all scientific data supporting regulations need federal peer review. Journal peer review "is to screen out material which has clear defects or is not significant," says the Center for Regulatory Effectiveness, an industry-supported think-tank involved in drafting the federal information quality law enacted in 2000. This process focuses on whether scientific material should be "disseminated to the scientific community where it can then be subjected to further scrutiny and attempts to replicate and validate its findings and conclusions." Thus, journal peer review is not designed to support the reliability of published material--"in contrast with the apparent purpose of federal agency peer review."

Regulatory Checkbook, a conservative-leaning policy group that encourages the use of the best available science and economics in regulation, says, "Scholarly peer review is fundamentally different than governmental peer review." According to this organization, "Government peer review is closer to a consultant-client relationship."

ACC says: "It is absolutely essential that OMB's peer review requirement apply with equal force to journal articles. This will ensure that the science they present is subject to the requisite degree of scrutiny, and that members of the public have some ability to challenge the findings of a published study."

However, ACC notes that most government documents that would fall under the OMB policy are not articles that have appeared in peer-reviewed journals. Instead, they are science documents, such as risk assessments, compiled by agencies.

But sometimes a published study does influence regulation, ACC points out. An example is a report by researchers at Harvard School of Public Health that linked exposure to fine particulate matter in air with illness and death in urban settings. EPA relied on this study in 1997 when it tightened air quality standards for fine particulates.

That regulation was controversial--though ultimately upheld. It also sparked debate when the federally funded researchers refused to turn over their data to the public because of confidentiality concerns. This controversy led Sen. Richard C. Shelby (R-Ala.) to tack an amendment to a spending bill. That rider, which became law in 1998, requires that data generated through federal grants be made available to the public through the Freedom of Information Act.

Some in industry say that a more open peer review process is needed for regulatory science, in contrast to journal peer review that is often done by reviewers whose identities are unknown to the author.

"Scientists have to clearly understand that once science enters the public policy arena, it must be made as transparent as possible," the U.S. Chamber of Commerce says. "While the traditionally used closed peer review process in which peer reviewer identities are withheld from public scrutiny may serve the science community well, it is not in the greater public interest."

But Rutgers professors of public policy Stuart Shapiro and David H. Guston say the situation isn't that simple. "There is a tradeoff between the incentives that anonymity gives peer reviewers to provide impartial feedback and the government's need for openness and transparency," they say.

Several groups involved in industry and commerce recommended avenues for public input during the peer review process, which others argued will severely impede the ability of agencies to regulate.

For example, ACC and the National Association of Realtors believe that agencies should give the public a chance to comment on the selection of peer reviewers, the charge to peer reviewers from the agency, and any information that the reviewers receive. API also suggests that peer review may be appropriate for an agency's response to public comments on a scientific regulatory issue.

ANOTHER PROVISION in the draft guidance advises agencies against selecting peer reviewers who are receiving or seeking "substantial funding from the agency through a contract or research grant." ACC praised OMB for raising "the critical, but previously neglected, prospect that peer reviewers may have compromised independence because of their ties to agencies, either as employees or due to financial dependence." But the provision drew a prickly response from academics.

"OMB should not presume that scientists are tainted by receipt of research funding from a particular agency," says the Council on Governmental Relations (COGR), an association of more than 150 research universities. "More appropriate would be a focus on excluding potential reviewers with a professional or personal financial interest in the industry which is the subject of the anticipated regulations."

Alberts of NAS says, "Were we to exclude as committee members all those scientists whose research is or has been funded by the National Institutes of Health when providing advice to that agency, we would severely weaken the expertise needed to provide [the agency] with sound advice." Shapiro and Guston of Rutgers point out that one of the goals of grant programs at some regulatory agencies "is to create a cadre of experts to contribute to informed policy debate." And the Ecological Society of America adds, "By excluding these researchers, OMB would exclude the very people most likely to provide a useful review to the agency."

Another item of contention concerns the scope of the information that would fall under the guidelines.

ACC argues that the sweep of OMB's proposal is not large enough because it would cover only science documents supporting regulations that have a possible impact of more than $100 million annually. "Federal agencies routinely issue work products that do not in and of themselves have a cost impact of $100 million a year yet may possess other attributes that make them good candidates for additional scientific scrutiny," ACC says. These include EPA's guidelines for conducting risk assessment on chemicals for either cancer or noncancer health effects and the agency's Integrated Risk Information System, a database that provides EPA's expert judgments on how much exposure to a chemical is safe.

COGR takes a different view. "The scope of the proposed guidance is broad and not justified in comparison to the resources it will consume both on the part of the federal agencies and the research community," the group contends.

The Center for Regulatory Effectiveness says OMB should extend the proposed peer review requirements to all information that is disseminated by the government. Peer review should not merely apply to regulatory documents such as risk assessments, it argues.

Some question whether the scientific community can provide all the peer reviewers that may be needed for agencies to abide by the OMB draft guidance. "Where are all of these experts going to come from, and who is going to pay for all of their time?" asks Martha K. Raynolds, a research associate in geobotany at the University of Alaska, Fairbanks.

The American Enterprise Institute-Brookings Institution Joint Center for Regulatory Studies suggests that OMB explore ways to pay peer reviewers a reasonable wage for their time. "It is unlikely that reviewers will do much to improve the quality of regulatory analysis unless they are given adequate incentives to do so," says the center, run jointly by two policy analysis groups--the conservative-leaning American Enterprise Institute and the more liberal Brookings Institution.

Many see the OMB proposal on peer review as a solution offered to a problem that has not been defined. Shapiro and Guston offer a detailed analysis of the proposal using the same standards OMB uses in its reviews of federal regulations. Shapiro, who formerly worked at OMB reviewing regulations, and Guston fault the White House office for failing to demonstrate that existing peer review mechanisms have not always been sufficient to ensure the reliability of regulatory information.

In addition, "Peer review is no guarantor of the truth. The problem will be even greater for economic analysis ... since there is less consensus in the social sciences than the natural sciences about what constitutes appropriate work," the Rutgers scholars say. "Disagreement in the sciences is common. Disagreement in economics is rampant."

Alberts of NAS also points out that peer review does not eliminate scientific uncertainty. "Very high quality, peer-reviewed scientific research articles and reports by highly respected research teams can, and sometimes do, reach differing conclusions and results on substantially the same research subjects," he says. "It is simply characteristic of the initial difficulties often encountered in charting the unknown."

SOME CRITICS of the plan argue that it will hamper the government's ability to respond to urgent situations. AAAS expresses concern "that FDA may unintentionally be delayed in removing a questionable drug or therapy until after an additional level of peer review has been completed." The guidelines would allow an agency to skip or delay peer review before taking action in such situations--but only if OMB grants a waiver.

The proposal "would strip agencies such as EPA, FDA, and the Centers for Disease Control of the authority to use their specialized experience to act quickly in response to emergencies," says OMB Watch, a watchdog organization. "OMB has never held authority to make critical health and safety decisions" and should not establish this power through the peer review guidelines, OMB Watch says.

Some business groups oppose the part of OMB's plan that would allow waiver of peer review for science supporting regulations that agencies must issue before a court-imposed deadline. Sometimes those court-ordered deadlines are established in settlement negotiations between an agency and the group suing it to change or issue a regulation. Therefore, OMB should require agencies to take into consideration the time needed for peer review when they negotiate deadline schedules for consent decrees, ACC and the Chamber of Commerce say.

Several policy think-tanks and some in industry are urging OMB to conduct a cost-benefit analysis of the draft guidelines. ACC is among them and says the proposal should be feasible to put in place and likely would provide net benefits. The chemical manufacturers' organization says any additional time and resources that agencies need to follow a standardized peer review process are "a reasonable cost of having a responsible and fair government."

As part of their analysis, Shapiro and Guston conducted a cost-benefit assessment of the proposal. They conclude that the main cost of the new guidelines is not the price tag of conducting peer review but the delay of regulations that will have a net benefit to society. They estimate OMB's proposal would cost about $325 million a year to implement. Holding back regulations that provide large net benefits to society could raise costs higher, they say.

Meanwhile, the Joint Center for Regulatory Studies suggests that OMB come up with a way to assess whether peer review of scientific documents used for regulation actually improves the quality of federal regulations.

The debate over the proposed peer review guidelines covers a plethora of issues, some apparently unforseen by OMB. The White House office originally had planned to issue a final version of the peer review guidance by February. But in light of the public comments, OMB has moved that date to "several months later," an OMB official tells C&EN. She did not elaborate.

In the meantime, many are waiting to see if OMB will form new links with the scientific community and industry to discuss peer review--or whether it will forge ahead with its plans on its own.