Is Peer Review Honest?

AN ASSISTANT PROFESSOR at a state university submitted a paper to a well-known chemistry journal. The paper quickly received two positive reviews, then a very tardy negative review. After a revision, the submission again languished in the hands of the negative reviewer.

That reviewer, the professor suspects, was a senior colleague who was doing experiments similar to those chronicled in the paper. The professor had requested samples from the colleague to further the work but never received them. When the frustrated assistant professor contacted the journal editor and asked whether the colleague was indeed the tardy reviewer, the paper was accepted nine hours later.

That wasn't this assistant professor's only experience with the sharp end of what appears to be conflict-of-interest impropriety. The professor submitted to a federal agency a grant application that didn't get funded. A year later the professor noticed that someone from the agency's study section was working on the same topic. "I don't think it was that novel an idea—it was an obvious thing to try," the professor acknowledges of the proposal. But in light of a seeming conflict of interest, "it's a tough thing to swallow."

Separately, another associate professor had run into some troubling behavior on the part of his senior colleagues. In a grant review study section that he was part of, he witnessed a friend failing to recuse himself from reviewing an application of a competitor. The friend gave the application a poor review and the grant didn't get funded. "He may have had reasonable criticisms," the associate professor says, "but he shouldn't have been reviewing that grant."

Neither scientist felt comfortable being named for fear of professional repercussions. But when younger faculty members are asked about their experiences with peer review, stories such as these are common. Many have similar anecdotes of their own or have heard them from others.

In some cases, such as the grant review study group situation, there is fairly good evidence of wrongdoing. But more often the evidence is circumstantial, such as the assistant professor's suspicion about why her paper was held up in review and why her grant wasn't funded. Researchers say the vague nature of such cases makes it difficult to make accusations. There's also often a power imbalance, in that a junior researcher will often be accusing an older, more established colleague—someone who might be asked to review future papers or grant applications or weigh in on a tenure decision.

WHEN JOURNALS and funding agencies are queried about whether there is a problem with peer reviewer integrity, the response is often a firm "no." Peter J. Stang, editor of the Journal of the American Chemical Society (JACS) and a chemistry professor at the University of Utah, says that in six years he hasn't seen any significant problems with reviewers, although he notes that any peer review system will have some people who abuse it.

Peter Gölitz, editor-in-chief of Angewandte Chemie , says his journal has ways to prevent and address problems with reviewers. Authors can exclude up to five possible reviewers when they submit a paper, and "we honor such requests unless there are compelling reasons not to do so," Gölitz says. Papers generally go out to three reviewers; if their reviews are divergent, the editor can adjudicate or choose to send the paper to an additional referee.

Biochemistry Editor Richard N. Armstrong, a professor of biochemistry and chemistry at Vanderbilt University, in Nashville, says the procedures for his journal are similar, although authors may exclude only up to three potential reviewers. Both JACS and Biochemistry are published by the American Chemical Society, as is C&EN. Armstrong also says that journal editors may enter notes about specific reviewers in the ACS electronic manuscript tracking system. Such notes can include potential conflicts of interest. When asked whether Biochemistry has had any problems with reviewer behavior, Armstrong says he occasionally gets a complaint from a suspicious author but has never found any evidence of impropriety.

Some funding agencies also say they don't see a problem with reviewer integrity. Antonio Scarpa, director of the National Institutes of Health's Center for Scientific Review, says that in more than two years he's received perhaps two or three vague and unsubstantiated complaints about reviewer behavior.

The rosters of NIH grant study sections are public, and researchers applying for grant applications can ask that particular scientists not review their grants. The scientific review administrator decides whether to honor the request. After review, if applicants believe their grants were judged unfairly, they can appeal to an advisory council for re-review by a different study section.

The National Science Foundation receives about 45,000 grant applications annually. Complaints of grant-related misconduct are sent to the Office of the Inspector General (OIG) for investigation. James T. Kroll, head of administrative investigations at OIG, says that about once a year NSF has a substantiated case in which a reviewer plagiarizes material from a grant application that he or she reviewed. In the seven years Kroll has been at OIG, the office has investigated two or three cases in which reviewers have allegedly not excused themselves from reviewing grant proposals with which they had competitive conflicts of interest. None of those cases was substantiated, he says.

Yet the perception of reviewer wrongdoing persists, especially among younger researchers. Older faculty, when queried by C&EN, say they've never experienced anything like the stories told by their junior colleagues. "In all the years I've been publishing science, I've never encountered an instance where I thought I was being treated unfairly by a reviewer or editor," says Vanderbilt's Armstrong. "Sometimes I've disagreed, but I never came to the conclusion that someone was out to get me."

Because researchers are often afraid to report suspicions, many cases of potential wrongdoing are never investigated and bad feelings can linger. Such under-reporting also makes it less likely that journals or funding agencies can learn the full extent of ethical problems.

The current difficult funding climate also tends to enhance perceptions of wrongdoing. Gregory A. Petsko, a professor of biochemistry and chemistry at Brandeis University, in Waltham, Mass., notes that the source of such perceptions may be mistrust stemming from increased anxiety, rather than increased wrongdoing. "People are very anxious and very frightened," Petsko says, noting that the current funding climate is the worst he's seen in a 35-year career.

The anonymity of peer review may also play a role. "The fact that peer reviewers are basically anonymous gives them cover," says Janet D. Stemwedel, an assistant professor of philosophy at San José State University, in California, and an expert in scientific ethics. At the same time, she says that outright reviewer misconduct may be less of a problem than unconscious actions to protect one's advantage at the expense of a competitor. "I'm not really optimistic about individuals' abilities to be aware of their own conflicts of interest," she says.

However, reviewer wrongdoing is almost surely a relatively uncommon event in the chemical research enterprise. "The community of chemists I know has a very, very high level of integrity," says Nobel Laureate Elias J. Corey, an emeritus professor of chemistry at Harvard University.

Nevertheless, it may take only a few to harm the overall reputation of many. Also, the perception, particularly among less established scientists, of more widespread reviewer impropriety hurts morale at a time when young researchers already may be discouraged. Corey would advise his colleagues to be mindful of the Golden Rule—to treat others as you would like to be treated—and emphasizes that trust is critical in chemical research if it is "to maintain its vigor, value to our society, and intellectual excellence."

» Comment on this article