Chemists like to experiment, just not with opening peer review

Peer review almost pushed Ulrich Pöschl to leave science. About 20 years ago, Pöschl, a chemist at the Max Planck Institute for Chemistry, was frustrated after reading yet another almost incoherent article in a peer-reviewed journal. The review system was supposed to improve good scientific studies and weed out bad ones before they were published. “Peer review was not really achieving what you would hope,” Pöschl says. Feeling disheartened, he says, “I was really almost about to quit science.”

Instead of dropping out, Pöschl funneled his frustration into action. In 2001, he brought together several colleagues to create one of the first science journals to use a system called open peer review. Traditional anonymous reviews are shared only with a paper’s editors and authors, but this new system aimed to be more transparent. The journal Pöschl helped launch, Atmospheric Chemistry & Physics, also allows authors to open their paper up for public comments.

Pöschl’s motivation “was really the desire to improve the flow of information between reviewers, authors, and readers,” he says. “Everyone would have more information to move forward.”

Dissatisfaction with the traditional peer review system has only grown since. It has prompted an increasing number of scientists and journals to experiment with ways to improve peer review. Many of these experiments focus on open peer review, which makes the process more transparent, both to authors and to the public. Some of the most prominent participating publications include the journals F1000Research and Nature Communications.

With open peer review, “we can assure ourselves that the quality-assurance process at journals is what we say it is,” says Tony Ross-Hellauer, who studies peer review and open science at the Know-Center, an Austrian data company.

Skeptics say that no one knows whether peer review is really broken because it hasn’t been studied enough. What’s needed, they say, is a change in the incentive system to improve reviews themselves by rewarding overworked reviewers for participating.

Many chemists are skeptical. They think the traditional peer review system is working well, or at least well enough. Only a handful of the hundreds of chemistry journals have experimented with new peer review paradigms, compared with dozens in biology and medicine. And surveys have shown that chemists are among the scientists least likely to support changes to peer review. C&EN asked experts why that might be.

Experiments in peer review

Research journals are one of the fundamental ways scientists communicate with one another. Beginning in the 1940s, journals began employing primarily anonymous peer review, in which a paper’s authors can see reviews but do not know who reviewed their paper. The reviews are never shared publicly.

But peer review in that form, some say, leaves something to be desired. Critics say the anonymity of reviewers encourages bad behavior: Without the fear of consequences, some reviewers include harsh language in their evaluations; they unfairly attack scientific competitors, and they discriminate, either consciously or unconsciously, against women and minority scientists. What’s more, the secrecy of the traditional peer review system prevents the public from understanding and trusting science, detractors contend. The increasing prominence of the open access movement—some scientists, funders, and members of the public who advocate that all published papers be immediately and freely available—has also brought a push for greater transparency in peer review.

Definitions of open peer review vary widely, but most are talking about two things: open reports and open identities. “Open reports” means that anonymous reviews are published along with an accepted paper. “Open identities” means that reviewers’ names are published along with a paper. Journals conducting open peer review experiments could be sharing reports or identities or both, and these disclosures are made either automatically or if an author opts in.

One prominent example of a journal practicing open review is F1000Research, a biosciences journal that requires open reports and open identities. The journal was founded in 2012 by publishing entrepreneur Vitek Tracz with the aim of “removing the mystery behind editorial decision-making,” says Rebecca Lawrence, managing director of F1000, which publishes F1000Research.

Papers submitted to F1000Research are automatically accepted if they pass a series of preliminary checks to confirm the authors’ identities and scan for plagiarism, among other things. Then the review process begins. Flipping peer review to the end of the process changes the way both authors and reviewers interact with the reports, Lawrence says. “It is much more helping the author improve the quality of the article rather than an antagonistic type of process.”

Revealing reviewers’ names also helps avoid many of the issues that plague anonymous reviews, such as disparaging remarks about the experiments, Lawrence says. Reviewers won’t say such bad things if their names are attached to their evaluations, she says.

Another example is Nature Communications, a multidisciplinary journal that, since 2015, has allowed authors to decide whether to publish reviews with their accepted papers. About 60% of authors agree to publish their anonymous reviews, says Magdalena Skipper, editor in chief of Nature and former editor of Nature Communications, both published by Nature Research.

That doesn’t hold across all scientific disciplines, however. Nature Communications lost several regular reviewers in chemistry when it decided to make reviews open, and most chemists decline to publish reviews of their papers, she says.

Still, Skipper thinks that peer review is evolving. Nature tried an open peer review experiment in 2006, which failed when only 5% of authors opted in. “There was essentially no appetite for this,” Skipper says. But attitudes toward open peer review are changing. Now “we are actually in very active discussions about extending this transparent peer review system” to other Nature publications, including the flagship journal, she says.

Whatever it decides, Nature will not make open reports mandatory or require that reviewers reveal their names. “What guides us in our thinking is the understanding of the needs of the different communities,” Skipper says. “I’m in favor of there being a choice. It is rare that in any walk of life one size fits all.”

Open reports and identities

In February, the Howard Hughes Medical Institute and ASAPbio—a nonprofit organization that advocates for change in scientific publishing—convened approximately 90 scientists, advocates, and journal editors to talk about the future of peer review.

A poll taken at the meeting showed great support for open reports, with over 80% of attendees agreeing that peer review reports should be published.

Open reports can be helpful for many reasons. At Nature Communications, even though reports are available, few people look at them. “But for those who are really interested, that information can be instructive,” Skipper says. Reading them can be excellent training for young researchers learning to write reviews. Researchers working in the same subject area can benefit from reviewers’ detailed critiques of previous experiments. Openness is also helpful if a paper ever undergoes scrutiny for research misconduct or fraud, because people can see the discussion between reviewers and authors.

Ronald Vale, a University of California, San Francisco, biochemist and Howard Hughes Medical Institute investigator, hopes more openness can also prevent the erosion of the public’s view of science. “This is good for science overall,” says Vale, who is also president of the board of directors of ASAPbio.

Attendees at the February meeting were less convinced about the value of publishing reviewers’ identities. This finding is in line with survey results published by the Know-Center’s Ross-Hellauer and colleagues last year (PLOS One 2017, DOI: 10.1371/journal.pone.0189311). That online survey of over 3,000 scientists showed that 60% of researchers believe open reports should be published by more journals.

But slightly over 50% of respondents say that open identities would make peer review worse, primarily because they fear junior researchers would be unable to give candid feedback on a senior author’s paper without fear of retaliation. At F1000Research, Lawrence says she’s seen more support than retaliation for critical reviews by young scientists, but she recognizes the problem. “We shouldn’t be condoning senior researchers reacting to junior researchers’ criticisms in such a way as to cause harm. We need to improve the way we interact with each other,” she says.

After the meeting, ASAPbio Executive Director Jessica K. Polka, Vale, and other colleagues published a commentary in Nature calling for journals to publish anonymous reviews (Nature 2018, DOI: 10.1038/d41586-018-06032-w). So far, over 300 journals, predominantly those in the medical and biomedical fields, have signed a letter agreeing to publish reviews.

Still, the scientific community isn’t ready to make the same call for open identities. After peer review experiments have been running longer, “the culture of peer review may start to change so more people may feel comfortable having their identity revealed,” Vale says.

Overall, the conference confirmed that “there is a lot of interest in making peer review work the best that it can,” Polka says. “Peer review plays such an important role in science and the basic knowledge we are building up over time.”

Critics and credit

Not everyone is advocating for science to move toward a new, more open system of peer review, though. Flaminio Squazzoni, a sociologist at the University of Brescia, in Italy, says scientists are advocating for open peer review without really studying it first.

“There are a lot of negative positions about peer review that are not well founded,” says Squazzoni, who has spent the last few years leading a group of scientists studying peer review. “It is not true that peer review doesn’t work. It is not true that it is dramatically biased.”

In 2014, Squazzoni founded the European Union-funded group Peere to help scientists study peer review—both the traditional system and experimental ones. The group has worked with journals to get access to data documenting their peer review experiments.

Squazzoni’s biggest concern with open peer review systems is that they might be driving out scientists who would normally be critical of studies. These are the people you want policing the quality of science, Squazzoni contends.

“Transparency is a positive, but there are other values that need to be considered,” he says. “It is very important that we don’t lose people because of the model we endorse.”

Scientists should be studying peer review experiments, Squazzoni says, but they should also spend more time looking at the current incentive system for reviews. “We can gain more by changing incentives around peer review than by changing the different models of peer review.”

Right now, “peer review at its base is an altruistic community action,” the Know-Center’s Ross-Hellauer points out. Scientists volunteer to serve as reviewers, whether it’s because they believe in community service or they want to improve the scientific literature, he says.

But journals often have trouble finding enough reviewers. A report on the state of peer review found that the number of review requests that scientists receive is increasing, even though the current system doesn’t give them credit for their volunteer work. Squazzoni says peer review might be improved if publishers and employers acknowledged reviewers for their efforts.

One idea is to incorporate credit for reviewing papers into tenure and promotion evaluations. The website Publons helps scientists track when they have reviewed papers for a particular journal, says Andrew Preston, cofounder of the website. And scientists are interested in getting credit for their work. Publons has registered more than 500,000 researchers who have collectively reviewed over 3 million papers.

Currently, the Royal Society of Chemistry’s journal RSC Advances has the third most reviews on Publons’s site, in part because RSC has integrated Publons citations into its review system, Preston says. That shows him chemists are interested in getting recognition for their reviews. But Publons hasn’t caught on in the larger chemistry community. “I see the field of chemistry as being more traditional,” he says.

Why chemists are skeptical

Chemists’ conservatism extends beyond explicitly getting credit for reviewing papers. Surveys consistently show that chemists are more skeptical than other scientists about changing how peer review works.

In Ross-Hellauer’s survey, biologists and life scientists were more accepting of open peer review than chemists. “The differences in research cultures are very real,” he says.

The American Chemical Society, which publishes over 50 chemistry journals, is keeping a close eye on open peer review experiments but hasn’t jumped in yet, says Sarah Tegen, ACS’s vice president for global journals development. ACS publishes C&EN.

While ACS is constantly working to make its peer review system better, it hasn’t yet seen the chemistry community call for a shift away from the traditional approach, she says. “If people were beating down my door, I would say let’s go ahead.”

No surveys show why chemists might be more skeptical of open peer review than other fields. Nature’s Skipper thinks that the problem is not a fear of innovation, because she has seen chemists embrace other innovations, such as social media.

Fields like genomics and high-energy physics, with large, shared data sets, are forced to collaborate and tend to be more amenable to open peer review than chemists, F1000’s Lawrence says.

Another explanation for chemists’ skepticism of peer review experiments could be the highly competitive nature of chemistry. Keeping your work secret “is a way of generating advantage,” Squazzoni suggests.

Or the reason might be that there has been no real driver for change. “It could just be inertia in terms of shaking up the system,” says Johns Hopkins University chemistry professor J. D. Tovar, who studies organic semiconductors. “I haven’t had any issues that would cause me to rethink or revisit the philosophy behind peer review.”

As more chemists learn about open peer review, they might want to try new things. Skipper says she has watched the broader scientific community move toward more transparency in publishing with great satisfaction, especially in light of concerns about reproducibility and the public’s increasing skepticism of science. As part of that, journals have required researchers to deposit data in public portals and more openly disclose conflicts of interest, Skipper says.

“Just as we expect and request transparency from researchers, we should offer that parallel transparency in our own process.”

With additional reporting from Tien Nguyen.