C&EN Talks With Leonard P. Freedman

Twenty-eight billion dollars. It’s a shocking amount of money, almost as much as the annual budget of the National Institutes of Health. But it’s how much a recent estimate suggests the U.S. science enterprise is spending each year on preclinical medical research that cannot be reproduced.

Irreproducibility is not just a problem because money is misspent on research results that can’t be trusted, explains Leonard P. Freedman, president of the Global Biological Standards Institute, a nonprofit organization that advocates for the use of standards and best practices in research. It also delays the development of potential treatments and adds to already soaring drug development costs.

Despite its importance, “nobody had really put a dollar figure on it,” Freedman says. So he and his coauthors estimated those costs in a paper published in June. They also tried to pinpoint the causes of the reproducibility problem and propose solutions (PLOS Biol. 2015, DOI: 10.1371/journal.pbio.1002165).

The good news is that a small change could make a big difference in the amount of irreproducible research, Freedman says. “Even if you cut the irreproducibility rate in half, the return on investment would be huge.”

In recent years, the irreproducibility problem has drawn growing attention from the community, especially NIH. But little has changed in how scientists do their work. Freedman and his colleagues at GBSI sought to show that the reproducibility problem requires action, he says. And nothing grabs people’s attention like money. So he teamed up with two Boston University economists—Iain M. Cockburn and Timothy S. Simcoe—to try to put a dollar figure on the reproducibility crisis.

The authors started with previous estimates of the amount of irreproducible preclinical research, which soar as high as 89%. To be conservative, they decided to use 50% for their calculations. Industry, academia, and nonprofits spend $58.4 billion on this research each year. Half of that—$28.2 billion—was their estimated cost of irreproducible research.

It’s a fairly simple calculation, Freedman says, but the number is a good starting point for discussion. “The intent was to be a little bit provocative, but also to be as accurate as we could be.”

After the paper came out, some critics said the authors overestimated the problem. But Freedman says even if the estimate is high—and he’s not convinced it is—the problem is still worth the research community’s attention.

“Is it $28 billion? Even if it’s $10 billion, it’s an enormous number,” he says.

Freedman and his colleagues also attempted to identify the reasons for irreproducibility. The top cause they identified was improper use of biological reagents and reference materials, which can affect a researcher’s findings. Other sources of irreproducibility were bad study design, data analysis and reporting problems, and inadequate laboratory protocols.

Each of these could be addressed before research is published if scientists make changes in how they do their work, says Freedman, who had careers in both academia and the pharmaceutical industry before starting up GBSI. But that’s unlikely to happen voluntarily, he says. “There are only two major leverage points: the funders and the journals.”

Both groups have started to address the problem, primarily through voluntary guidelines. Freedman has been surprised to see how difficult it is for funders and journals to make the requirements mandatory. “From where I’m sitting, until some of these guidelines or recommendations are required, compliance is going to be low.”

Training the next generation of scientists may be the best way to avert problems that lead to irreproducible research. The current model of a principal investigator mentoring graduate students doesn’t always lead to the changes needed to improve the quality of research, Freedman says. For example, he met a fifth-year cell biology graduate student who didn’t know about a simple assay that could solve the problem of cell misidentification. “We really need to reexamine how we are training students,” he says.

Many researchers consider irreproducibility a problem just in academia, but Freedman says pharmaceutical companies also need to take it seriously. As drug companies cut back on internal R&D, more and more of their basic research is coming out of academia. So are future drug company employees.

“If I were still in pharma,” he says, “I would be pretty concerned about these issues.”