Advertisement

If you have an ACS member number, please enter it here so we can link this account to your membership. (optional)

ACS values your privacy. By submitting your information, you are gaining access to C&EN and subscribing to our weekly newsletter. We use the information you provide to make your reading experience better, and we will never sell your data to third party members.

ENJOY UNLIMITED ACCES TO C&EN

Reproducibility

Reactions

June 15, 2019 | A version of this story appeared in Volume 97, Issue 24

 

Letters to the editor

Reproducibility versus replicability

I am responding to “Research Reliability Must Improve” (C&EN, May 13, page 16). In my opinion, in the context of computational chemistry, the emphasis should be on replicability instead on reproducibility. Apart from problems involving only integers, exact reproducibility is impossible to achieve in most cases since it would require the use of the exact same source code, compiler, operating system, and hardware, as any change in any of the above could affect floating-point operations. A further obstacle is the frequent use of commercial or proprietary software. Even if a result is successfully reproduced, it only verifies that the first calculation did not make a mistake in the inputs or did not involve cheating, and this is of little scientific impact.

Replication of a result is much more useful. Replication involving the same algorithm as the original software but a different implementation verifies both implementations. Even more useful is replication using a different algorithm since that can provide verification of the theories involved as well or, in case of disagreement, start a discussion about the source of discrepancy, be it theoretical or software related. In fact, presentation of a new theory or algorithm should, whenever possible, either run some of the tests that replicate existing results obtained with a different theory or algorithm or include the replication of the new result with a different algorithm. Given the number of approximations involved in most computational chemistry work, note also that consistency with a different algorithm is a better argument for the correctness of the theory or algorithm presented than agreement with experiment.

Furthermore, I would like to emphasize the importance of consistency checks not only on the final results but at every level of the calculation: (a) checking input data for values outside the permissible range and any possible inconsistency (for example, hydrogens having more than one bond), no matter how unlikely or unreasonable it sounds and (b) performing periodic consistency checks during the calculation that exploit redundancies in the data structures (for example, in neighbor lists, check if i is a neighbor of j, then j is a neighbor of i).

Mihaly Mezei
New York City

Views expressed on this page are those of the author and not necessarily those of ACS.

CORRECTION

This letter was updated on June 19, 2019, to correct the example of a consistency check: if i is a neighbor of j, then j is a neighbor of i. The original said, “Then I is a neighbor of j."

 

Advertisement

Article:

This article has been sent to the following recipient:

0 /1 FREE ARTICLES LEFT THIS MONTH Remaining
Chemistry matters. Join us to get the news you need.