Letters to the editor
Reproducibility versus replicability
I am responding to “Research Reliability Must Improve” (C&EN, May 13, page 16). In my opinion, in the context of computational chemistry, the emphasis should be on replicability instead on reproducibility. Apart from problems involving only integers, exact reproducibility is impossible to achieve in most cases since it would require the use of the exact same source code, compiler, operating system, and hardware, as any change in any of the above could affect floating-point operations. A further obstacle is the frequent use of commercial or proprietary software. Even if a result is successfully reproduced, it only verifies that the first calculation did not make a mistake in the inputs or did not involve cheating, and this is of little scientific impact.
Replication of a result is much more useful. Replication involving the same algorithm as the original software but a different implementation verifies both implementations. Even more useful is replication using a different algorithm since that can provide verification of the theories involved as well or, in case of disagreement, start a discussion about the source of discrepancy, be it theoretical or software related. In fact, presentation of a new theory or algorithm should, whenever possible, either run some of the tests that replicate existing results obtained with a different theory or algorithm or include the replication of the new result with a different algorithm. Given the number of approximations involved in most computational chemistry work, note also that consistency with a different algorithm is a better argument for the correctness of the theory or algorithm presented than agreement with experiment.
Furthermore, I would like to emphasize the importance of consistency checks not only on the final results but at every level of the calculation: (a) checking input data for values outside the permissible range and any possible inconsistency (for example, hydrogens having more than one bond), no matter how unlikely or unreasonable it sounds and (b) performing periodic consistency checks during the calculation that exploit redundancies in the data structures (for example, in neighbor lists, check if i is a neighbor of j, then j is a neighbor of i).
New York City
Views expressed on this page are those of the author and not necessarily those of ACS.
This letter was updated on June 19, 2019, to correct the example of a consistency check: if i is a neighbor of j, then j is a neighbor of i. The original said, “Then I is a neighbor of j."