Metal-organic frameworks scrutinized for reproducibility

The topic of experimental reproducibility has been a concern for the fields of psychology and biomedical science in recent years. Now, a team of Georgia Institute of Technology chemical engineers has extended the scrutiny to materials science.

Jongwoo Park, Joshua D. Howe, and David S. Sholl have taken a look at the reproducibility of data collected on how much gas metal-organic frameworks (MOFs) adsorb. Their conclusion: Little reported data has been reproduced, and researchers need to focus on more consistent measurement protocols and reporting standards to help advance the field (Chem. Mater. 2017, DOI: 10.1021/acs.chemmater.7b04287).

Thousands of MOFs—porous compounds made of metals coordinated to organic ligands—have been prepared over the past 25 years with proposed applications as catalysts, separation materials, and gas-storage materials. Reproducibility of MOFs will be critical to their eventual use, for example, as hydrogen storage materials for fuel-cell-powered cars.

The Georgia Tech team assessed adsorption isotherms, which quantify the amount of an adsorbing species that clings to the surfaces of the porous materials at a certain temperature and pressure. The researchers focused on CO₂ adsorption, comparing the results of peer-reviewed MOF studies collected in a publicly available database curated by the National Institute of Standards & Technology. They evaluated the experimental data and conducted molecular simulations to compare how well different isotherms collected under comparable conditions match for the same MOFs. And they did so without attempting to judge which materials are better or which measurements are correct, or to assign causes to any observed variations in the measurements. The researchers emphasize that their analysis, drawn from some 28,000 isotherms on 6,000 adsorbent materials, represents a limited picture of MOFs, in part because the database includes only a fraction of the data measured globally.

Overall, the team found that about 20% of the literature data is not reliable because of the methods used, and only 15 of the MOFs evaluated have sufficient data to allow strong conclusions to be drawn about reproducibility. But the researchers note these numbers should improve over time as more data is added to the database. While the results have immediate implications for the characterization of gas adsorption in porous materials, Sholl believes the results are more important for demonstrating a widely applicable approach to assessing reproducibility in materials chemistry.

NIST’s Daniel W. Siderius, who helps maintain the MOF database but is not a coauthor on the study, stresses that the data are still “quite sparse” but that the Georgia Tech researchers did an admirable job tailoring the analysis to the sample size and have a sound technique in spite of that restriction. “The adsorption community will find the results to be quite interesting, as it is one of the first rigorous attempts to evaluate reproducibility and provide a general sense of uncertainty in isotherm measurements,” Siderius says.

His takeaway: The results “should be alarming to the community,” particularly as MOFs have been proposed as the basis for important future technologies. “The study should motivate the development and dissemination of best practices, standard operating procedures, and data-reporting conventions that would help eliminate unreliable data.”

Omar M. Yaghi of the University of California, Berkeley, who has helped establish MOFs as a new class of materials, says he’s unsurprised by the results given that adsorption isotherm measurements are dependent on variation in sample quality.

Yaghi points out that the trends in the data are consistent in showing the physical process of CO₂ adsorption, even if the sets of measurements are not perfectly reproducible because of defects, impurities, and experimental and experimenter inconsistencies. The sheer volume of data and number of MOFs are “a recipe for variations and error, especially since it is being done by different labs around the world,” Yaghi says. “How is this different than any other human endeavor, science or otherwise?”

Yaghi notes that the materials research community has been evolving intuitively to study these variations for the most promising adsorbents and zooming in on them, even without the new analysis. He believes the analysis should not change the conclusions about the potential utility of MOFs.

Sholl agrees. “I suspect that thinking about this reproducibility issue will generate a lot of interest and spark a wide-ranging discussion in the chemistry community,” he says.