Advertisement

If you have an ACS member number, please enter it here so we can link this account to your membership. (optional)

ACS values your privacy. By submitting your information, you are gaining access to C&EN and subscribing to our weekly newsletter. We use the information you provide to make your reading experience better, and we will never sell your data to third party members.

ENJOY UNLIMITED ACCES TO C&EN

Publishing

New tool detects AI-generated chemistry papers

Machine learning tool can distinguish human- and AI-generated introductions in chemistry journals with at least 92% accuracy

by Krystal Vasquez
November 16, 2023 | A version of this story appeared in Volume 101, Issue 38

 

Chemists from the University of Kansas have developed a machine learning tool that can detect with at least 92% accuracy when the introductions of chemistry papers are written using ChatGPT (Cell Rep. Phys. Sci. 2023, DOI: 10.1016/j.xcrp.2023.101672).

To distinguish between human- and AI-generated text, the tool uses a machine learning algorithm to analyze 20 writing style features, such as the presence of common punctuation marks and the inclusion of some chemistry-specific terminology. The new detector was trained and tested on the introductions of papers published in 10 journals from the American Chemical Society and introductions generated by ChatGPT based on the ACS papers’ titles or abstracts. ACS publishes C&EN.

The researchers say they focused on the introductory sections since that’s the part of a paper that authors are most likely to write with an AI text generator. “The reason why we developed [this tool] is because we wanted to understand how common it is for people to pass off AI writing as their own,” says the study’s coauthor, Heather Desaire.

During the tests, the new AI text detector outperformed existing detection tools such as the text-​classifier produced by OpenAI, the maker of ChatGPT. The OpenAI tool distinguished between human- and AI-written introductions with only 10–56% accuracy. In contrast, the chemistry-specific detection tool was 98–100% accurate.

The detector was also able to successfully spot AI-generated introductions derived from chemistry journals that weren’t initially included in its training set between 92% and 98% of the time.

Advertisement

Article:

This article has been sent to the following recipient:

0 /1 FREE ARTICLES LEFT THIS MONTH Remaining
Chemistry matters. Join us to get the news you need.