In a groundbreaking study published on November 6 in Cell Reports Physical Science, researchers introduced a machine-learning tool capable of discerning chemistry papers authored by the ChatGPT chatbot. This specialized classifier, demonstrating superior performance compared to existing AI detectors, offers a promising avenue for academic publishers to identify papers created by AI text generators.
Heather Desaire, a co-author and chemist at the University of Kansas, emphasizes the tool's targeted approach for heightened accuracy, stating, "we were really going after accuracy" by focusing on a specific type of paper. This deviation from a general detector approach suggests that tailoring software to distinct writing styles could significantly enhance the development of AI detectors.
The ChatGPT detector, initially described in June, employs machine learning to analyze 20 features of writing style, including sentence length variation and word frequency. The latest study focused on training the tool using introductory sections from ten chemistry journals published by the American Chemical Society (ACS). Notably, the tool achieved 100% accuracy in identifying ChatGPT-3.5-written sections based on titles and 98% accuracy when titles were absent, relying solely on abstracts.
When compared to other AI detectors, including ZeroGPT and an OpenAI text-classifier tool, the ChatGPT catcher outperformed in identifying AI-written introductions. Notably, the system exhibited a high degree of specialization for scientific journal articles but faltered when faced with articles from university newspapers.
Debora Weber-Wulff, a computer scientist specializing in academic plagiarism, commended the study's use of stylometrics on ChatGPT. However, she cautioned against viewing AI-detection tools as a panacea for broader issues in academia, such as the pressure to rapidly produce papers. While the ChatGPT detector marks a significant advancement, it is essential to address the underlying societal challenges in academic writing and research.
