How to distinguish texts written by ChatGPT from those by humans? A detector achieves this, 100%.
Mathilde Fontez, editor-in-chief of the scientific magazine Epsiloon, tells us today about an impressive detector, which thwarts the already extraordinary performance of ChatGPT: how to distinguish texts written by artificial intelligence from those of humans?
franceinfo: American researchers have just developed a detector that can distinguish between texts written by humans and by AI?
Mathilde Fontez: Yes, big issue. We remember the historic screenwriters’ strike in Hollywood: a month ago, they obtained guarantees that a minimum of humans will continue to write the screenplays. But we still need to distinguish these texts from others, put controls in place…
And the problem also arises on the science side: most scientific journals have changed their publication guidelines to prohibit the use of AI – because ChatGPT is amazing, incredibly powerful, but it can sometimes “hallucinate” : write totally false things, without knowing why, and add untruths in scientific articles.
It is precisely on science that this team from the University of Kansas focused. She trained a program to study scientific articles in chemistry journals…
Is this an AI that has been trained to detect AIs?
Yes, it’s the AI that controls the AI – it’s a bit dizzying. It is a learning program, which was trained on 100 articles written by humans, and 200 written by ChatGPT especially for the occasion. Researchers have given it features to examine writing style: variation in sentence length, the frequency of certain words, punctuation marks…
And it managed to detect, in 98 to 100% of cases, texts that had been written by ChatGPT. He even succeeded with magazines for which he had not been trained.
This is the first GPT cat detector?
No, there were others. OpenAI, the company that designed ChatGPT, had already developed one. Another one is called zeroGPT. But their performance is much less impressive: they find AI in only 10 to 65% of cases.
The strong point of this new detector is that it is specific: it has been trained for a particular type of text: scientific publications in chemistry journals. And as a result, it is more precise. That doesn’t mean that he couldn’t tackle other texts if we modify his training.