If the ChatGPT conversational robot manages to bluff humans, it is unmasked by a relatively simple computer tool designed by the Montreal firm Draft & Goal.
Online since the beginning of the week, the ChatGPT Detector analyzes any text of at least 400 characters and slices in less than five seconds. In the ten trials of The Press, he consistently spotted the ChatGPT-generated texts with 100% certainty, considering the five real articles to be human-written. To add a layer of difficulty, the articles were automatically translated into English by another artificial intelligence tool, Google Translate. The detector only works in English for now.
Profitable discovery
Behind this tool, we find an administrator trained in engineering, Nabil Tayeb, and his co-founder Vincent Terrasi, a graduate in engineering in project management and software engineering from the Graduate School of Computer Engineering in France. The two men based in Montreal began to take an interest in artificial intelligence generating text at the end of 2019, first with the aim of offering marketing tools for analysis and writing blogs or web pages.
PHOTO PROVIDED BY DRAFT & GOAL
Nabil Tayeb, CEO and co-founder of Draft & Goal
“We had made a small module that could tell if a text automatically generated and edited by a human was still perceived as AI, explains Nabil Tayeb, CEO, over the phone. After a while, we realized we were super good at detecting AI. »
Put aside, this “small module” came back to life with the public launch of ChatGPT at the end of November. The unprecedented verisimilitude of the chatbot’s responses has fascinated millions of users, but sparked fears that it could be used to plagiarize or misinform.
“We thought we could make our AI detector a product, we prepared that during the Christmas holidays, says Mr. Tayeb. We released a beta version this week. [préliminaire] to let people use it and form an opinion. »
According to the first tests, the ChatGPT Detector has an efficiency rate of 93%. A second version to be launched next week should reach 96%, and the detector will accept French within two weeks.
We started in English because it was the path of least resistance, because there is a lot more content in English on the internet and the tools are mainly in that language.
Nabil Tayeb, CEO and co-founder of Draft & Goal
Even though he got a perfect score in the tests of The Press, the Draft & Goal tool still has weaknesses, specifies the CEO. “There are still types of texts which are marginal cases and which are more difficult to detect. And some texts with a lot of short sentences can be a problem, but we think we have found the solution. »
ChatGPT flaws
But how can such a simple tool, designed without deep learning by two people with the support of two developers and a few advisers, defeat a chatbot like ChatGPT which has benefited from hundreds of millions of dollars of investment and of the work of dozens of experts since 2015?
It’s that despite its impressive performance, a conversational robot like ChatGPT still has problems. “In many cases, when there is consensus or the questions do not require a high level of interpretation, we will have correct answers, explains Mr. Tayeb. But on certain themes, the AI will make mistakes or will simply invent sources. »
Without revealing his recipe, the CEO specifies that his detector analyzes five signals, weighted differently, to trace the intervention of the AI.
He tips his hat to his co-founder, Vincent Terrasi. “He’s really, really good, he’s been working on AI for a long time. Is it easy? Like everything, it’s easy once you know how to do it. There are perhaps a dozen projects already on the market that detect AI, and we can suspect that there are many more in research laboratories. »
If he tracks down his weaknesses, however, Mr. Tayeb believes that ChatGPT really constitutes an impressive “inflection point”, “the beginning of the era of consumer AI”. Hence the need for a detector that prevents its use for misleading or unethical purposes. “There are going to be changes. Is it going to be for the best, or be dystopian? History will tell us. »