Canadian Citizenship Test | ChatGPT easily graduated

What if artificial intelligence had, on paper, what it takes to obtain Canadian citizenship? According to an exercise conducted by The Pressit is indeed the case.

To pass the Canadian citizenship test, a newcomer must obtain a minimum score of 75%. It is a question of collecting at least 15 correct answers on 20 questions, comprising four choices each.

In order to work with a meaningful sample, The Press obtained a document of 50 questions created to measure, from a specialized site. This maintains a regularly updated database containing several hundred questions that can be asked during a real test.

ChatGPT found the correct answer to 41 of these 50 questions, for a more than satisfactory score of 82%. And this, in the space of about fifteen seconds.

Thus, the conversational agent was able to respond adequately to questions relating to the history, laws, government and symbols of the Canadian federation.

Among his good answers, he was able to identify the inventor of the snowmobile, the origin of the word Canada or the minimum age to vote, in addition to having understood the process of selecting senators and the reason of the celebrity of the Bay of Fundy, for example.

Basically, “this exam is used to assess your knowledge of Canada and the privileges and responsibilities of Canadian citizenship,” reads a video on the government’s official website. Adults aged 18 to 54 must take it.

For Canadians who are more in the category of “Mr., Mrs. Everybody” and who have never had to take it… is passing the citizenship test easy? It turns out not, according to a recent survey.

This week, the Léger firm did the exercise with 1,512 Canadian citizens: only 23% of them achieved the passing grade. Their cumulative average was 49%, which is far from the result for obtaining citizenship.

It explains

When comparing human results with those of ChatGPT, a clear winner stands out. Artificial intelligence is one step ahead.

So, could people taking the test – online – ditch the good old study manual and take the easy route? This issue, already present in institutions of higher education, does it now extend to examinations held by the federal government?

“Before the examination can begin, customers must confirm their identity through their personal information, they must take a photo of themselves and their personal identification documents using a webcam” , responds Immigration, Refugees and Citizenship Canada (IRCC).

“The system used for the citizenship test can also detect if the applicant has opened other windows, tabs, browsers or applications during the test,” adds the IRCC.

Is an artificial intelligence that passes such a test synonymous with the imminent domination of machines over humans? Absolutely not, you can breathe.

“ChatGPT didn’t have to find nuances or answer complicated questions that test verbal comprehension,” says Laurent Charlin, senior member of the Quebec Artificial Intelligence Institute Mila and associate professor at HEC Montreal, in an interview. “I’ve looked at sample questions, and they’re relatively basic facts. Simplicity helps. »

“The other thing that’s important: chances are he’s already had access to some or all of these questions through his database,” he adds.

At the end of May, The Press just reported that ChatGPT had failed the Bar School exam miserably. Faced with complex questions aimed at testing the knowledge of university law graduates, the tool developed by OpenAI had obtained a poor score of 12%.

It is in the nuance and the complexity that the conversational agent is lost, affirms Laurent Charlin. “It’s a tool that has strengths, but is imperfect. He has an ability to answer questions in different areas, but sometimes says things that are completely wrong. »

“ChatGPT, we have not yet properly assessed all of its capabilities. You have to keep a fairly critical mind. He is capable of doing a lot of things, but we need more evaluation and exploration to see what he does little or does not do. »