It has been written that ChatGPT generates detailed responses that are comparable in quality to what an average student can write. Therefore, the application raises questions about plagiarism and its identification by teachers, as well as the process of reflection and its evaluation with current tools.
And yet.
Three statements
I teach Quebec literature at the college level. As an intellectual exercise, I present my students with three statements that relate to a work that we have just read and studied. The students choose one, analyze it, make connections with the knowledge acquired in the course, organize the structure of their answer then its content and, at the end of the process, write an answer of around 900 words in correct French.
Out of curiosity, I subjected the ChatGPT application to such statements. What takes my students several hours of reflection lasted only a few seconds here. First, I asked if, in The sisters-in-law of Michel Tremblay, it was fair to say that “sorority dominates” within relationships.
In a text of just over 250 words, roughly speaking, the application replied that “in sum, sisterhood is a major theme in The sisters-in-law by Michel Tremblay and [que] the piece highlights the importance of female friendship and solidarity in an often unequal world”.
I submitted the application to a second statement, comparative this time. I asked if “Marie-Ange and Rose experienced a similar alienation”. This time, the app replied that “Both of them are therefore alienated by their marriage and their life as housewives, and they find comfort and help from their friends, who share their worries and hopes. They also stand together in the face of life’s difficulties and support each other in difficult times. »
Finally, the argument based on a little too many generalities, I wanted the application to clarify its answer by providing me with proof of what it was saying, like my students. So I made a third and final statement: “Is it fair to say that in The sisters-in-law of Michel Tremblay, modernity is idealized? Discuss and prove with excerpts. »
This is where the answer provided is most thought provoking.
“Among these women, some idealize modernity and aspire to a better, freer and more independent life. This is particularly the case of Marie-Ange, who dreams of traveling and discovering the world, and who feels trapped in her life as a housewife. She says in particular: “Me, what I would like is to see the country. I’m tired of being locked up here taking care of the house. I like to see new things, meet new people. I would like to be free.” (Act I, scene 2).
For her part, Rose dreams of a freer and more independent life, and she idealizes modernity as a way to achieve it. She says in particular: “Me, what I would like is to have my own life, my own house, my own furniture, my own laundry. I would like to be free.” (Act I, scene 3).
In sum, it can be said that in The sisters-in-law by Michel Tremblay, modernity is idealized by certain characters who aspire to a better, freer and more independent life. These characters see modernity as a way to achieve their dreams and personal fulfillment. »
Four observations
Four observations can be made from this brief experiment. First, as announced, the application did indeed remember the previous questions relating to sorority and alienation, as well as the structure which invited to compare the fate of Marie-Ange and Rose.
The other three findings are a little more troubling. First, it is most striking, the quotes taken from the work are pure inventions. All these sentences used as evidence simply do not exist in The sisters-in-law. Next, anyone familiar with the work and its context knows that an inescapable characteristic of Tremblay’s piece, created in 1968, is the use of the joual by its author, a peculiarity which caused a scandal at the time, but which is ignored by the application in the creation of proofs. Finally, last element, the references used to locate these quotations in the work refer to marks of writing that the author did not use: the scenes are not numbered in this play. In short, the evidence is false and the references are decoys.
Superintelligence and critical thinking
Superintelligence, as defined by Nick Bostrom, is “any intellect that greatly exceeds the cognitive performance of human beings in all possible domains”. We are still in the era of low intelligence and “dumb” algorithms, in the sense that machines are blind and do not understand anything, neither what they say nor what is asked of them. They simply aggregate information into the deluge of data with which they are stuffed. ChatGPT didn’t read Tremblay and blindly forged evidence to answer my query. Shameless. Without ethical concern for the truth.
If a student had had his work written by this application, I would have been able to discover the subterfuge, because I know the object of study very well. In a few years, in a few months perhaps, we will be somewhere else. The algorithms will be more efficient, and the question will arise again. And this time, maybe I will have a less clear-cut point of view. I have concluded that the evidence is false and the references are decoys: for the moment, ChatGPT does not pass the Turing test in literature. But would that have been the case in a field where I don’t have that expertise? Would that have been the case with a super-intelligent machine that would have read and understood Tremblay? How to know ?
The challenge of machines is the same as that of humans: you have to know how to discern the true and the false. You have to use a critical mind. So there is still a future in education. At least, for a while.