Our education system is often described as catastrophic, in tatters. At the same time, our international PISA test scores have been admirable for 20 years, test after test, especially in mathematics.
Once again this year, the results of this exam by the 15-year-old students are admirable.1. If the catastrophic Quebec system ends 1er in Canada and 8e in the world in math, both among the weak and the strong, imagine the state of the network elsewhere…
But these results from Quebec are doubtful, we hear. The sample does not seem reliable and could include too many private schools, as if the dice were loaded.
This feeling could be exacerbated this year and not only because of the strikes. According to what I discovered, Canada and Quebec did NOT meet international PISA sampling standards.
Ah good ! So this proves the deception? Not so fast.
In summary, for the results to be valid, the participation rate of the 29,234 randomly selected students in Canada must reach 80%, otherwise the PISA consortium calls them into question. Have the weaker ones been absent?
However, the response rate was below 80% in 7 of the 10 provinces. Alberta has a response rate of only 62.9%, British Columbia 73.1% and Ontario 79.1%.
Quebec is at 79.3%, a bit below the 80% mark.
The phenomenon is explained by the remnants of the pandemic in 2022, the examination having been carried out in spring 2022 (the results were published on December 5, 2023).
That proves the subterfuge, right? Wait, wait.
Given these below-standard rates, the PISA consortium required a detailed analysis of the sample to be carried out. More specifically, the organization asked Statistics Canada – the grand master of statistics – to compare the profile of responding students to those who did not respond. For Quebec, we used the scores of the two groups on the provincial exams.
Conclusion: Students responding to the PISA exam have a better profile than non-respondents, so that Canadian PISA scores could be higher than the real strength of all students. The consortium still publishes the results, but specifies that they must be “treated with caution”2.
Ah ha! you might say, that’s the scam! Still a little patience.
Canada is not alone in this boat. The United States, United Kingdom, Denmark, Australia and 8 other participating regions around the world are in a similar situation.
Quebec ? Each of the 7 problematic provinces have results that are probably too high for their English-speaking students, but for Quebec, in particular, the analysis of this bias is not conclusive for French-speaking students, according to Statistics Canada3.
In other words, the scores of French-speaking respondents from Quebec in math are considered reliable. In short, no statistical deception for most of our students.
To convince the remaining skeptics, know that the schools are not selected by the government, but by an external body mandated by the PISA consortium. The sample is intended to be representative and the selection is made randomly.
Another element: it is true that certain students are completely excluded from the exam by school staff, ultimately, because of an intellectual, physical or language incapacity or because they were in virtual mode, which is prohibited by PISA. Except that the exclusion rate in Quebec is 2e lowest in Canada, at 1.7% (the Canadian exclusion rate is 4.4%).
Intellectual disability is cited 2 to 3 times more often in English Canada than in Quebec as a reason for exclusion (0.8% in Quebec compared to 1.8% in Ontario and 2.8% in British Columbia)4.
So much for the doubts. Admirable results now, which focus mainly on math this year (in addition to reading and science).
The average score of young Quebecers in math was 514, compared to an average of 497 in Canada. Quebecers in the French-speaking network had 515 and those in the English-speaking network, 500.
The result of Quebecers is judged to be statistically higher than that of other provinces and the average of the 80 regions of the world that participated (score of 472). The scores are expressed on a scale based on the year 2003, the average of which was 500 among participating countries.
The United States, France and Sweden, countries that some revere for various reasons, had significantly lower scores of 465, 474 and 482.
Another angle of analysis: the proportion of students who do not pass the minimum required to participate fully in life in society (level 2 of 6 levels).
In Quebec, 17% of our students do not reach this level 2, compared to an average of 22% in Canada. New Brunswick, Saskatchewan, Nova Scotia and Newfoundland have more than 30% of their students failing level 2. Ouch!
Elsewhere, it’s just as bad. France is at 29%, Germany at 30% and the United States at 34%, twice the rate in Quebec. With such a proportion of weak students, imagine the composition of the classes!
But the real disaster lies elsewhere. In Mexico, two thirds of 15-year-old students do not pass level 2, a level which is the basis for understanding and functioning well, let us remember. And in Morocco, from where we siphon a large part of the most talented for immigration, this rate is 82%! Dramatic!5
That said, almost everywhere, results are down, mainly due to COVID-19. In Canada, the decline has been gradual over the 6 PISA exams since 2003, but the decline in 2022 is much more marked, at 497.
By comparison, Canada scored 518 in 2012, the last year when math was the main subject of PISA. Quebec went from 536 in 2012 to 514 in 2022.
That’s it for the results.
I do not question the numerous testimonies from teachers about the difficulties of so-called regular classes in Quebec, victims of skimming from private schools and some from the public which offer special programs (international, etc.). Victims also, in my opinion, of the failings of certain parents, but that’s another story.
But I tell myself that there is something that escapes us, seeing the enviable scores of Quebec for 20 years.
In any case, congratulations to the teachers and school teams who manage to do so well in a context that is not always easy. And I wish you – I wish us – a good understanding!
1. PISA is the Program for International Student Assessment (PISA), administered by the Organization for Economic Co-operation and Development (OECD).
2. Another problem, the response rate of the selected schools, which must be 85%, was lower in 2 provinces, namely Quebec (69.9%) and Alberta (57%). However, Statistics Canada’s detailed analysis concluded that there was no significant difference (or non-response bias) between responding and non-responding schools.
4. The diehards will say that the greater dropout in Quebec explains the good results, but experts argue that dropout mainly affects students over 15 years old, i.e. after the age of the PISA test, because school attendance is compulsory up to 16 years old.
5. Among the 7 participants who surpass Quebec in math, there are two developed regions of China, Hong Kong and Macao, which represent only 0.6% of mainland China. Taiwan and Japan are very strong and Singapore, in first place, has absolutely exceptional results (average of 575).