The R-score is the fairest known measure

The Quebec College Student Federation (FECQ) has published a leaflet entitled “Towards a reform of university selection” contesting the validity of this measure of academic performance at college, a position reported by the daily The Press in its February 15 issue. ^⁠1

In this document, the FECQ says it relies on “simulations” produced by the FNEEQ-CSN and the Fédération des cégeps to question the fairness of the R rating. The next day, still in The Pressthe FNEEQ-CSN demanded nothing less than an immediate moratorium on the use of the R rating, also relying on these “simulations” that it says it carried out in 2019. ^⁠2

In response to a request from the Bureau de coopération interuniversitaire (BCI), we examined the issue as external experts.

In its opinion entitled “For a reform of university admission”, the FECQ relies on the examination of three fictitious groups produced by the FNEEQ-CSN. In the analysis of these fictitious cases, which she wrongly calls simulations, she blithely confuses marks and academic performance, alleging implicitly that the former are a faithful reflection of the latter when we have known for more than 30 years that such it’s not the case. Moreover, a closer examination of the data presented in this opinion illustrates the extent to which these three fictitious groups are implausible since the variability of college grades is associated, in these groups, with only a little more than 3% high school grades when it is established that this association is at least 12 times higher. ^⁠3 It should also be added that this strong influence of previous academic performance on the success of studies is also strongly confirmed between college and university since the variability of graduation rates in baccalaureate programs is explained at… 99% by the R rating ! ^⁠4

Remember that at the college level, the evaluation of students is not, with the exception of the uniform language test, the subject of uniform ministerial tests as is the case, among others, in 4^and and 5^and secondary, so that to compare marks given by different teachers in groups with highly variable characteristics for multiple courses belonging to disciplines as different as, for example, mathematics, physics, sociology, literature and the visual arts, as these analyzes do, amounts to comparing apples and oranges, to use a well-known expression. Elementary methodological error!

For its part, the Fédération des cégeps has also produced an analysis of eight fictitious groups from which it questions the fairness of the CRC (College performance rating) currently in force, believing that it demonstrates that it contains biases that would favor students from strong homogeneous groups to the detriment of strong students from weak homogeneous groups, a position taken up in a slightly different formulation by the FECQ.

Here too, the fictitious cases presented are in fact only examples of fictitious groups which 1) do not represent typical cases of college reality, 2) are based on an incomplete understanding of the methods of calculating the R score which leads to errors in the assessments of these and 3) then result in comparisons of R score where we also confuse the score obtained, Z score and academic performance.

But there is more. A statistical analysis we conducted of the eight groups in question, six of which represent atypical distributions, shows that even for this small, unrepresentative subset, the current R-score remains the most reliable known measure. equitable academic performance of college students.

We are obviously aware that any evaluation measure can be improved, which is amply confirmed by the history of the changes made to the measurement of the academic performance of CEGEP students in order to make it increasingly fair.

Nor are we unaware that measuring the academic performance of college graduates is a sensitive subject that must be dealt with rigorously. At the very least, we share a common point of view with the various stakeholders in the file: a measure of academic performance is fair if it does not unduly favor or disadvantage a student based on the characteristics of the groups in which he is assessed.

However, for lack of having demonstrated, using real data that is not subject to selective sorting subject to confirmation biases, and a proven statistical approach to dealing with them, that the R score of students is statistically dependent on the strength and dispersion of the groups, the views expressed publicly in recent days as to the unfairness of the R rating are unfounded and biased. Unfortunately, they only fuel the inevitable rumors that accompany any evaluation measure, which undermine students’ confidence in the procedural fairness of university admission and which can lead them to choose programs or institutions. which are based only on illusions.

* Co-signatories: Fernand Boucher, retired registrar, University of Montreal; Martin Riopel, Vice-Dean for Research, Faculty of Education, University of Quebec in Montreal