Posted at 2:00 p.m.
In a recent article, The Press took up the results of a ranking drawn up by a British consulting firm which “propels” Quebec to the rank of “7and world power in AI”. This study was commissioned by Forum IA Québec, a non-profit organization created by the Legault government in 2020, whose board of directors is mainly composed of members linked to organizations directly involved in the artificial intelligence economy ( AI). Therefore, should we be surprised that the sponsored study makes Quebec a “leader” that equals, even surpasses, nations such as Germany and Japan? The President and CEO of Forum IA Québec can swear hand on heart that Tortoise is a “very neutral firm that applies a very rigorous methodology”, let’s take a closer look at the seriousness of this ranking.
Pompously baptized “Global AI Index”, the ranking of each country is in fact established by adding seven heterogeneous indices, defined arbitrarily and according to an opaque methodology.
This first observation is confirmed by a closer examination of the seven indices which are broken down into a myriad of indicators (143 in total) whose relevance, or even the relationship with the object supposed to be measured, is often difficult to determine. In this, the classification established by Tortoise tramples on two fundamental characteristics of any good indicator, namely the homogeneity of the composition and transparency.
Take the “research” index, in which Quebec ranks 5and in the world and which weighs 21.27% in the overall ranking. Why this number rather than 20, 22, or even 25%? We don’t know… The index certainly includes a valid indicator, namely the total number of scientific publications of a country in AI, but it mainly includes indicators that bibliometrics experts consider invalid because they are incorrect. constructed, such as the “h-index” of researchers or the non-normalized sum of citations received by articles published in AI from each country. Added to this are indicators that are pointless, such as the number of quotes received per million inhabitants! Not to mention those that have no connection with AI, such as the number of universities ranked in the top 100 from the magazine Times Higher Education… in the discipline of physical sciences. In addition to including outlandish indicators, the “research” index therefore does not respect another fundamental criterion of validity, namely correspondence to the object measured.
Let’s take another indicator, that of “government strategy”, for which Quebec also receives a high score. It depends in part on public investments made in AI and we know that both provincial and federal governments have invested nearly $1.2 billion in it since 2017. However, making money flow in a given sector does not in any way mean that we act as a “strategist”. Just think of the federal government’s “Innovation Superclusters Initiative,” one of which was dedicated to AI for an investment of $230 million over five years. According to a report published at the end of 2020 by the Office of the Parliamentary Budget Officer (PBO), the government has apparently not established any quantifiable indicator to measure the real effect of these “superclusters” on the increase in productivity of companies or on the creation of products or processes. The PBO concludes that it is unable to “say whether the innovation superclusters initiative does or will really accelerate innovation”. We have already seen better in terms of “strategy”!
We could thus multiply the examples of opaque or poorly constructed indicators, such as the inclusion of the number of people indicating “data scientist” in their LinkedIn profile to measure the index of “talent” in AI. We prefer to conclude by recalling the need to remain vigilant in the face of this type of improvised “rankings” which have no other function than to generate buzz media. These pseudo-scientific rankings are only marketing tools that serve to arouse the interests of a small number of promoters, while legitimizing a posteriori risky government policies whose costs will be borne by taxpayers and not by companies supposedly ” innovative”.