Karen Robson and Robert S. Brown
The results of international education rankings known as PISA are in, as of December. Canadians may have learned that Canada had a top ranking (along with Korea, Finland, Estonia and Ireland) among OECD countries in reading but was outperformed by China and Singapore and Macao, a Portuguese-speaking resort city within China with a population of around 700,000.
But what do any of these results mean, and for whom?
As education analysts and researchers, we believe that much more needs to be known about student performance, and the contexts or characteristics that may be impacting student outcomes, than can be grasped through surface-level comparisons of different societies and educational systems.
As part of the Gateway Cities project at McMaster University, along with our research team, we spent some years examining the post-secondary paths of students in the Toronto District School Board compared to other large cities including Chicago
We found that whether or not high school students were admitted to post-secondary education was greatly influenced by large differences between cities such as the kind of track or academic stream that students enter in high school, whether or not they access special education programming and also their racialized identities.
In our comparative research, we noted significant complexities in looking at cities that have many things in common (large, diverse English-speaking cities with high immigration). We found,for example, that Black male students in Chicago were more likely to go to university than male Blacks in Toronto.
Given these findings, we question just how much societies can usefully learn from the crowded scene of country-wide reading results and league tables, particularly when limited variables are factored in. For example, PISA captures whether students are immigrants, but not whether they are studying in a lower academic stream or their racialized identity.
Predicting the future?
The OECD implemented PISA almost a generation ago, at a time when the use and credibility of one-time standardized tests or ranking tables were at their height.
Since then, the highest level of education for most developed nations has moved from secondary to post-secondary.
Outcomes for 15-year-olds are important — but only insofar as they predict future post-secondary achievement. The PISA process, however, reflecting the earlier focus on one-time standardized tests, has no ability to do this. The focus of PISA on three separate subjects likewise reflects the thinking of a different age that is worth re-examination.
The reader of PISA reports could easily conclude that reading, science, and mathematics are subjects that are completely unrelated to each other, and are only important in isolation.
Long-term cohort outcomes
Our examination of how all students in a single cohort or age group students do in the long term questions the current fixation on individual subjects without reference to each other. Achievement patterns between subjects is strongly linked, and measures that combine different subjects tend to be stronger predictors of post-secondary achievement than individual subject results.
For example, one of us was part of a team that studied learning and achievement outcomes of students who attended both Toronto District Sschool Board (TDSB) schools and the University of Toronto.
We found that achievement in each of the four mandatory Grade 9 subjects (English, mathematics, science and geography) was strongly connected to university graduation. However, a much stronger predictor was combining the number of Grade 9 courses passed, with the achievement of these four subjects.
When we studied learning and achievement outcomes of students who attended both TDSB schools and the University of Toronto we found that [achievement in each of the four mandatory Grade 9 subjects (English, mathematics, science and geography) was strongly connected to university graduation]– but a much stronger predictor was combining the number of Grade 9 courses passed with the achievement of these four subjects.
Note that these outcomes are based on entire courses rather than standardized tests. Similarly, University of Chicago research has demonstrated that ninth grade GPA is a stronger prediction of college graduation than standardized tests like the SAT.
Long term achievement
Rather than fixating on individual PISA reading, science and math results, we should be looking at how these subjects relate to each other in a composite of student achievement – in particular, how the subjects relate to longer-term post-secondary achievement.
Another problem is how PISA data enables trend comparisons over time within the same system. In Canada’s most recent 2018 report, PISA measures reading scores over time a range from 2000 to 2018 (showing significant decline); math scores from 2012 to 2018 (showing a small but significant decline); and science for only two unique years (2015 and 2018).
Without manually cobbling together the data from various reports, it is difficult to know just how much math scores are suffering in Canada – but it is around 20 points (a slightly larger decline than that seen in reading).
More data needed
Education researchers have pointed out that educational systems in Canada are often behind others globally in collecting data that would help researchers better understand issues affecting how equitably communities can access education.
Large gaps in Canadian data infrastructure make it nearly impossible to examine associations between various student characteristics and their educational outcomes. This results in a dearth of evidence to create evidence-based education policy.
While it is possible for researchers to examine some public-release data files, suppress Statistics Canada exclude “sensitive” variables.
Countries such as the United States and the United Kingdom collect race data in their PISA survey.
Visible minority status is discussed in reference to the 2009 data, but in no Canadian release since.
The 2012 questionnaire posted online does not carry race questions. The 2015 and 2018 questionnaires are not currently posted anywhere publicly (at least that we can find), but extrapolating from the [CMEC report, they did not carry any questions about race.
While PISA reports findings on low-income and immigrant students, glossing over this important factor of race associated with educational attainment and achievement in Canada severely compromises our understanding (and recognition) of systemic racism in the Canadian education system.
It suggests race does not matter, when an increasing body of evidence is strongly indicating that it does.
Instead of comparing Canadian students to those of similar age in Estonia and Macao, we need to be asking ourselves what these exactly data are telling us and perhaps more important, what they don’t.