«

»

Print this Post

Questioning PISA:  examine the purpose, not just the rankings

952_since_2000__the_oecd_has_been_evaluating_the_knowledge_and_skills_of_the_worlds_15_year_olds_through_its_programme_for_international_student_assessment__pisa__test

Intercambio #9, January 2017 – IDEA

Questioning PISA:  examine the purpose, not just the rankings

By Dr. Larry Kuehn, BCTF

PISA–what does it mean–Nov 2016-ref version

 

Every three years the Organization of Economic Cooperation and Development releases the results of international standardized testing it conducts in dozens of countries around the world. And every release of the test results creates a global explosion of comments on the state of education–most of them negative. The publication of the 2015 test results in December 2016 fits the pattern.  Politicians from the few countries that top the tables will pat themselves on the back for their schools doing well.  Most will find their countries further down the list and will point fingers and use the results to support policy proposals they claim will produce better future results.

 

The PISA (Programme for International Student Assessment) is an international examination given every three years to evaluate education systems by testing 15-year-old students.  PISA is a project of the Organization for Economic Cooperation and Development (OECD).  The OECD’s perspective on education is illustrated by its description of what education systems are included–in 2012 “65 economies” – not countries – participated.

 

In 2012 PISA assessed reading, mathematics and science, with some students also evaluated on “creative problem solving.”  The 2015 assessments included those areas, but with “collaborative problem solving” replacing “creative problem solving.”  Students were essentially given a computer simulation to identify how well they could “collaborate.”

As would be expected, testing across different countries, cultures and languages presents complex methodological issues.  A number of critiques call into question technical aspects, including one in the prestigious publication, Education Researcher (2016).  It questions three elements: sampling participants, the achievement estimation model, and measuring trends.

Of more interest, though, in understanding the impact of PISA is the assumptions and values on which it is based.   The quotes in this next section are from a collection of articles in a book called PISA Under Examination, edited by Pereyra, Kottoff and Cowen.  Drawing on their analysis, these are eight significant issues that should be considered in deciding how much weight should be placed on PISA results in directing change in school systems.

  1. PISA is a tool for control and governing globally within the political frame of neoliberalism.

In effect, nations are turning over to an international organization the definition of the purposes of education.  The OECD is made up of thirty of the most developed nations, and its view of education  focuses primarily on neoliberal conceptions of individual “choice” within a capitalist system.  PISA’s impact though goes far beyond the nations in the OECD.  Some 80 “economies” have participated in PISA since the first tests were applied in 2000.

Cowen encapsulates multiple uses of PISA in the control and governance of education:

PISA is ‘ranking’ knowledge.  Thus, like Olympic medals or world rankings in football, it can be used domestically as disciplinary knowledge for the governance of (educational or sporting) systems; it can be used domestically as ‘legitimation’ knowledge for justifying reform of whatever kind; and it can be used as a form of cultural triumphalism. (Cowen 2011, 262)

The rankings are the tool to translate the global framework into local decisions about education.

  1. PISA’s underlying thesis of is “reforming the future society by making people in the present.” (Popkewitz 2011, 32)

Curriculum traditionally focused on content that reflects cultural and social knowledge aimed at incorporating the young into an established nation or society.  PISA is not interested in curriculum and in no way tests students on their knowledge of a curriculum.

Rather, PISA is focused on “competencies” that are not confined within any specific cultural knowledge.  To some degree this is a practical matter; If you want to give a single test that will be applied in sixty or more “economies” with different languages and cultures, you must decontextualize it, or face legitimate claims that you are favoring one set of national curricular knowledge over others.

But the purpose of competencies goes beyond the practical issues of cross-cultural test-making. It assumes the dominance of the neoliberal, global economy and the preparation youth need to fit into it.  The conceit that flows from this is the assumption that the test-makers can identify “knowledge we do not know if we need to face a future we cannot foresee.”

Curricula are no longer the organization and specific order of content of specific school subjects; it is replaced by basic concepts, models and competencies.  (Lundgren 2011, 24)

  1. The PISA-expected future is a “Knowledge Society” based on production transformed so capital can move “from being located in tools and machinery to be in human competencies.” (Lundgren 2001, 21)

Proponents of the PISA claim to “measure school systems’ contribution to the competitiveness of the nation in the new global economic demands.”  (Popkewitz 2011, 33)

The restricted framework of PISA is reflected in the newest addition to the test–“collaborative problem solving.”  This is a computerized simulation whose value is reflected in a belief that the future economy will be based on the student becoming a collaborative problem solver in a digitally-based economy.

Should an education system be primarily focused on building competitiveness in the global economy? Or should other social and cultural purposes have place and priority?  Are there not, for example, other purposes of collective action and different images of a future economy that use collaborative problem-solving?

Rather than contributing to competitiveness in a global economy, might other conceptions, such as “emancipation” as in Brazilian education theorist Paulo Freire’s vision, be the purpose of education?

  1. The ‘competencies’ of the ‘lifelong learner’ make up the PISA image of what the student is and should become. (Popkewitz 2011, 37)

Popkewitz summarizes the meaning of ‘lifelong learner’ from a range of literature:

The lifelong learner embodies enlightenment qualities of reason and rationality (science) as a mode of life (en)visioned to express individuality as a life of never-ending processes of making choices, innovation, and collaboration.  Individual agency is the self-actualization and self-motivation to a life of choice….  The only thing not a choice being making choices. (Popkewitz 2011, 40)

Virtue is managing effectively the limits and opportunities of the environment through steering one’s performances in a continual feedback loop of self-assessment. (Popkewitz 2011, 41)

PISA, itself, is the collective image of its definition of the lifelong learner–steering performances in a three-year loop of tests and reports.

  1. Testing students on an international scale is a way of having numbers as a point of comparison. In the case of PISA it is seen as a tool for representing competitive strength in the global economy.

Numbers and comparisons easily transfer into sports metaphors of league tables.  The numbers appear to summarize complex actions and events.  Numbers give the appearance of objectivity, fairness and impartiality, although they have within them elements that are not really comparable. (Popkewitz 2011, 34)

In addition to the tests, PISA collects survey data on the student and family, as well as institutional factors that are used to explain differences in performances.  These factors include socio-economic status and inequities in resources provided for the institutions of the school systems.  These important explanatory features appear in reports after the release of the league tables, and get little attention compared to the initial announcements of rankings.

  1. The comparisons of country results in the league tables over time may disguise positive or negative changes in the performance on the tests.

Students in country A may have achieved higher results in their PISA scores than in previous tests, but country B also has improved results that are greater than those of A.   This may push country A down in the rankings below B, leading to quick, media judgments that the students in country A are now doing worse than in the past.

  1. Responses to PISA results in Latin America.

Schools in societies with comparable traditions, cultures and with similar resources devoted to education might be appropriately compared under some circumstances.  In contrast, as Troehler points out:

The world according to PISA is the globally harmonized world of interaction… But there exists nothing like the experiences of students across the world in contrast to experiences within the “specific cultural context of a single country,” for experiences are always situated within a specific cultural context.”  (Troehler 2011, 255)

The institutional bases of most of the OECD countries (Mexico excepted), have relatively similar levels of resources devoted to education. This is not usually the case with non-OECD countries whose students are participating in the PISA testing.  Student scores in those Latin American countries that participate in the PISA have generally been low, and fall far down the rankings.  A. Bolivar contends that these results have much to do with inequality and underfunding of public education in the region:

But it is senseless to compare the performance of the Latin American systems –plagued by gaps of inequality–with the performance of systems like Finland and the other more socially egalitarian countries.  Latin America will gain nothing right now by imposing high standards –internationally competitive–if they are not accompanied by an ‘internationally competitive” spending, especially for poor students, and if these students are not supported, together with their families from day one.  (Bolivar 2011, 71)

Changes made in Latin American countries, supposedly in response to PISA results have, in fact, followed political or ideological positions that existed prior to PISA tests, according to A. Bolivar.   The PISA rankings are simply used to justify these pre-existing positions.

  1. The worldview that is the base of PISA is antithetical to indigenous conceptions of the role of education.

Indigenous conceptions of education are culture and place based.  Education is the development of a person competent within the culture, speaking the indigenous language that holds cultural meaning and rooted in the specific place of the cultural experience.

The PISA conception of education is harmonized globally with competencies that are assumed to provide conditions for success wherever they are applied.

This conflict can be seen in the context of the Mexican education system.  The recent wave of neo-liberal education reforms are rationalized as responding to PISA and measures of a global standard of education.  It should not be a surprise that the greatest resistance to those education reforms are centred in the states with significant indigenous populations, as culturally relevant education would be eliminated under these reforms.

In conclusion, PISA can be challenged from a number of perspectives:  methodological and technical issues; a rejection of the “human capital” ideology on which it is based; the ranking system that is used for political purposes and the paving over of significant issues and differences; the neo-colonial process that imposes the perspectives of the most “developed” countries on all others; and a globalism that continues the centuries-long suppression of indigenous cultures.

PISA should be challenged on all these bases.

References:

Bolivar, A. (2011). The dissatisfaction of the losers:  PISA public discourse in Ibero-American Countries.  In Pereyra, M., Kotthoff, H. and Cowen, R., eds.  (2011).  PISA Under Examination:  Changing Knowledge, Changing Tests, and Changing Schools, 61-74.  Rotterdam:  Sense Publishers.

Cowan, R. (2011).  CODA.  In Pereyra, M., Kotthoff, H. and Cowen, R., eds.  (2011).  PISA Under Examination:  Changing Knowledge, Changing Tests, and Changing Schools, 259-264.  Rotterdam:  Sense Publishers.

Lundgren, U. (2011). PISA as a political instument:  One history behind the formulating of the PISA programme.  In Pereyra, M., Kotthoff, H. and Cowen, R., eds.  (2011).  PISA Under Examination:  Changing Knowledge, Changing Tests, and Changing Schools, 17-30 . Rotterdam:  Sense Publishers.

Pereyra, M., Kotthoff, H. and Cowen, R., eds.  (2011).  PISA Under Examination:  Changing Knowledge, Changing Tests, and Changing Schools.  Rotterdam:  Sense Publishers.

Popkewitz, T. (2011).  PISA: Numbers, standardizing conduct, and the alchemy of school subjects.  In Pereyra, M., Kotthoff, H. and Cowen, R., eds.  (2011).  PISA Under Examination:  Changing Knowledge, Changing Tests, and Changing Schools, 31-46.  Rotterdam:  Sense Publishers.

Rutkoski, L. and Rutkowski, D.  (2016).  “A call for a more measured approach to reporting and interpreting PISA results.”  Educational Researcher.  AERA. May 2016.  252-257.

Troehler, D. (2011). Concepts, cultures and comparisons.  PISA and the double German discontentment.  In Pereyra, M., Kotthoff, H. and Cowen, R., eds.  (2011).  PISA Under Examination:  Changing Knowledge, Changing Tests, and Changing Schools, 245-258.  Rotterdam:  Sense Publishers.

Permanent link to this article: http://idea-network.ca/research/questioning-pisa-examine-the-purpose-not-just-the-rankings