Learning about learning assessments

by Andreas Schleicher
Director, OECD Directorate for Education and Skills
Claudia Costin
Senior Director, Education Global Practice, World Bank 

How do large-scale student assessments, like PISA, actually work? What are the key ingredients that are necessary to produce a reliable, policy relevant assessment of what children and young people know and can do with what they know? A new report commissioned by the OECD and the World Bank offers a behind-the-scenes look at how some of the largest of these assessments are developed and implemented, particularly in developing countries.

A Review of International Large-Scale Assessments in Education: Assessing Component Skills and Collecting Contextual Data provides an overview of the main international, regional, national and household-based large-scale assessments of learning. The report shows how the major large-scale assessments have several things in common that contribute to their reliability and relevance. For example, they each produce clear frameworks to describe the philosophy, content, test design and response styles of their tests. These frameworks not only guide the creation of items (questions or tasks in a test paper) for the test, but also act as a way of communicating information about the assessment to the broader community.

The mode of delivery for most of the large-scale assessments is paper and pencil, but there is a shift towards computer-based assessment and this will undoubtedly be the main mode of delivery in the future as it increases efficiency and reduces data error. All of the assessments covered by the report collect contextual information that can be related to the test scores and help to inform policy choices. The reviewed surveys devote considerable time and resources to coder training and coding itself –this is the process of marking students’ responses with codes once tests are complete, including the steps taken to confirm that coding is being undertaken with acceptable reliability. In one or two cases, methods and approaches have been developed to include out-of-school children in learning assessments.

The report gives particular emphasis to learning from large-scale assessments in developing countries and makes recommendations in the following areas for the benefit of the OECD’s PISA for Development project: assessment frameworks; scoring; modes of delivery; collection of contextual information; methods and approaches to include out-of-school children in learning assessments; and analysis, reporting and use of the data collected.

The report also reveals some little known facts about the major large-scale assessments. Did you know that an assessment used in French-speaking mainly African countries uses questionnaires to collect information on whether students are working outside of school? Analysis of the results helps countries to determine whether working hinders students’ learning. And did you know that a large-scale assessment in Latin America routinely collects information on food, transportation, medical and clothing programmes and relates the data to student test scores? Or that a regional assessment in southern and eastern Africa finds that the active involvement of relevant government staff in research is one of the most important factors in converting analysis of the results of assessments into policies and changed practice?

Initially commissioned to provide recommendations for designing the PISA for Development project, the report is a valuable reference for policymakers, development organisations and other stakeholders with an interest in developing or participating in large-scale learning assessments.

Links: 
A Review of International Large-Scale Assessments in Education: Assessing Component Skills and Collecting Contextual Data
The Experience of Middle-Income Countries Participating in PISA 2000-2015
Towards the development of contextual questionnaires for the PISA for development study
PISA for Development Technical Strand C: Incorporating out-of-school 15-year-olds in the assessment
PISA in Low and Middle Income Countries
For more on PISA for Development, visit: www.oecd.org/pisa/aboutpisa/pisafordevelopment.htm
Photo credit: © epicurean / iStockphoto

The challenges of widening participation in PISA

by Andreas Schleicher
Director, OECD Directorate for Education and Skills
Claudia Costin
Senior Director, Education Global Practice, World Bank

Since 2000, the OECD Programme for International Student Assessment (PISA) has been measuring the skills and knowledge of 15-year-old students in over 70 countries. PISA does not just examine whether students have learned what they were taught, but also assesses whether students can creatively and critically use what they know.

Of course, such international comparisons are never easy and they aren’t perfect. But they show what is possible in education, they help governments to see themselves in comparison to the education opportunities and results delivered by other education systems, and they help governments to build effective policies and partnerships for improving learning outcomes.

But as the number of countries joining PISA kept rising, it became apparent that the design and implementation models for PISA needed to evolve to successfully cater to a larger and more diverse set of countries, including a growing number of middle-income and low-income countries who want to participate in the assessment.

In response to these challenges, the OECD and the World Bank just released a report titled The Experience of Middle-Income Countries Participating in PISA 2000-2015, which provides valuable lessons and insights based on the experiences of more than 40 PISA-participating countries. It establishes a strong rationale and foundation for enhancing PISA to make it more relevant to a wider range of countries. It also provides insights for the World Bank and other development partners on how to better support countries to participate in these exercises and to analyse and use the data in effective ways.

The report shows that while demand for participation in PISA among middle-income countries is increasing, these countries face both financial and technical obstacles to participating, including the need to translate and manage the assessment, and code student responses. The report also shows that the political, regulatory, and cultural environment of these countries can also affect whether, and how easily, the assessment can be conducted.

To maximize the benefits of participating in PISA, the report recommends that the OECD take five actions:

1. Adjust the PISA test instruments to better measure differences between the highest- and lowest-performing students and, in particular, distinguish performance differences at the lowest levels of proficiency;
2. Revise the contextual questionnaires so they are more relevant to low-income country contexts and policy issues;
3. Evaluate the impact of PISA participation on middle-income countries’ capacity to conduct international assessments; 
4. Tackle financial and technical challenges through partnerships with donors and through capacity building; and
5. Extend outreach to local stakeholders in these countries.

Action is already being taken on these recommendations through the PISA for Development initiative. This project is already working to enhance the PISA instruments and will undertake field trials in seven developing countries during 2016. The final results of PISA for Development, which are expected in 2018, will provide local policy makers with new evidence to diagnose shortcomings in their education systems and inform new policies. In the meantime, the PISA for Development countries will benefit from peer-to-peer exchanges with other members of the PISA global community. The enhanced PISA instruments will be made available to all countries for the 2021 cycle of the assessment.

The OECD remains committed to working with the World Bank and other partners in maintaining and developing PISA as a global yardstick for measuring success in education. This is especially relevant in the context of the recently adopted Sustainable Development Goals as PISA provides valuable information about the level and distribution of quality and equity within a country’s education system.

Together, we will continue to contribute our expertise and platforms to encourage international collaboration on education through the PISA surveys, and to assist policymakers and practitioners throughout the world to use them more productively. 

Links:

The Experience of Middle-Income Countries Participating in PISA 2000-2015
Network on Education Quality Monitoring in the Asia-Pacific

For more on PISA for Development, visit: www.oecd.org/pisa/aboutpisa/pisafordevelopment.htm
Related Working Papers:

• Towards the development of contextual questionnaires for the PISA for development study 
• PISA for Development Technical Strand C: Incorporating out-of-school 15-year-olds in the assessment
• PISA in Low and Middle Income Countries

Photo credit: © Istockphoto/Henk Badenhorst