The Benefits and limitations of international educational ...unesdoc.unesco.org/images/0011/001176/117629E.pdfPublished in the series: Trends in education The benefits and limitations
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Published in the series:Trends in education
The benefits and limitationsof international educational
achievement studies
Albert E. Beaton et al.
A paper copy of this publication may be obtained on request from:[email protected]
To consult the full catalogue of IIEP Publications and documents on ourWeb site: http://www.unesco.org/iiep
Co-operation Agency (Sida) has provided financial assistance for the publication of
this bookle
Published by:International Institute for Educational Planning/UNESCO
International Institute for Educational Planning http://www.unesco.org/iiep
CONTENTS
Pages
Preface 7
Explanatory note 9
Introduction 11
I Benefits and limitations of participating inan international achievement study 13
What benefits do international studies of achievement havefor the development of curricula in participating countries? 13
How can international studies relating education systemcharacteristics to educational achievement levels contributeto the improvement of education systems? 14
How can international studies of educational achievementcontribute to the training of educational researchers in eachcountry? 17
What information is needed to assure decision-makers that thetests used in international studies are equally fair to allparticipating nations? Is there such a thing as a level playingfield and how important is it? 18
What can be done to ensure that international studies givepriority to the major policy questions in all participatingcountries? How can decision-makers ensure that theirparticular concerns are addressed? 20
II Scope of research questions in an internationaleducational achievement study 22
To what extent can the operation of a national system ofcurriculum (vs. a state curriculum or school district curriculumor school-based curriculum) be assessed in an international study? 22
To what extent can variation in achievement among countriesbe related to different approaches to curriculum sequencing? 23
To what extent can the operation of different types ofexamination systems be assessed in an international study? 24
International Institute for Educational Planning http://www.unesco.org/iiep
Contents
6
To what extent can examining the variation among countriesprovide useful information on the effect of a factor that isinvariant within countries? 25
To what extent does variation among school means at a given agelevel differ from country to country and what factors areassociated with such variation? 26
To what extent do different countries change their relative position in school achievement levels over time and howimportant is this? 27
What is the relationship between achievement levels andeconomic productivity for a nation? 28
III Comparison within and between education systems ininternational achievement studies 30
How can achievement be compared across different age orgrade groups and surveys several years apart? 30
If a country is involved with two agencies in internationalachievement studies and has its own national studies,what can it do to ensure that it can obtain a picture of as manygrades as possible in several subject areas? 31
If a comparison of sub-groups of students within a countryis to be undertaken, what are the sub-groups that should becompared and why? 32
International Institute for Educational Planning http://www.unesco.org/iiep
The benefits and limitations of international educational achievement studies
14
given to, and the content covered by, different subject areas. Such
differences do not necessarily imply any deficiency because a country
may well have good reasons for adopting a curriculum involving
variations from the international pattern, for example reasons
associated with the state of development of its education system.
The information provided on students’ comparative performance
on questions and sub-sections of a cross-national achievement test
can be of direct benefit to a country in helping to determine whether
to change the curricular emphases given to different subject areas
and/or different topics. Changes in curricular emphases designed to
improve achievement levels in one subject area, however, may
involve a trade-off in other subject areas.
How can international studies relating education systemcharacteristics to educational achievement levels contributeto the improvement of education systems?
International studies of educational achievement typically reveal
substantial differences in the average level of student performance
in different countries, whatever the subject area. Such information
generates a question of more immediate interest to education
systems. Why do some countries have higher levels of achievement
than others? Most international studies collect information that
allows for a detailed examination of factors likely to inf luence
educational achievement.
Attention has mostly been focused on the following broad
groupings of explanatory factors:
1 Home background: for example, the educational levels and
occupations of parents, educational resources within the home
International Institute for Educational Planning http://www.unesco.org/iiep
The benefits and limitations of international educational achievement studies
18
also benefit from the wider perspectives taken in international studies
and the detailed analyses which these studies have to make of the
philosophy and practice of curriculum development and of the
teaching approaches adopted in different countries.
What information is needed to assure decision-makers that the testsused in international studies are equally fair to all participatingnations? Is there such a thing as a level playing field and howimportant is it?
In principle, a single set of test questions that is used to construct
an international test cannot be perfectly appropriate for all students
in all countries, in the sense that all students have had an equal
opportunity to learn the subject matter that is tested. Curricula vary
from country to country in terms of both content and sequencing
and thus students always have different opportunities to learn. This
variation in curricula, however, should not be seen as invalidating
international studies because one of the important aims of these
studies is to identify and document such differences. To achieve a
fair basis for international comparisons, international tests should
incorporate sub-groups of items that provide a valid assessment of
most of the important components of each national curriculum. Two
types of valid comparisons can then be made among countries. First,
when two countries share one or more important common
curriculum components, then the relevant subsets of test items may
be used to make direct comparisons using the same items. Second,
where it can be agreed that a single underlying trait (or dimension of
knowledge) is being assessed by the test (or a subset of the test),
then new technologies based on modern ‘Item Response Theory’
may be used to make valid comparisons of countries on groups of
test items that ‘overlap’ but are not completely congruent.
International Institute for Educational Planning http://www.unesco.org/iiep
The benefits and limitations of international educational achievement studies
20
contribute to student performance. An international test is an agreed-
upon criterion that measures the net effect of all such factors. For
example, countries may have different starting ages or retention
policies, as well as curricula and pedagogical practices, that contribute
to student performance and these must be considered when
comparing countries’ performance levels. At the end of secondary
school, the differences among countries are more pronounced and
obvious since systems vary substantially with respect to the number
of grades required for graduation, apprenticeship programmes, drop-
out rates, and the like. Such factors have to be taken into account
when interpreting the differences in achievement levels. Perhaps
the most important finding of an international assessment at this
level is how school systems vary around the world. An international
test provides a common method for assessing the output of the
various systems when, as young adults, the students enter the
workforce or higher levels of education.
What can be done to ensure that international studies give priority tothe major policy questions in all participating countries? How candecision-makers ensure that their particular concerns are addressed?
There are two ways to ensure that major policy questions are
addressed in international studies. First, international studies are by
their nature co-operative ventures, and by joining a study at the very
beginning, each country’s representatives will have a chance to
contribute to the planning of a study. Some studies (such as the
Southern Africa Consortium for Monitoring Educational Quality
(SACMEQ) studies listed in the bibliography) commence by
developing policy questions common to all participating countries,
and then plan studies to answer these questions (and only these
questions). The SACMEQ approach ensures that policy questions that
are of importance to all countries are addressed in the research design.
International Institute for Educational Planning http://www.unesco.org/iiep
The benefits and limitations of international educational achievement studies
22
II SCOPE OF RESEARCH QUESTIONS IN ANINTERNATIONAL EDUCATIONAL ACHIEVEMENT STUDY
What are the research questions that can be addressed in such
studies that cannot be addressed in a national study or can be
addressed only in a limited way?
To what extent can the operation of a national system of curriculum(vs. a state curriculum or school district curriculum or school-basedcurriculum) be assessed in an international study?
In international studies of educational achievement a distinction
is often made between intended and implemented curricula. The
intended curriculum of an education system is the planned
curriculum and represents the expectations of what should be taught
in a particular subject at a particular grade level. Persons responsible
for the intended or planned curriculum at a national level should be
in a position to review individual test items to determine whether
they are testing what is included in the national curriculum. By
engaging in such a review, it is possible to identify those parts of an
international test that are a part of the national curriculum in a
particular subject at a particular grade level. It is then possible to
develop a test score based on that nation’s curriculum. Furthermore,
it is possible to compare a country’s performance on that subset of
questions and items from an international test with that of other
countries on that same subset of questions and items.
The point here is that a country can obtain scores focused on its
own curriculum for all other participating countries. This offers an
important avenue for making what many educationists believe to be
‘fair comparisons’. Research evidence shows that while these fair
comparisons are important, there are major similarities in curricula
International Institute for Educational Planning http://www.unesco.org/iiep
The benefits and limitations of international educational achievement studies
26
International Study of Mathematics Achievement (Husén, 1967), some
participating countries commenced formal schooling at age five,
others at age six, and still others at age seven. The researchers in that
study were interested in the consequences of different starting ages.
This question was investigated by comparing the achievement test
performance of students in the three starting age groups at age 13.
The results showed that there was no difference between starting at
age five or six, but that students in countries that started school at
age seven scored slightly lower at age 13. In a similar way, the effect
of other factors that are invariant within countries but vary among
countries can be assessed.
To what extent does variation among school means at a given agelevel differ from country to country and what factors are associatedwith such variation?
Some of the variation in student achievement test scores in a
country is due to differences in the performance of students in the
same class or same school, while some of the variation is due to
differences in the average performance of students in different
classes or different schools. International studies have shown that
school means differ much more widely in some countries than in
others.
In a 1981-86 international study, for instance, the average science
achievement level was much the same for all classes, both at Grades
4 to 5 and Grades 8 to 9 in Finland, Japan, and Sweden. In contrast,
school means differed considerably for one or both of the grade
groupings in Italy, the Netherlands, the Philippines, Ghana, and
Nigeria (Postlethwaite; Wiley, 1992). In Italy and the Netherlands,
the large differences among school means could be attributed to the
‘streaming’ of students into different types of schools at Grades 8/9.
International Institute for Educational Planning http://www.unesco.org/iiep
Comparison within and between education systems in international achievement studies
31
If a country is involved with two agencies in internationalachievement studies and has its own national studies, what can it doto ensure that it can obtain a picture of as many grades as possiblein several subject areas?
National assessment studies are concerned with measuring
achievement in particular subject areas within a nation at particular
grade levels. This can focus on assessing achievement levels at one
point in time, or in assessing change over time. In national assessment
studies, one major focus is on the differences among sub-groups of
students or schools or regions within countries. The sub-groups used
in these analyses are often nation-specific. The major thrust of
international studies is to compare achievement among nations. There
is often little focus on differences among sub-groups of students or
schools within countries. Nevertheless, the data from international
studies can often be used to conduct important national analyses.
Both national and international assessments are costly, especially
when the opportunity costs of student and teacher time are included.
Although international studies are uniquely appropriate to answer
questions about how a country compares with other participating
countries, it is important for administrators to ensure that they gain
maximum benefits for their involvement. The surest way is for a
country to get involved in an international study at the beginning so
that it has an important part in shaping the study’s design - including
the grades or ages that are sampled and the subject matter and topics
being assessed. No single country can be permitted to dominate the
development of an international study, and so no single country is
likely to achieve all that it wants from an international study. A country
must be selective in choosing the studies in which it will participate.
In some cases, it will be possible to augment an international study
with probes into national issues through additional test booklets or
questionnaires that address national issues. If national and
International Institute for Educational Planning http://www.unesco.org/iiep
The benefits and limitations of international educational achievement studies
48
The data-collection manual should cover every possible detail that
must be taken into account when conducting the data collection.
This involves ‘school forms’ and ‘student forms’ to ensure that the
correct schools are selected, the correct students are tested (and
not others), and the correct teachers are selected (where
questionnaires or tests are being administered to teachers). A second
manual is usually prepared for the data collectors and details
everything to be done within each selected school. A third manual
spells out (a) what each test administrator has to do and say during
the actual testing sessions, (b) the procedures and timing for the
administration of the instruments, and (c) how to parcel up the
instruments and return them to a central point. There should be
very few, if any, missing schools and very few missing students in the
data collection. Again, the authors of the reports should report the
percentage of missing schools and missing students. It is often said
that not more than 10 percent of schools should be missing from the
sample and not more than 20 percent of the students. However, since
there are no completely valid procedures for dealing with missing
data, these figures should be taken as absolute maximum levels.
■■■■■ Data recording, data cleaning, and the calculation of sampling weightsData recording, data cleaning, and the calculation of sampling weightsData recording, data cleaning, and the calculation of sampling weightsData recording, data cleaning, and the calculation of sampling weightsData recording, data cleaning, and the calculation of sampling weights
The data are usually recorded on computers at the National
Centres. Typically the study provides data-entry software which is
used in all countries. Good data-entry software provides a number
of initial checks on the data that can be corrected immediately during
the data-entry process. These data are then returned to the
international central point where further data cleaning takes place.
There are always ‘extra’ errors in data entry no matter how good the
data-entry program. By undertaking consistency checks it is possible
to identify questions on the questionnaires where an error occurred
on the part of the respondent. These ‘problems’ are reported back to
national centres; they then contact the schools for elucidation and
International Institute for Educational Planning http://www.unesco.org/iiep
Appendix 2
49
then send the ‘correct’ data back to the international data-processing
centre. The necessary changes are then made. This ‘cleaning’ process
can take a long time, especially when there are many countries in the
study. However it should be mentioned that a data set from one
country, where some carelessness is evident in the data collection
and/or data entry, can take an inordinate time to clean. Finally, in
order to account for different probabilities of selection (due to
disproportionate selection across strata, inaccurate sampling frames,
missing data, etc.) sampling weights have to be calculated.
All of these points should be expected to have been covered in
reports on properly conducted international studies, but it is
incumbent upon the reader to check this.
■■■■■ Data analysesData analysesData analysesData analysesData analyses
Some of the analyses will be simple and others complex. If ‘dummy
tables’ have been produced at the onset of the study, the analyses are
undertaken to complete the tables. If the reader is not experienced
in data analysis, it is usually wise to have experts advise him or her on
the appropriateness of the analyses for the questions posed.
■■■■■ Reports emanating from the studyReports emanating from the studyReports emanating from the studyReports emanating from the studyReports emanating from the study
The reports should be clearly written and deal with each of the
policy issues in turn. The source of the data under discussion should
always be clear, as should arguments concerning the interpretation
International Institute for Educational Planning http://www.unesco.org/iiep
General bibliography
57
IIEP Publications and Documents
More than 1,200 titles on all aspects of educational planning havebeen published by the International Institute for EducationalPlanning. A comprehensive catalogue, giving details of theiravailability, includes research reports, case studies, seminardocuments, training materials, occasional papers and reference booksin the following subject categories:
Economics of education, costs and financing.
Manpower and employment.
Demographic studies.
Location of schools (school map) and micro-planning.
Administration and management.
Curriculum development and evaluation.
Educational technology.
Primary, secondary and higher education.
Vocational and technical education.
Non-formal, out-of-school, adult and rural education.
Disadvantaged groups.
Copies of the catalogue may be obtained from IIEP Publications on
International Institute for Educational Planning http://www.unesco.org/iiep
About the International Academy of Education
The International Academy of Education is a not-for-profit scientificassociation that aims at fostering scholarly excellence in all fields of educationby promoting educational research, its dissemination, and the implementationof its implications. Therefore, the Academy is dedicated to strengtheningthe contributions of research, solving critical educational problemsthroughout the world, and providing better communication among policy-makers, researchers and practitioners. Towards those ends, the Academyprovides timely syntheses of research-based evidence of internationalimportance, as well as critiques of research, its evidentiary basis, and itsapplication to policy.
The seat of the Academy , which was founded in 1986, is at the Royal Academyof Science, Literature and Arts in Brussels, Belgium, and its co-ordinatingcentre is at Curtin University of Technology in Perth, Australia
The current members of the Board of Directors of the Academy are:
Erik De Corte,University of Leuven, Belgium (President)
Barry Fraser,Curtin University of Technology, Australia (Executive Director)
Jacques Hallak,International Institute for Educational Planning, France
Michael Kirst,Stanford University, US
Ulrich Teichler,University of Kassel, Germany
Margaret Wang,Temple University, US
Herbert Walberg,University of Illinois at Chicago, US (Vice President)
International Institute for Educational Planning http://www.unesco.org/iiep
The International Institute for Educational Planning
The International Institute for Educational Planning (IIEP) is an internationalcentre for advanced training and research in the field of educational planning.It was established by UNESCO in 1963 and is financed by UNESCO and byvoluntary contributions from Member States. In recent years the followingMember States have provided voluntary contributions to the Institute: Denmark,Germany, Iceland, India, Ireland, Norway, Sweden, Switzerland and Venezuela.
The Institute’s aim is to contribute to the development of educationthroughout the world, by expanding both knowledge and the supply ofcompetent professionals in the field of educational planning. In this endeavourthe Institute co-operates with interested training and research organizations inMember States. The Governing Board of the IIEP, which approves the Institute’sprogramme and budget, consists of a maximum of eight elected members andfour members designated by the United Nations Organization and certain of itsspecialized agencies and institutes.
Chairman:Lennart Wohlgemuth (Sweden)
Director, The Nordic Africa Institute, Uppsala, Sweden.
Designated Members:David de Ferranti
Vice President, Latin America and Caribbean region, The World Bank,Washington, USA.
Carlos FortinDeputy to the Secretary-General, United Nations Conference on Tradeand Development (UNCTAD), Geneva, Switzerland.
Miriam J. HirschfeldDirector, Division of Human Resources Development and CapacityBuilding, World Health Organization (WHO), Geneva, Switzerland.
Jeggan SenghorDirector, African Institute for Economic Development and Planning (IDEP),Dakar, Senegal.
Elected Members:Dato’Asiah bt. Abu Samah (Malaysia)
Corporate Adviser, (Education) Lang Education, Kuala Lumpur, Malaysia.Klaus Hüfner (Germany)
Professor, Freie Universität Berlin, Berlin, Germany.Faïza Kefi (Tunisia)
Minister of the Environment, Ariana, Tunisia.Tamas Kozma (Hungary)
Director-General, Hungarian Institute for Educational Research, Budapest,Hungary.
Teboho Moja (South Africa)Visiting Professor, New York University, New York, USA.
Yolanda M. Rojas (Costa Rica)Professor, University of Costa Rica, San José, Costa Rica.
Michel Vernières (France)Professor, University of Paris I, Panthéon-Sorbonne, Paris, France.
Inquiries about the Institute should be addressed to:The Office of the Director, International Institute for Educational Planning,7-9 rue Eugène-Delacroix, 75116 Paris, France.