USES AND INTERPRETATIONS OF INTERNATIONAL ASSESSMENT DATA PERSPECTIVES FROM THE INTERNATIONAL ACADEMY OF EDUCATION INVITED SESSION AERA, APRIL 6,2014.

USES AND IN

TERPR

ETATI

ONS OF

INTE

RNATIO

NAL ASSESSMENT

DATA

PE

RS

PE

CT

I VE

S F

RO

M T

HE

IN

TE

RN

AT

I ON

AL

AC

AD

EM

Y O

F ED

UC

AT

I ON

I NV

I TE

D S

ES

SI O

N

AE

RA

, A

PR

I L 6

, 20

14

PARTICIPANTS

The uses of PISA in Mexico Maria de Ibarrola and Sylvia Schmelkes, Mexico

Validity of Inferences from International Assessment: Cautions about two typical uses

Kadriye Ercican, Canada

Using International Assessment to Investigate Cognitive Performance Differentials by Socioeconomic Status

Servaas Van de Berg, South Africa

Measuring College Value Added: a delicate instrument Richard J. Shavelson,USA

Collaborative Problem Solving in Different National Settings Patrick Griffin, Australia

Maria de IbarrolaDepartment of Educational Research

Center for Research and Advanced Studies, MexicoSylvia Schmelkes

President, National Institute of Educational Evaluation, Mexico

AERA Invited Session for the International Academy of Education 

Uses and Interpretations of International Assessment Data: Perspectives from the International Academy of Education (IAE)Philadelphia, PA, USA April 6 2014

The uses of PISA* in Mexico. 2013 radical reforms for improving

quality of education

*Program for International Student Assessment

Purposes of this talkReport the dramatic results of PISA in MexicoDescribe some uses of PISA in educational

policy decision making for improving quality (or at least for improving PISA results)

Enumerate the radical educational reform proposed in 2013 (and its emphasis on teachers)

Give an account of the serious lack of consensus on the measures taken

(This presentation will not deal with PISA psychometric, statistical or internal validity)

Mexico among OECD countriesMexico became an OECD country member in

1994Part of National policies for the “modernization” of the country:

( signing NAFTA also for instance) .A huge reform in 1989 for the modernization of education (At about the same dates, the Zapatista Movement of Liberation

exploded in the South of the country, reminding us of another, and many others, Mexicos)

As an OECD contry member, Mexico’s first participation in PISA was in 2000: (Reading)Regular participation after that every three years:

2003: Mathematics2006 ( Sciences)2009 ( Reading)2012 ( Mathematics )

Among OECD countries, Mexico… Has the lowest income per

capita: 6400 US doll / vs a mean of 23 100

Is the second country with increasing inequality amongs its population: 10 % of the richest sector of the population is 29 times richer than the lowest 10%

Spends only a third ( 7. 4% of GNP) in social expenses as against 21.9% as a mean for all OCDE countries

Spends half( USD 5 000 or less) per child per year from primary through tertiary education in 2010 against OECD countries USD 9 300

Schooling for children of Migrant daily laborers: poorest among poors

A synthesis of results, PISA 2012

Mathematics

(OECD+-500)

Sciences

(OECD+-500)

Reading

(OECD+-500)

Measure of outcomes ( points) 413 415 424

Rank among 65 countries 53 55 52

Rank among 8 Latin America countries 2 4 3

Rank among the 34 OECD countries 34 34 34

% of students in performance levels

Below level 2

High, levels 4-6

Below level 2

High , levels 4-6

Below

level 2

High, levels 4-

6

55% 4% 47% 2% 41% 5%

The PISA shock! After 9 years of schooling, 52 % of 15 year old

students do not have the minimum competences for the Knowledge Society (nor the Constitutional educational objectives mandated ).

Only about 4% have the desired competences

National Educational Research has tried to make public awareness of the poor educational resources and the poor educational outcomes since the 1960’s; highly critical of educational policies

New awareness, now based in a “solid, scientific, international evidence of students’ achievements” that has interested many other social sectors in the educational problems of the country

Bad students and bogged down!

Last among OECD countries

8/10 students underachievers

0.1% excellent in Sciences

77 years to achieve OECD mean, a 149 years to get the first place!

The search for causes and explanations

Mexico’s development status: “not yet a developed nation”

Poor socioeconomic conditions of most students, and deep cultural differences

Small proportion of GNP destined to education, and unequally distributed among modalities and levels

Bad Teacher working conditions, poor initial and continuous training, large class size, etc

27% of schools without water; 42% without sewage; 20% schools without desk and chairs for teachers

Poor eduational governance and administration at school, local, regional, State, and National level

USES of PISA in Mexico, as proposed by OCDE exploration

Changes in educational policies and practices in light of PISA results YES

Use of PISA policy findings in national/federal education debate and policy-making YES

The role of PISA in national/federal assessment and evaluation policies and practices YES

Setting and revising curriculum standards YES

Setting and monitoring performance targets and indicators YES

Breakspear, S. (2012), “The Policy Impact of PISA: AnExploration of the Normative Effects of InternationalBenchmarking in School System Performance”, OECDEducation Working Papers, No. 71, OECD Publishing.

http://dx.doi.org/10.1787/5k9fdfqffr28-en

Progressive acceptance of PISA as a valid and reliable indicator of the quality of education, in spite of the explanations, (because they are taken into account in the construction of the instrument)

Setting and monitoring performance targets and indicatorsObjective 1 of the Education Sector

Program 2007-2012 establishes as a goal for 2012 to raise performance on PISA to 435 points as an average for both mathematics and reading, taking the average score of 392 points, attained in PISA 2006, as the base.

The objective was not reached (415 points), some propose a new target, others warn against the use of this kind of

targets as substitute to an integral education

Setting and revising curriculum standards In 2011 a huge curriculum reform was launched

integrating K-12.“The Agreement for the Articulation of Basic Education

asserts that the whole curriculum should set a vision for 2021 that includes generalizing the competencies described at PISA Level 3; eliminating the gap between the students who perform below Level 2 and those who perform at or above that level”.

As of 2013 the reform is critcized as a huge and monumental disaster for Basic education, a new basic curriculum debate has been called at a National level . What is today’s Basic Education: knowledge, abilities and

valuesWhat is the profile of a basic education graduateWhat should be the curricular structure of basic education

Led to the formation, or increased the scope, of a national/federal assessment

systemPISA (2000) EXCALE (2003) ENLACE (2006)

Scope International National National

Design and application

OECD National Institute for the Evaluation of Education

Federal Ministry of Education

Periodicity Every 3 years (different emphasis)

Programmed for different grades, ar every 4 years

annual

Coverage sample sample censal

Evaluated population

15 year old 3rd,6th and 9th grade

Grades 3, 4, 5, 6, 7, 8 9, 12

Aims Measuring competences for life in the knowledge society

Educational system achievement, quality and obstacles

Ranking student’s achievement, Linking results with teacher incentives and school budgets

Results National and State National and State Ranked by individual schools

Unreliability of ENLACE(Students with best results)

ENLACE PISA

2009 0.9 0.7

2012 7.0 0.6

2009 20120

1

2

3

4

5

6

7

8

ENLACE

Linear (ENLACE)

PISA

Linear (PISA)

Linear (PISA)

ENLACE was linked to teacher incentives since 2006Always criticized as for its validity

And was suspended in 2014

Weakness of the instrument?

Corruption? Fraud?

Private sectors Defend ENLACE

Light improvements in PISA resultsBetween 2003 and 2012, Mexico’s schooling

for young people increased from 55% to cover 75% of the age group, but the country is still far from reaching the desired coverage rate of a 12 year compulsory education.

Between 2003 and 2012, Mexican students increased the points achieved in PISA results; from 385 to 413

And Finland lost a higher number WHY?? regression to the mean? Weakness of

the instrument?

15 year old coverage and PISA evaluation cycles

"2000" "2003" "2006" "2009" "2012"0%

20%

40%

60%

80%

100%

Ciclos evaluativos de PISA

51.658.1

62.966.2 69.6

SueciaFinlandia

Nueva ZelandaIslandia

AustraliaEslovaquia

República ChecaFranciaBélgica

HolandaCanadá

DinamarcaHungríaUruguay

TailandiaAlemania

Macao-ChinaFederación Rusa

IndonesiaItalia

PortugalTurquíaPolonia

MÉXICOTúnezBrasil

-40 -30 -20 -10 0 10 20 30 40Diferencia porcentual 2003 - 2012

Meaningful differences 2003-2012 in the mean performance in mathematics by country

Looking for the key factors to improve qualitySocioeconomic conditions of the country?

Inequality has increased!Educational Budget?

Expenditure per student is the lowest and has not improvedHuge differences in expenditure by region and school modality:

provisional measures became regular poor educational systems for poor sectors

Improving Infrastructure?Several programs to improve school infrastructureSeveral efforts to introduce ICTs in schools and USE them for

pedagogical means But recent census ( 2014) reveals serious problems

School autonomy ?Several efforts and norms for decentralizing school operation up

to school levelBut teacher labor conditions do not foster collegiate work

It is the Teachers…!!!.

“What determines school progress is the quality of the teaching body!

“The teacher makes the difference. And if his/hers performance takes place within a context of clear and ambitious objectives, ( high standards) accompanied by measure tools, training and informatin to adjust the system, the difference is still better”.

(PISA 2012, Mexico. Mtra. Gabriela Ramos, Directora de Gabinete y Sherpa, Dec. 2013)

The secrets of Finland success: expert teachers: Highly valued; rigourously selected; very well trained; wide

definition of functions/adequate working time; complete pedagogial freedom; associated to Universities; continuous trained; regularly evaluated…

The 2013 Reform: evaluation is the name of the game Constitutional amendments and a New General Law of

Education: a fundamental right to receive QUALITY EDUCATION

Full Autonomy to the National Institute of Educational Evaluation (INEE)

The role of INEE: formative evaluation! The INEE is the MAIN authority in evaluation matters: 19 General

Directorates to fulfill all the responsibilities assigned by law

Law of Teaching as a professional service Evaluation of teachers for entry, promotion and permanence, as indicated by

INEE Radical change in the actors and the criteria that sustain those decisions:

the Union is out

A system of reliable information on teachers: National census (March 2014)

After a century of building a National education system we get to know how many teachers are there in the payroll. ( and 13% are not accounted for)

Teachers’ evaluation. A passionate debate Educational authorities ambiguous game:

A call for National and Regional proposals on a “new educational model” But ...Drawing up laws at the highest level and Normative policy

instruments at high speed The Very strong National Union of Teachers: evaluation is accepted

but “Labor rights have to be respected” Radical Union’s sections in poorest states violently oppose the laws:

stoppages, strikes, highway and street blockades, Strong private sector NGOs pressing for teacher evaluation and

highstake results: Teachers can be fired !and some (many?) of them should be.

Regular Teachers offended by an evaluation that does not consider their daily teaching conditions and do not recognize their efforts and contributions for improvement

Scholars and Researchers: trying to make a sound and objective evaluation of the 2013 reforms and its effects: is it a labor reform or an educational proposal?

Evaluation is the information, not the solution.

Deep and serious lack of consensus on what is quality of education and how to achieve it

within an extremely diverse and unequal society Uncomplete reforms, continuous changes, ambiguous

measures, broken links among the many layers of implementation

Serious gaps and contradictions between discourses, laws, budget and daily conditions

All sorts of active and passive opposition coming from different sectors

Educational Research (National and International) has a

huge bibliography on how complex Educational change and improvements are.

Sorry, We have not achieved the proper formula