1 Knowledge Economy Forum Organisation for Economic Cooperation and Development (OECD) Results from PISA 2003 Istanbul March 22, 2005 Dr John Cresswell OECD/Directorate for Education
11
Knowledge Economy Forum
Organisation for Economic Cooperation and Development (OECD)
Results from PISA 2003Istanbul March 22, 2005
Dr John CresswellOECD/Directorate for Education
22
In the dark all education systems look the same
33
In the light, differences between education systems can be seen
44
In the light, differences between education systems can be seen
55 Origins of PISA OECD work on education statistics and
indicators major development commenced in late 1980s most of it funded by voluntary contributions substantial Member engagement through networks
Network on educational outcomes led by US developed a proposal for measurement of outcomes Education Committee formally initiated activity in
1996– 11 Members initially committed
Council decision in 1997– established decentralised Part II programme (became PISA)– Council required that OECD face no costs or financial risks– virtually all OECD Members signed on by then– Education Ministries have paid all costs and bear all risks
6666
OECD Partner countries
OECD countries
PISA 2000 country participation
7777
OECD Partner countries
OECD countries
PISA 2003 country participation
8888
OECD Partner countries
OECD countries
PISA 2006 country participation
99
Making international comparisons of achievement requires decisions
about...
what to assess,
whom to assess.
1010Deciding what to assess...
looking back at what they were expected to have learned
OR
looking ahead to what they can do with what they have learned.
For PISA, the OECD countries chose the latter.
1111PISA assessments
Reading literacy Using, interpreting and reflecting on written material.
Mathematical literacy Recognising problems that can be solved
mathematically, representing them mathematically, solving them.
Scientific literacy Identifying scientific questions, recognising what counts
as scientific evidence, using evidence to draw conclusions about the natural world.
1212
Development of the PISA tests
1313 Development of assessments Frameworks by international experts Assessment materials
submitted by countries developed by research consortium screened for cultural bias
– by countries– by expert, international panel– items with prima facie cultural bias removed at this stage
translated from English & French originals trialled to check items working consistently in all countries
Final tests items shown in trial to be culturally biased removed best items chosen for final tests
– balanced to reflect framework– range of difficulties– range of item types (constructed response, multiple choice)
1414
Measuring mathematical literacy inPISA 2003
1515 Mathematical literacy in PISA The capacity to:
identify, understand and engage in mathematics; make well-founded judgements about the role that
mathematics plays in an individual’s current and future:
– private life– occupational life– social life with peers and relatives– life as a constructive, concerned and reflective citizen.
Seen as depending on… mathematical knowledge and skills, ability to think and work mathematically, ability to apply the knowledge in a wide variety of
contexts.
1616 Measuring mathematical literacy in PISA 2003
Content Space and shape (assessed in PISA 2000) Change and relationships (assessed in PISA 2000) Quantity Uncertainty
Process skills Reproduction: use of practised knowledge, routine procedures… Connections: somewhat familiar but not routine… Reflection: insight, creativity in choosing mathematical
concepts… Context
Personal Educational or occupational Public Scientific
1717 Space & shape item
Answers: Yes, No, Yes, Yes
Process skill: connections
Context: educational quasi-realistic problem typical in maths classes not genuine occupational problem
Form: complex multiple-choice
Source: OECD (2004) Learning for tomorrow’s world: First results from PISA 2003, Figure 2.4a, p.52.
1818 Change and relationships item
Scores: 1 for n = 140x0.8 = 112 but no further work shown 2 for correct steps/min but not m/min; correct m/min but not km/hr; correct method but error of calculation; correct km/hr but not giving m/sec 3 for correct m/min (89.6) and m/hr (5.4), rounding acceptable.
Process skill: score 1=connections score 2=connections score 3=reflectionContext: personal
Form: open-constructed
1919 Quantity item
Form: open constructed response
Answer: Yes, with adequate explanation
Process skill: reflectionContext: public
Form: short constructed response
Answer: 12 600 ZAR (unit not required)
Process skill: reproductionContext: public
Form: short constructed response
Answer: 975 SGD (unit not required) Process skill: reproductionContext: public
2020 Uncertainty item
Scores: 1 for “No, not reasonable” but explanation lacking detail (e.g. focusing on exact increase in number of robberies without comparison with total) 2 for “No, not reasonable” with argument focusing on only small part of graph shown, ratio or percentage increase, or need for trend data.
Process skill: connections
Context: personal
Form: open- constructed
2121Deciding whom to assess...
grade-based sample
OR
age-based sample
For PISA, the OECD countries chose the latter, selecting 15-year-olds in school as the population.
2222 Key features of PISA 2003 assessment Information collected
each student
– 2 hours on paper-and-pencil tasks (subset of all questions)
– ½ hour for questionnaire on background, learning habits, learning environment, engagement and motivation
school principals
– questionnaire (school demography, learning environment quality)
Sample 275,000 students 41 participating countries
2323 PISA sampling requirements
Population: all 15-year-olds in school Sample
minimum of 150 schools per country two random samples: schools and replacement
schools if school declines, replacement school is invited stringent requirements set by countries (85% of
selected schools, 80% of selected students within schools)
2424
Results from PISA 2003
2525 PISA provides five key benchmarks for the quality of education systems
1. Overall performance of education systems
2. Equity in the distribution of learning opportunities
Measured by the impact students’ and schools’ socio-economic background has on performance…
… not merely by the distribution of learning outcomes
3. Consistency of performance standards across schools
4. Gender differences
5. Foundations for lifelong learning Learning strategies, motivation and attitudes
2626300 350 400 450 500 550 600
Finland
Korea
Canada
Czech Rep.
Austria
Germany
Slovak Rep.
Poland
Hungary
Latvia
USA
Russian Fed.
I taly
Greece
Serbia
Turkey
Mexico
OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 2.5c, p.356.
Mean mathematics scores – selected countries
2727OECD
Level 6
Level 5
Level 4
Level 3
Level 2
Level 1
BelowLevel 1
OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 2.5a, p.354.
What students can do in mathematics
15%
21%
22%
18%
10%
4%
11%
2929 What students can do in reading
10%
22%
12%
6%
22%
29%
OECD Average
Level 5
Level 4
Level 3
Level 2
Level 1
Below Level 1
OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 6.1, p.443.
3030
0%
20%
40%
60%
80%
100%
Fin
land
Kor
ea
Can
ada
Pola
nd
Uni
ted S
tate
s
Ger
man
y
Aus
tria
Latv
ia
Cze
ch R
epub
lic
Hun
gary
Ital
y
Gre
ece
Slo
vak
Rep
ublic
Rus
sian
Fed
erat
ion
Tur
key
Mex
ico
Ser
bia
Percentage of students at each of the proficiency levels in reading
OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 6.1, p.443.
3131 Performance in all domains
350
400
450
500
550
600
Hong KongFinland
KoreaNetherlands
LiechtensteinJ apan
CanadaBelgiumMacao
SwitzerlandAustralia
New ZealandCzech Rep.
I celandDenmark
FranceSwedenAustria
GermanyI reland
Slovak Rep.Norway
LuxembourgPoland
HungarySpainLatviaUnited
Russian Fed.Portugal
I talyGreeceSerbiaTurkey
UruguayThailandMexico
I ndonesiaTunisia
Brazil
Mathematics
350
400
450
500
550
600
350
400
450
500
550
600
Reading
350
400
450
500
550
600
Science Problem Solving
32323232
Securing an equitable distribution of learning opportunities
Measured by the impact students’ and schools’ socio-economic background has on performance – not merely by the distribution
of learning outcomes
3333
-3 -1 1 3-3 -2 -1 0 1 2 3
HighStu
dent
perf
orm
ance
Social background and student performance
AdvantagePISA Index of social backgroundDisadvantage
Low
OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Figure 4.8, p.176.
34343434
Ensuring consistent performance standards across schools
Between and within-school variation in performance
3535
0
20
40
60
80
100
120
140
Tur
key
Hun
gary
Jap
an
Bel
gium
Ital
y
Ger
man
y
Aus
tria
Net
herl
ands
Cze
ch R
epub
lic
Kor
ea
Slo
vak
Rep
ublic
Gre
ece
Swit
zerl
and
Luxe
mbou
rg
Port
ugal
Mex
ico
Uni
ted
Sta
tes
Aus
tral
ia
New
Zea
land
Spa
in
Can
ada
Irel
and
Den
mar
k
Pola
nd
Swed
en
Nor
way
Fin
land
Icel
and
Is it all innate ability?Variation in student performance
OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 4.1a, p.383.
3636
- 80
- 60
- 40
- 20
0
20
40
60
80
100Tur
key
Hun
gary
Jap
an
Bel
gium
Ital
y
Ger
man
y
Aus
tria
Net
her
land
s
Cze
ch R
epub
lic
Kor
ea
Slo
vak
Rep
ublic
Gre
ece
Swit
zerl
and
Luxem
bou
rg
Port
ugal
Mex
ico
Uni
ted S
tate
s
Aus
tral
ia
New
Zea
land
Spa
in
Can
ada
Irel
and
Den
mar
k
Pola
nd
Swed
en
Nor
way
Fin
land
Icel
and
Variation of performance
between schools
Variation of performance within
schools
Is it all innate ability?Variation in student performance in mathematics
OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 4.1a, p.383.
3737
- 80
- 60
- 40
- 20
0
20
40
60
80
100
Tur
key
Hun
gary
Jap
an
Bel
gium
Ital
y
Ger
man
y
Aus
tria
Net
her
land
s
Cze
ch R
epub
lic
Kor
ea
Slo
vak
Rep
ublic
Gre
ece
Swit
zerl
and
Luxem
bou
rg
Port
ugal
Mex
ico
Uni
ted S
tate
s
Aus
tral
ia
New
Zea
land
Spa
in
Can
ada
Irel
and
Den
mar
k
Pola
nd
Swed
en
Nor
way
Fin
land
Icel
and
Variation in student performance in mathematics
Variation of performance between
schools
Variation of performance within schools
Variation explained by socio-economic level of students and schools
OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 4.1a, p.383.
3838
200
500
800
-3 -2 -1 0 1 2 3
Stu
dent
perf
orm
ance
School performance and schools’ socio-economic background - Japan
AdvantagePISA Index of social backgroundDisadvantage
Figure 4.13
Student performance and student SES
Student performance and student SES within schools
School performance and school SES
School proportional to size
40404040
Bridging the gender gap
Performance, attitudes and motivation
4141Gender differences
In reading, girls are far ahead In all countries, girls significantly outperform boys in reading
In mathematics, boys tend to be somewhat ahead In most countries, boys outperform girls
…but mostly by modest amounts…… within classrooms and schools, the gender gap is often larger
Strong problem-solving performance for girls suggests……that it is not the cognitive processes underlying mathematics
that give boys an advantage……but the context in which mathematics appears in school
Gender differences in interest and attitudes towards mathematics are significantly greater than the observed performance gap
– Girls report much lower interest in mathematics, more negative attitudes and much greater anxiety with mathematics…
…and this may well contribute to the significant gender difference in educational and occupational pathways in mathematics-related subjects
44444444
Creating strong foundations for lifelong learning
Performance, attitudes and motivation
4545 Interest in and enjoyment of mathematics
0 10 20 30 40 50 60 70
I enjoy reading about mathematics.
I look forward to my mathematics lessons.
I do mathematics because I enjoy it.
I am interested in the things I learn about
mathematics.
OECD average Greece Latvia Slovak RepublicPercentage of students
- 60 - 50 - 40 - 30 - 20 - 10 0 10 20 30 40 50 60
Change in
mathematics per unit
of the index
OECD average Greece Latvia Slovak Republic Score points
OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 3.4, p.367 and Figure 3.4, p.126.
4646 Anxiety in mathematics0 20 40 60 80 100
I often worry that it will be difficult for me in
mathematics classes.
I get very tense when I have to do
mathematics homework.
I get very nervous doing mathematics
problems.
I feel helpless when doing a mathematics
problem.
I worry that I will get poor marks in
mathematics.
OECD average AustriaCzech Republic Switzerland
Percentage of students
- 60 - 50 - 40 - 30 - 20 - 10 0 10 20 30 40 50 60
Change in
mathematics per unit
of the index
OECD average Austria Czech Republic Switzerland Score points
OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 3.8, p.374 and Figure 3.8, p.139.
47474747
Some features of successful education systems
Insights from earlier PISA analysis
4848
350
400
450
500
550
600
0 10,000 20,000 30,000 40,000 50,000 60,000 70,000 80,000 90,000 100,000
Student performance and spending per student
Mexico
Greece
Portugal Italy
Spain
GermanyAustria
Ireland
United States
Norway
Korea
Czech republic
Slovak republicPoland
Hungary
Finland
NetherlandsCanada Switzerland
IcelandDenmark
FranceSweden
BelgiumAustralia
Japan
R2 = 0.28
Cumulative expenditure (US$)
Perf
orm
an
ce in
math
em
ati
cs
5050Governance of the school system
In many of the best performing countries Decentralised decision-making is combined
with devices to ensure a fair distribution of substantive educational opportunities
The provision of standards and curricula at national/subnational levels is combined with advanced evaluation systems
– That are implemented by professional agencies Process-oriented assessments and/or
centralised final examinations are complimented with individual reports and feed-back mechanisms on student learning progress
5151 Public and private schools
0 20 40 60 80 100
Luxembourg
J apan
I taly
Switzerland
Finland
Denmark
Czech Republic
Sweden
Hungary
Austria
Portugal
United States
Netherlands
Slovak Republic
Korea
I reland
Spain
Canada
Mexico
New Zealand
Germany
OECD
United Kingdom
Government schools
Government dependent private
Government independent private
-15
0
-10
0
-50
0 50 100
Observed perf ormance diff erence
Diff erence af ter accounting f or socio-economic background of students
Diff erence af ter accounting f or socio-economic background of students and schools
Private schools perform better
Public schools perform better
5252Organisation of instruction
In many of the best performing countries Schools and teachers have explicit strategies
and approaches for teaching heterogeneous groups of learners
– A high degree of individualised learning processes– Disparities related to socio-economic factors and
migration are recognised as major challenges Students are offered a variety of extra-
curricular activities Schools offer differentiated support
structures for students– E.g. school psychologists or career counsellors
Institutional differentiation is introduced, if at all, at later stages
– Integrated approaches also contributed to reducing the impact of students socio-economic background on outcomes
5353M
athe
ma
tics
perf
orm
ance
Decreasing effect of socioeconomic background
High performanceLow SES effect
Low performanceLow SES effect
Low performanceHigh SES effect
High performanceHigh SES effect
UruguayTurkey
I talyPortugal
LatviaUnited States Spain
NorwayHungary Poland
LuxembourgSlovak Republic
AustriaGermany I reland
DenmarkFrance Sweden
Czech RepublicI celand
Australia
J apanBelgium
New ZealandSwitzerland Macao-China
Canada
Netherlands
FinlandHong Kong-China
Korea
Liechtenstein
Russian Federation
Greece
Serbia
400
425
450
475
500
525
550
5757 Disciplinary climateStudents’ views
0 20 40 60 80 100
Students don't listen to what the teacher says.
There is noise and disorder.
The teacher has to wait a long time for students
to quieten down.
Students cannot work well.
Students don't start working for a long time
after the lesson begins.
OECD averageHungary Russian Federation Serbia Slovak Republic Turkey
Percentage of students
OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 5.3a, p.408 and Figure 5.3, p.217.
5858 Teachers’ morale and commitmentPrincipals’ views
0 20 40 60 80 100
The morale of
teachers in this
school is high.
Teachers work with
enthusiasm.
Teachers take pride
in this school.
Teachers value
academic
achievement.
OECD average Austria
Czech Republic Hungary
Russian Federation Turkey
Percentage of students
OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 5.5a, p.412 and Figure 5.5, p.223.
5959Further information
www.pisa.oecd.org– All national and international publications– The complete micro-level database
email: [email protected]