Predictive validities: SAT and PSU (faculty: Arquitectura) SOURCE: Pearson, Final Report Evaluation of the Chile PSU, January 2013.

Language Mathematics SAT Writing PSU Social Science

0

0.1

0.2

0.30.4

0.5

0.6

SAT PSU Arquitectura

Predictive validities: SAT and PSU(faculty: Arquitectura)

SOURCE: Pearson, Final Report Evaluation of the Chile PSU, January 2013

Language Mathematics SAT Writing PSU Social Science

0

0.1

0.2

0.30.4

0.5

0.6

SAT PSU Educacion

Predictive validities: SAT and PSU(faculty: Educacion)

SOURCE: Pearson, Final Report Evaluation of the Chile PSU, January 2013

Language Mathematics0

0.1

0.2

0.30.4

0.5

0.6

CTA Pearson

Predictive validities of the PSU(CTA v Pearson estimates)

SOURCE: Pearson, Final Report Evaluation of the Chile PSU, January 2013; CTA

Incremental Predictive validities (engineering): (controlling for NEM)

SOURCE: S.A. Prado, Estudio de Validez Predictiva de la PSU y Comparacion con el Sistema PAA, Universidad de Chile

U. Chile PUC U. Chile PUCLanguage & Math Language & Math + subject test

0

5

10

15

20

25

30

35

PAAPSU

Content coverage (values from 0% to 100%)

It is not fair to expect students to master content to which they have not been exposed. …or, to compare students who have been exposed to students who have not.

…how much of the content domain of a test has been taught in the schools.

SOURCE: Centro de Estudios Mineduc, Cobertura Curricular en Ensenanza Media Lenguaje y Comunicacion – Matematica, Septiembre 2012

Municipal Subvencionado Pagado0

25

50

75

100

Percentage curricular coverage in Chilean high schools, by type of school: 2012

Mathematics, Level 1



Language & Communication, Level 2


25

50

75

100





25

50

75

100





25

50

75

100


Percentage curricular coverage in Chilean high schools, by type of curriculum: 2012


Humanista Cientifica Technico Profesional Polivante0

25

50

75

100


Percentage curricular coverage in Chilean high schools, by type of curriculum: 2012


Numanista Cientifica Technico Profesional Polivante0

25

50

75

100


Percentage of Chilean high schools with full curricular coverage, by subject area: 2012

Levels 1--4

Mathem

atics

Langu

age &

Communication

0%

25%

50%

75%

100%

Do NOT Cover 100%Cover 100%

Subgroup differences

Differences in test scores among subgroups (e.g., gender, ethnic, school type) should be due only to differences in the attribute measured by the test and not to systematic biases in the test.

111

170

46

8595

124

43

51

0102030405060708090

100110120130140150160170180190

2002 2003 2004 2005 2006 2007 2008 2009 2010

Bre

chas

Brechas PSU Matemáticas para toda la muestra

0

20

40

60

80

100

120

140

160

180

200

2002 2003 2004 2005 2006 2007 2008 2009 2010

Bre

chas

Brechas PSU Matemáticas para toda la muestra PP Muni-TP Brecha Sin Ajustar

PP Muni-TP Brecha Ajustada

PP Muni-CH Brecha Sin Ajustar

PP Muni-CH Brecha Ajustada

0

20

40

60

80

100

120

140

160

180

200

2002 2003 2004 2005 2006 2007 2008 2009 2010

Bre

chas





0

20

40

60

80

100

120

140

160

180

200

2002 2003 2004 2005 2006 2007 2008 2009 2010

Bre

chas




PP Muni-CH Brecha Ajustada0

20

40

60

80

100

120

140

160

180

200

2002 2003 2004 2005 2006 2007 2008 2009 2010

Bre

chas





SOURCE: Koljatic, Silva, & Phelps, Consequential Tests and Conflicts of Interest: The Case of Chile’s PSU, forthcoming, 2014

Growing gaps in PSU Mathematics raw & adjusted scores, by type of curriculum: 2002—2010

106

161

44

79

86

113

36

44

0

10

20

30

40

50

60

70

80

90

100

110

120

130

140

150

160

170

2002 2003 2004 2005 2006 2007 2008 2009 2010

Bre

chas

Brechas PSU Lenguaje para toda la muestra

0

20

40

60

80

100

120

140

160

180

200

2002 2003 2004 2005 2006 2007 2008 2009 2010

Bre

chas





0

20

40

60

80

100

120

140

160

180

200

2002 2003 2004 2005 2006 2007 2008 2009 2010

Bre

chas





0

20

40

60

80

100

120

140

160

180

200

2002 2003 2004 2005 2006 2007 2008 2009 2010

Bre

chas




PP Muni-CH Brecha Ajustada0

20

40

60

80

100

120

140

160

180

200

2002 2003 2004 2005 2006 2007 2008 2009 2010

Bre

chas





SOURCE: Koljatic, Silva, & Phelps, Consequential Tests and Conflicts of Interest: The Case of Chile’s PSU, forthcoming, 2014

Growing gaps in PSU Language & Communication raw & adjusted scores, by type of curriculum: 2002—2010

3. A chronology of mistakes

2000, initial proposal, SIES/PSU project

This proposal attempts a redesign of the tests currently used to select students for higher education in Chile. It is expected that [this new test will] have a positive impact in the efficiency of the selection process, improving the psychometric properties of the measuring instruments, and establishing a better articulation between the selection system and the secondary education curriculum.

SOURCE: Proyecto FONDEF, Reformulacion de las Pruebas de Seleccion a la Educacion Superior

…the Academic Aptitude Test for entry to the university system is under revision, together with the universities belonging to the Council of Rectors. This instrument of entry selection, needs also to be aligned with the new curriculum and may become an exit exam from the secondary education system.

2001 (World Bank & MINEDUC)

SOURCE: World Bank, Implementation Completion Report on a Loan in the Amount of $35 million to the Republic of Chile for Secondary Education, 2001

A chronology of mistakes (cont.)

Predictive validities: SAT and PSU (faculty: Arquitectura) SOURCE: Pearson, Final Report Evaluation of the Chile PSU, January 2013.

Documents

chile slide

percentage curricular

cta slide

chile psu

estudios mineduc

nem source

arquitectura source

educacion source