1 Paris 06/06/2007 2 nd OECD workshop on measuring Education and Health volume output Alain GALLAIS, OECD/STD/NAFS Draft chapter 2 on education output.

Paris06/06/2007

2nd OECD workshop on “measuring Education and Health volume output” Alain GALLAIS, OECD/STD/NAFS

1

Draft chapter 2 on education output

Background document: « chapter 2: education services »

Paris06/06/2007


2

General links between output and outcome

Hill services and result of the production processPeter Hill (1975) says the production of services is inseparable from their

use or consumption: two economic units are simultaneously involved in service production whereas production of a good takes place within a single producer unit.

Output of services should be viewed as the attributable contribution of the producer to the outcome.

UnitProduct Producer Consumer

Good Output

Service Outcome

Paris06/06/2007


3

Definitions: education output Eurostat handbook on prices and volume in NA“the quantity of teaching received by the students, adjusted to allow for the

qualities of the services provided, for each type of education”

“In the area of education, the output can be defined as the quantity of teaching (that is, the transfer of knowledge, successfully or not) for a pupil, whereas the related outcomes are the skill and knowledge that a pupil achieves.”

ISCED definition“organised and sustained communication designed to bring about

learning”, where

“Communication” involves the transfer of information (messages, ideas, knowledge, strategies, etc.) between two or more persons; […]

“Learning” is taken as any change in behaviour, information, knowledgeknowledge, understanding, attitudes, skills, or capabilities which can be retained and cannot be ascribed to physical growth or to the development of inherited behaviour patterns

Paris06/06/2007


4

Definitions: education output (2)

This handbook proposal “expected transfer of knowledge and skills provided by an

education unit, for each type of level of education / curriculum” Transfer of knowledge and skills can be seen as the true quantity

indicator, and change in academic scores due to school could provide a right measurement.

Expected, we could say « average », means that each pupil receiving the same teaching is supposed to consume the same output, although some pupils will assimilate more than others (what the Eurostat handbook called « successfully or not »). This difference in outcome is supposed due to the pupil/consumer.

The ISCED-97 levels of education and secondarily fields of education define the natural strata.

Paris06/06/2007


5

ISCED-97 levels of education

Level 0 Pre-primary education

Level 1 Primary educationor first stage of basic education

Level 2 Lower secondaryor second stage of basic education

Level 3 (Upper) secondary education

Level 4 Post-secondary non-tertiary education

Level 5 First stage of tertiary education

Level 6 Second stage of tertiary education

Paris06/06/2007


6

ISIC rev. 4 and ISCED 97

ISIC rev 4 classes ISCED-97 levels of education

8510 Pre-primary and primary education

Levels 0 and 1

8521 General secondary education Levels 2 and 3 oriented general

8522 Technical and vocational secondary education

Levels 2 and 3 oriented vocational and technical

8530 Higher education Levels 4, 5 and 6

8541 Sports and recr. education

Not classified in ISCED-97 levels of education

8542 Cultural education

8549 Other education n.e.c.

8550 Educational support activities Not explicitly mentioned in ISCED-97 levels of education

Fo

rma

l edu

catio

n (o

ur fo

cus)

info

rma

l

Su

ppo

rt

act

iviti

es

But no R&D or adm., like in COFOG or UOE

Paris06/06/2007


7

Basic formula

Ivolume

= Iquantity (number of enrolled pupils)

x Iquality ed (average transfer of knowledge)

x Iquality non ed (total expenditures / core educational) Without information on the transfer of knowledge, we find the

number of pupils, 2nd best quantity indicator for Eurostat. If we assume that the transfer is continuous and proportional

to time of teaching, we get the “pupil x hours”, 1st best quantity indicator for Eurostat.

Non educational quality is theoretically better estimated by an additive formula if detail by secondary product is known.

Paris06/06/2007


8

Consistency between quantityand quality indicators

If pupil-hours are the quantity indicator and change in scores the quality indicator, it leads to a double accounting:

Imagine a pupil with an attendance of 90% instead of 100%. As he has received 10% less teaching, it is probable that he will perform

only 90% of the change in score of his “normal” schoolmates. If we cumulate the quantity effect (pupil hours) and the quality indicator

(change in scores), we obtain 81% instead of 90%, the expected transfer of knowledge for this pupil.

transfer of knowledge= number of pupils getting 1 year older * average transfer of 1 year= number of pupils moving up 1 year * average transfer of 1 grade

If the quality indicator relies on scores at a given age, the quantity indicator should be the simple number of enrolled pupils. If it relies on scores at a given grade, the quantity indicator should be the number of pupils moving up.

Paris06/06/2007


9

Overview of inputs, output, outcomes in volume

Inputs Activities / Processes

Output

Direct outcome

Indirect

outcomes

Number of teachers and

staff, by qualification

Pupil-hours, class size, pedagogy

Employment, future real

earnings from labour market

growth of GDP

Number of teachers

Pupil-hours, class size, pedagogy

Transfer of skills and

knowledge by levels of education

Knowledge and skills:

estimated by scores, degrees

Well-rounded

citizens ?

other envir.

factors

Pupils: inherited skills, family backgr.,

entry educ. status, gender?

Methods Input

Output w/o expl. outcome-

based quality adjustment

Output with expl. outcome-based

quality adjustment

Educ. outcome: assessment

and achievement

Human capital, others

Aggregates National accounts Welfare / policy

productivity

Cost-effectiveness

Paris06/06/2007


10

Education output = average transfer of knowledge and skills, some remarks

No accounting of the distribution of individual levels of knowledge and skills, although “reduction of inequalities” is usually one of the targets of the education policy: consistent with the idea of “individual services” and common rules in NA for FC, income, etc.

No accounting of healthy, wealthy, well-rounded people / citizens, which are sometimes difficult to measure and which are more “indirect” outcomes.

We measure usually education status (outcome) at the end of a curriculum, rarely at the beginning, we need therefore a model (part of the pupil / consumer, time-lag) in order to estimate the part of the output in the outcome.

Paris06/06/2007


11

The model: education output and outcome

chart 1: Education status with age

0

100

200

300

400

500

600

700

0 3 6 9 12 15 18age

leve

l of k

now

ledg

e an

d sk

ills

inherited skills family contribution education unit better school

valu

e ad

ded

by th

e ed

ucat

ion

unit

exit

- en

try

in th

e ed

ucat

ion

syst

empr

imar

y +

seco

ndar

y ed

ucat

ions

Paris06/06/2007


12

Size of the SD: does it matter for NA?

The SD of the distribution of education levels can be standardized to 10% or 20% of the average

PISA, PIRLS, TIMSS standardized to 20%, US NAEP to 10%. The human capital theory (relying on future real earnings) suggests

8-10% (2 studies) or 20% (1 study).

But it does not matter in a NA view The convention for the relative size of 1 SD has an impact both on

cross-section (better school vs average school) and longitudinal distances (between ages 6 and 15 for instance, which will provide a measurement for average schools).

The quality adjustment in equivalent number of years of teaching is a ratio between cross-section and longitudinal distances.

Paris06/06/2007


13

Relative scores across agesAcademic scale across ages with 1 SD = 10%

0

0.002

0.004

0.006

0.008

0.01

0.012

age 15 age 12 age 9 age 6

1 SD at 15

1 SDat 12

1 SDat 9

Academic scale across ages w ith 1 SD = 20%

0

0.001

0.002

0.003

0.004

0.005

0.006

0.007

0.008

0.009

age 15 age 12 age 9 age 6

1 SD at 15

1 SD at 12

1 SD at 9

The larger the SD at age 15, the larger the output between ages 6 and 15, and constant proportions between numerator and denominator of QA.

Paris06/06/2007


14

=> For PPP, let us scale the average outcomeat age 15 = 500 and 1 SD = 20% (PISA)

PISA measures a national average outcome at age 15. To reflect the output of the national school systems, it should be neutralized of the family contribution (ESCS correction) and subtracted of inherited skills. The formula should be: ESCS corrected PISA – X.

Box 2.1: The PISA Economic, Social and Cultural Status (ESCS)

The PISA index of economic, social and cultural status (ESCS) is designed to capture broad aspects of a student's family and home background. It is derived from sub-indices based on:

- i) the highest occupational status of the student's parents (ISEI: the International Socio-Economic Index of Occupational Status);

- ii) the highest level of education of the parents converted in years of schooling;

- iii) an index based on home possessions (in fact three sub-indices: the PISA index of family wealth; the PISA index of home educational resources; and the PISA index of possessions related to “classical” culture in the family home).

Paris06/06/2007


15

Estimation of what is to be subtracted to ESCS corrected PISA scores: at least 180

Ideally, we should observe inherited skills at age 15, or more exactly inherited skills + average family contribution (for ESCS only neutralizes the differences in family contribution) at age 15. Requires a sophisticated model.

But we can assume that education status at age 6 reflects only inherited skills + average family contribution. Can be extrapolated with the help of 2 national longitudinal tests: the US NAEP and the UK 4 fey stages, which suggest about 180 at age 6 in average (with a convention of 1 SD = 20%).

Inherited skills + average family contribution can increase between ages 6 and 15, so that we could subtract more.

Paris06/06/2007


16

Age

US NAEP(1 SD = 10%)

UK 4 key stages(1 SD ~ 20%)

re-scaled US(age 15 =

500,1 SD = 20%)

re-scaled UK(age 15 =

500,1 SD = 20%)

re-scaled US (age 15 =

500,1 SD = 10%)

re-scaled UK(age 15 =

500,1 SD = 10%)

6 (203) (12.52) 182 180 352 340

7 15.51 217 223 368 362

8 253 267 385 383

9 231 288 310 401 405

10 323 353 418 426

11 27.49 359 396 434 448

12 394 426 451 463

13 429 456 467 478

14 33.79 465 487 484 493

15 (288) (34.71) 500 500 500 500

16 35.63 535 513 516 507

17 307 571 527 533 513

impact of 1 SD on output (age 15 - 6) 34% 31% 31% 31% 34% 31%

Paris06/06/2007


17

Time-lag management

grade \ year of outcome

2000 2001 2002 2003

9th grade (14-15 year old)

PISA 2000


2/3 PISA 2000 + 1/3 PISA 2003


1/3 PISA 2000 + 2/3 PISA 2003


PISA 2003

Total 2000: 2 PISA 2000 + 2 PISA 2003Total 2001: PISA 2000 + 8/3 PISA 2003 + 1/3 PISA 2006

Lower secondary education

Paris06/06/2007


18

Time-lag management and revisionsYears Lower secondary

(hypothesis of 4 years)Primary(hypothesis of 5 years)

2000 PISA 2000then ½ PISA 2000 + ½ PISA 2003

PISA 2000 then PISA 2003

then 1/5 PISA 2003 + 4/5 PISA 2006

at last 1/5 PISA 2003 + 3/5 PISA 2006+ 1/5 PISA 2009

2001 PISA 2000

then 1/4 PISA 2000 + 3/4 PISA 2003



then 1/15 PISA 2003 + 14/15 PISA 2006


2002 PISA 2000

then 1/12 PISA 2000 + 11/12 PISA 2003

at last 1/12* PISA 2000 + 2/3* PISA 2003 + 1/4* PISA 2006

PISA 2000 then PISA 2003 then PISA 2006

then 6/15 PISA 2006 + 9/15 PISA 2009


2003 PISA 2003

then ½ PISA 2003 + ½ PISA 2006


then 1/5 PISA 2006 + 4/5 PISA 2009


Paris06/06/2007


19

Other outcome-based quality indicators

Although we think that academic scores are the first best direct education outcomes, some curricula could be better estimated (or are more traditionally estimated) by obtainment of a degree or professional insertion (this latter an indirect outcome).

Professional insertion: employment rate (in the right qualification if possible) and real earnings can be used (multiplied) and are often suggested by human capital theory (discounted sum of future real earnings as return on “GFCF” in education), but they are to be considered in an incremental point of view: what is obtained after the curriculum minus what would have been obtained without, if we want to measure the “output”.

Paris06/06/2007


20

Which output/outcome for tertiary education? Degrees are certainly the most natural outcome indicators

for tertiary education, but they are of different “values”. Harmonization of degrees in Europe with the Bologna

process (bachelor, master, doctorate), but what are the relative values of these 3 degrees?

Distinguish by (ISCED-97) field of education; Double degree for one curriculum: double outcome? Curriculum shared between different countries/units; The output should not be null for students leaving without

degree. Time-lag adjustment. Number of credits reflect perhaps more directly the “quantity

of transfer of knowledge and skills”, i.e. our definition of the output, and avoid most problems quoted above. Still need to distinguish by field of education and perhaps by “prestige” of university.

Paris06/06/2007


21

If not “outcome-based” then “outcome-oriented”

If the output is defined as the attributable contribution of the producer to the outcome, and if the outcome cannot be directly measured, then some “inputs-based” or “process-based” quality indicators could be used, but “outcome-oriented”, it means converted in an expectation of incremental contribution to the outcome.

Example of the Italian model of the class size (see item 3.3). If the more classical way of treating different qualities as

separate goods was chosen, it would lead to a “quality presentation” but not to a “quality adjustment”: the price index would be nothing else than a cost weighted combination of cost indices, i.e. the same price index as in the input method.

Paris06/06/2007


22

Stratum for time-series Quantity Quality (ed) CommentPre-primary education Pupil-hours None Primary education: normal

Pupils

Contribution to scores

The sub-stratification of ISCED levels could be replaced by an estimate of the non-educational part (social services)

Primary education: special

Lower secondary: normal

Lower secondary: special

Upper secondary education: general + pre-technical or pre-vocational

Do not forget to subtract the entry education status

Upper secondary education: vocational

Employment rate after 1 year and/or real earnings if no scores available

In an incremental point of view and net of the economic trend

Post-secondary non-tertiary education

More occup. specific prog. tertiary education (5B)

More theoretically based programmes tertiary education (5A+6)

Credits 1st best

Degrees 2nd best

Pupils 3rd best

by field of ed.

“value” of degrees?

Paris06/06/2007


23

Stratum for cross-country Quantity Ed. quality Non ed. quality

ISCED 0: Pre-primary education

Pupils

(pupil-hours?)

None Should come in a future step from the UOE data collection, based on the “ancillary services”, afterreconciliation with NA.

ISCED 1: Primary education Pupils ESCS corrected average PISA score- 180

ISCED 2: Lower secondary: normal

Pupils

ISCED 3: Upper secondary education

Pupils

ISCED 4: Post-secondary non-tertiary education

Pupils None

ISCED 5B: More practical and occupationally specific programmes tertiary education

Pupils None at present

ISCED 5A and 6: More theoretically based programmes tertiary education

Pupils

(ECTS?)

(degrees?)

None at present (expectation of degrees?)

1 Paris 06/06/2007 2 nd OECD workshop on measuring Education and Health volume output Alain GALLAIS, OECD/STD/NAFS Draft chapter 2 on education output.

Documents

measuring education

education services

education unit

type of education

fields of education

area of education

cultural education

levels of education