3rd NRC Meeting, 9-12 June 2008, Windsor 1 ICCS 2009 Main Survey Sampling Quality Standards.

1 3rd NRC Meeting, 9-12 June 2008, Windsor

ICCS 2009 Main Survey

Sampling Quality Standards


Overview

• Why do we need sampling quality standards?

• Sampling quality standards

– For the student survey

– For the teacher survey

• Examples of how to meet the quality standards


Sampling quality standards

Quality standards require

low exclusion rates

high participation rates of

– schools

– students

– teachers


Why sampling quality standards?

Needed to avoid biased estimates

e.g., excluded groups of students could perform different from non-excluded students

e.g., non-responding teachers could have a different attitude towards citizenship than responding teachers

Needed to ensure comparability between countries

Needed to achieve quality of IEA surveys


Student survey: exclusion rates

Students can be excluded at three levels

Excluded schools before sampling

Excluded classes in sampled schools

Excluded students in sampled classes

Overall exclusions must not exceed 5%


Student survey – participating school

A school is regarded as participating if 50% of the sampled students replied.

In a few countries, two ore more classes will be sampled

For details, please join us in the country consultations


Student survey - participation categories

Three categories will be defined that describe the quality of the sample implementation

Of course, the aim is 100% of participation at all levels

The categories are defined as in other IEA student surveys

The categories refer to both school participation rates and student participation rates


Student survey participation rates – Category 1

A country will be placed in category 1 if

the school participation rate is ≥ 85% before using replacement schools

AND

the student participation rate is ≥ 85%.


Student survey participation rates – Category 2



AND

the school participation rate is ≥ 85% after using replacement schools

AND

the student participation rate is ≥ 85%.


Base of student participation rate

The base for each rate is different

The school participation rate is based on all sampled schools

The student participation rate is based only on the participating schools (sampled and replacement)


Student survey participation rates - second chance

Low response rates at one level can be compensated with high response rates at another level

If a country reaches 75% combined participation rate...

before using replacement schools: category 1

after using replacement schools: category 2


Example for combined participation rate

80% schools participation rate before replacement

95% student participation rate within schools

0.80 x 0.95 = 0.76

0.76 ≥ 0.75 OK!


Student survey participation rates – third chance

These requirements can be met

either with unweighted data

or with weighted data.

A country will be placed in category 3 if it fails to meet the requirements for category 1 and 2


Teacher survey

For the teacher survey, similar requirements apply

Teacher participation may be harder to achieve than student participation

However, there is no statistical justification to treat teacher data differently from student data


Teacher survey – participating school

A school is regarded as participating if at least 50% of the sampled teachers participate


Teacher Survey participation rates – Category 1



AND

the teacher participation rate is ≥ 85%.


Teacher Survey participation rates – Category 2



AND

the school participation rate is ≥ 85% after using replacement schools

AND

the teacher participation rate is ≥ 85%.


Teacher survey participation rates - second chance and third chanceLow response rates at one level can be compensated

with high response rates at another level

If a country reaches 75% combined participation rate...

before using replacement schools: category 1

after using replacement schools: category 2

All rates will be calculated unweighted and weighted

A country will be placed in category 3 if it fails to meet the requirements for category 1 and 2


Reporting policy

All data will be published!

Readers of the international report should be informed about the quality of sample implementation

Not meeting the requirements in one survey does not affect the reporting in the other survey

International Reports: different form of display

Category 1: without annotation

Category 2: annotated

Category 3: in a separate section of the tables


How can you enhance participation?

Examples of good practice

Taken from the SAQ and form other surveys

Some of these examples may work in your in your country


(1) Gain political support

Backing from the Ministry of Education

– Official bulletins

– On-line newsletters

– MOE website

– Ministry letters to schools, or even to individual teachers

Support by teacher unions

A steering committee underlines importance of the study


(2) Make yourself seen

Articles in teacher journals

Presentation of some FT results as examples

Flyers and brochures

Put emphasis on the international character of the study


(3) Dispel doubts

Explanation of the purpose of the study

Ensure confidentiality and data protection

Teacher meetings at sampled schools


(4) Provide motivation

Feedback to schools

Incentives to schools

Incentives to teachers

– Tokens

– Lottery


(5) Organize data collection best

Early school contact

Avoid testing in busy times of school year

Avoid testing at the same time as other surveys

Support school coordinator in his work

Support of school principal

Support of DPC


Thank you for your attention!

3rd NRC Meeting, 9-12 June 2008, Windsor 1 ICCS 2009 Main Survey Sampling Quality Standards.

Documents