ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany Training Workshop on the ICCS 2009 database Weights and Variance Estimation picture.

ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany

Training Workshop on the ICCS 2009 database

Weights and Variance Estimation

picture

Content of this presentation

WeightsWhat are they?Why do we need them?How do we use them?

Standard errorsWhat are they?Why do we need them?How do we estimate them?

What are weights?

Values assigned to all sampling units.Students, teachers, schools.

The weight of a sampled unit indicates the number of units in the population that is represented by this sampled unit.

Weights and selection probabilities

Weights are based on the sample selection probabilities.ICCS sampling units had different selection probabilities.

High selection probability small weight, Low selection probability large weight.

Weights get applied at each sampling stage.School sampling weights,Within-school sampling weights.

Weights and non-participation adjustments

Weights are adjusted for non-participation.Non-participation can happen at each level.

School level,Within-school level.

Non-participation adjustments compensate losses for specific groups of sampled units.

Why do we need weights?

Weights allow conclusions to be drawn about the population based on information from the sample.Weights allow estimates of population parameters.Un-weighted data only allow conclusions about the sampled units.

Population sample

Example: in the population, 20% of the students are in private schools (red), 80% are in public schools (blue).In the sample, 50% of the students are from private schools, 50% are from public schools.

Sample estimate

In order to estimate the correct proportion of students in the population, different weights must be assigned to the students in the sample.

Analyzing weighted data – a simple example

1:101:1

Un-weighted mean

nx

x )(Mean Unweighted

Weighted mean

wgtxwgt

x )(Mean Weighted

Weights in ICCS

The ICCS Data contain several weight variables. Total Student Weight: TOTWGTS,Total Teacher Weight: TOTWGTT,Total School Weight: TOTWGTC.

The IDB Analyzer automatically selects the correct weight.

ICCS example: civic knowledge in Chile

Unweighted: average of 493.83

ICCS example (cont.)

Difference: 10.8 score points.Reason for the difference: over-sampling of students in private schools.

13.7% of the tested students,5.9% of the sum of weights.

What are standard errors?

The standard error of a statistic is the standard deviation of the sampling distribution of that statistic.The sampling distribution is the distribution of the statistic for all possible samples of the same size and method.Since no one can select all possible samples, the standard error can only be estimated.

Title of presentation: "Weights and Variance Estimation"

Why do we need standard errors?

The ICCS results are based on samples.All ICCS results are therefore estimates of unknown population values.Standard errors can be used to measure how close these estimates are to the real values.

Standard errors and confidence intervals

Let ε stand for any statistic of interest (mean, percentage...)A 95% confidence interval is defined as

This is the black bar in Table 3.10 of the ICCS International Report.

With a confidence of 95%, the true mean is between 554.3 and 563.7...Take rounding into account!

SE 96.1

Standard errors and significance tests

Whenever you see a funny little triangle in the ICCS international report, standard errors are hidden in the background.

Estimating standard errors

In a simple random sample, estimating the standard error of a statistic ε is easy.

Just divide the standard deviation of the sample (s) by the square root of the sample size (n)

In a complex sample design like in ICCS, it is not as easy to estimate the standard error as in a simple random sample.

nsSE

^

Complex sample design - clustering

Clustered sample:Students within a school are likely to be more similar to each other than students from different schools.

Similarly for teachers.Clustering usually decreases sampling precisison.

Complex sample design – strata and weights

Stratification:Students within a stratum are more similar to each other than students from different strata.Similar for schools and teachers.Stratification usually increases sampling precision.

Weights:Using weights usually decreases sampling precision.They complicate the calculations.

Why not just use SPSS?

Standard software packages like SPSS will not provide correct estimates for standard errors.The software assumes that the data is from a simple random sample, and uses the incorrect formula.Generally, the estimate will be too small.

Jackknife Repeated Replication

Solution: Jackknife Repeated Replication (JRR).Used for estimating standard errors in complex designs.Basic idea: systematically re-compute a statistic on a set of replicated samples:

By setting the weights to zero for one school at a time,While doubling the weights of another school.

Estimate the standard error of a statistic from the variability of that statistic between the full sample and the replicates.

The basics of the JRR method

Jackknife variance estimation in ICCS:Participating schools are paired according to the order in which they were sampled.These school pairs are called jackknife zones.

JKZONES, JKZONET, JKZONECOne school in each zone is randomly assigned an indicator of 1 (0 for the other school).

JKREPS, JKREPT, JKREPCThis indicator decides whether a school gets its replicate weight doubled or zeroed.

A look inside the IDB Analyzer

Standard error:

2 3 4 5 67 8 9 10 11 1213 ...14 75...

75

1i

2i εεSE

1

^

ICCS example: teacher age in Chile

Standard error of the teacher age in Chile.SPSS just can’t do that:

Standard errors and plausible values

For ICCS achievement data, the standard error consists of two components.Sampling error:

This is what we just discussed.

Addtionally, measurement error:Resulting from the use of plausible values.

This is the topic of the next presentation.

Conclusion

Use sampling weights!

Compute standard errors using the JRR!

Thank you for your attention!