Data Management Seminar, 8-11th July 2008, Hamburg Data Entry Overview and Changes from the Field Trial.

Data Management Seminar, 8-11th July 2008, Hamburg

Data Entry OverviewData Entry Overviewand Changes from the Field and Changes from the Field

TrialTrial

Data Management Seminar, 8-11th July 2008, Hamburg2

The ICCS MS Data Entry Process

Data Entry Part 1– Entering School Data– Entering Teacher Data– Entering Student Regional Module Data

Data Entry Part 2– Entering Student Questionaire Data

Data Entry Part 3– Entering Student Achievement Data


Table of Content

1. Changes in Data Entry Process since Field Trial

2. Organizing the Data Entry Process 3. Data Manager’s Responsibilities 4. Necessary Material for Data Entry 5. Control Receipts of Survey Material 6. Organize Material for Data Entry 7. Organize Material for Scoring8. The Process of (Reliability) Scoring


Changes in Hierarchical ID System

School level– 8001, 8002, 8003...

Teacher level– 800101, 800102...

805201, 805202...

Student level– 80010101, 80010102...

80520101, 80520102...

School level– 1001, 1002, 1003...

Teacher level– 100101, 100102...

105201, 105202...

Student level– 10010101, 10010102...

10520101, 10520102...

Field trial Main Survey


One form of Student Questionnaire

Student

Questionnaire

Form A

Fie

ld t

rial

Student

Questionnaire

Form B

Student

Questionnaire

Form C

Main

Su

rvey

Student

Questionnaire


One form of Student Questionnaire

Codebook has:– One form of Student Questionnaire inside– No IDQUEST variable

Thus...


Introduction of Coding Module (ISG)

Fie

ld t

rial


Main

Su

rvey



Advantages– Entering the Occupational Data using

WinDEM– More convenient for the Data Entry Staff– Better coordination between the Puncher

and the Coder– Higher Data Entry speed– Less errors



Number of Achievement Booklets

Fie

ld t

rial

Main

Su

rvey

6 bookletsReliability booklets – only 1, 2, and 3

7 bookletsReliability booklets –1 through 7


Table of Content


2. Organizing the Data Entry Process 3. Data Manager’s Responsibilities 4. Necessary Material for Data Entry 5. Control Receipts of Survey Material 6. Organize Material for Data Entry 7. Organize Material for Scoring 8. The Process of (Reliability) Scoring


Data Documentation

Data

Organizing the Data Entry Process

Instruments/ Forms

InternationalCodebooks

WinW3S File

Data Entry Rules

...

...

DEUnit

Data Manager

12


Table of Content


2. Organizing the Data Entry Process3. Data Manager’s Responsibilities 4. Necessary Material for Data Entry 5. Control Receipts of Survey Material 6. Organize Material for Data Entry 7. Organize Material for Scoring 8. The Process of (Reliability) Scoring


Data Manager’s Responsibilities

Data EntrySoftware

Source ofInformation

Hardware

Software

Adaptation ofCodebooks

Verification

Documentation

Instruments

TrackingForms

Scoring Material

Organize:

Training in: Organize: Perform:

Send Out

MaterialsData Entry

StaffEquipment Data Files

Data Manager

14


Table of Content




From ICCS International Study Centrehttp://iccs.acer.edu.au/International codebooks

- Questionnaire codebooks - Achievement and reliability codebooks

Survey Operations Procedures (Unit 3) Windows Within-School Sampling

Software (WinW3S)Windows Data Entry Manager (WinDEM)

program IEA Survey System

Necessary Material for Data Entry

16


Important documents for data entry– Survey Operations Procedures (Unit 3)

Copy relevant parts of documents for key punchersCopies of survey tracking forms should be available for key punchers

Prepare Key Punchers for Data Entry

17


Planning for Required Resources

Staff for scoring and data entry depends on…– How much material?– How much time?– How much experience?– How much money?– How many persons available?

Computer requirements– IBM-compatible (Pentium processor or higher) – OS: Windows XP/Vista– Office 97 or higher

Ask countries that participated in previous IEA studies for their

experiences!

18


Amount of MaterialsScoring: Scorer

7 Achievement Booklets =

~ 3750 Student Achievement Test Booklets

~ 700 Student Achievement Test Booklets for Reliability Scoring

~ 150 School Questionnaires

~ 2250 Teacher Questionnaires

~ 3750 Student Questionnaires

~ 3750 Regional Module Questionnaires

~ 3750 Student Achievement Test Booklets

~ 700 Reliability Scoring Sheets

Data Entry: Key Punchers

19


Rece

ivin

g M

ate

rial

InstrumentPreparation

Test

Adm

inis

trati

on

Codebook Adaptation

Rece

ivin

g M

ate

rial

Data

Su

bm

issi

on

Sampling and Preparing

Test AdministrationScoring

Data Entryand Verification

20


Rece

ivin

g M

ate

rial

21


Table of Content




Control Receipts of Test Material

Record return date on ICCS MS School Sampling Form

23

Contact schools that have not returned testing material or where the material is incomplete!


Control Receipts of Test MaterialCheck that all survey tracking forms have been returned and completed correctly

...

...

24

Check participation statusparticipation status on tracking forms!

Contact schools that have not returned tracking forms or returned incomplete!


Control Receipts of Test MaterialCheck that all booklets/ questionnaires have been returned

Verify identification codesidentification codes on instruments!

Contact schools that have not returned all testing instruments!

25


Table of Content


2. Organizing the Data Entry Process3. Data Manager’s Responsibilities 4. Necessary Material for Data Entry 5. Control Receipts of Survey Material6. Organize Material for Data Entry 7. Organize Material for Scoring 8. The Process of (Reliability) Scoring


OrganizeMaterial

for Data Entry

Rece

ivin

g M

ate

rial

27


Separate questionnaires and tracking forms from booklets!Sort and forward…

Organize Material for Data Entry and Scoring

Forms + Questionnaires to

Data Entry Staff

Booklets to

Scorers

...

...

28

Data Management Seminar, 8-11th July 2008, Hamburg

Forms + Questionnaires to

Data Entry Staff

...

Sorting Forms and Questionnaires

29


1 0 0 1 0 1 0 1

1 0 0 1

1 0 0 1 0 1

1 0 0 1 0 1 1 0 0 1

0 1 0 1

0 1

1 0 0 1 0 1

1 0 0 1

Hierarchical ID SystemIDSCHOOL: 4 digits school

IDCLASS: 6 digits school + class

IDSTUD: 8 digits school + class + student

IDTEACH: 6 digits school + teacher

Unique identifier in ICG file

Unique identifier in xSG, xSE/xSL, xSA, xSR file

Unique identifier in ITG file

1 0 0 1

30


1004

Arrange Instruments for Data Entry

School Questionnaires

Sort by IDSCHOOL

...

31

1003

1002

1001


100102

Teacher Questionnaires

Sort by IDTEACH

...


32

100101

School 1001

100202

100201

School 1002


...

Sort by IDSTUD

Student Questionnaires


33

10010103

10010102

10010101

IDSCHOOL 1001IDCLASS 100101

10010203

10010202

10010201


10020103

10020102

10020101

IDSCHOOL 1002IDCLASS

100201

10020203

10020202

10020201


100202


...

Sort by IDSTUD

Regional Module Student Questionnaires


34

10010103

10010102

10010101


10010203

10010202

10010201


10020103

10020102

10020101


100201

10020203

10020202

10020201


100202


Table of Content


2. Organizing the Data Entry Process3. Data Manager’s Responsibilities 4. Necessary Material for Data Entry 5. Control Receipts of Survey Material6. Organize Material for Data Entry7. Organize Material for Scoring 8. The Process of (Reliability) Scoring


OrganizeMaterial

for Data Entry

OrganizeMaterial

for Scoring

Rece

ivin

g M

ate

rial

36


Sorting Booklets for ScoringTwo Teams for Scoring: A and B

Booklets to

Scorers

Scoring Team A

Scoring Team B

37


...

1,2,...,6,7

Steps for Booklet Arrangement

Step 1: Sort by Booklet Type

38


Arrange Test Booklets for Scoring


Booklet

1

Booklet

1

Booklet

1

Booklet

1

Booklet

2

Booklet

2

Booklet

2

Booklet

2

Booklet

5

Booklet

5

Booklet

5

Booklet

6

Booklet

6

Booklet

6

Booklet

6

Booklet

7

~536 ~536 ~536

~3750

~536

...

...

...


Step 2: Divide into Two Sets ...

Set A, Set B



Note: From now on the example in this presentation will focus on

booklet 1 only!



Step 2: Divide into Two Sets

Booklet

1

Booklet

1

Booklet

1

Booklet

1~268

~268

Booklet

1

Set A

Booklet

1

Set A

Booklet

1

Set A

Booklet

1

Set A

Booklet

1

Set B

Booklet

1

Set B

Booklet

1

Set B

Booklet

1

Set B

~536

Scoring Team A

Scoring Team B

~50%

~50%


Step 3: Divide into Batches ...

Batch 1, Batch 2, ...






Step 3: Divide into Batches

~268

Booklet

1R

Set A

Booklet

1

Set A

Booklet

1R

Set A

Booklet

1

Set A~ 25

Booklet

1R

Set A

Booklet

1

Set A

Scoring Team A

~ 25

Booklet

1R

Set A

Booklet

1

Set A

~ 25

Booklet

1R

Set A

Booklet

1

Set A~ 25

Booklet

1R

Set B

Booklet

1

Set B

~ 25

Booklet

1R

Set B

Booklet

1

Set B

~268

Booklet

1R

Set B

Booklet

1

Set B

Booklet

1R

Set B

Booklet

1

Set B

Scoring Team B

~ 25

Booklet

1R

Set B

Booklet

1

Set B

... ...


Step 4: Separate “R” Booklets







~ 5

Booklet

1R

Set A

Scoring Team A

Scoring Team B

~20

Booklet

1

Set A

Booklet

1R

Set A

Booklet

1

Set A

~25

Booklet

1R

Set A

Booklet

1

Set A

Booklet

1R

Set A

Booklet

1

Set A

...




Scoring Team A

Booklet

1R

Set A

~ 5

As Main Scorers

As Reliability Scorers

Scoring Team B

~20

Booklet

1

Set A

Booklet

1R

Set A

Booklet

1

Set A

~20

Booklet

1

Set A

Booklet

1R

Set A

Booklet

1

Set A

...

~20

Booklet

1

Set A

Booklet

1R

Set A

Booklet

2

Set A

~20

Booklet

1

Set A

Booklet

1R

Set A

Booklet

2

Set A

...

...

Booklet

1R

Set A

~ 5

...

Booklet

2R

Set A

~5

Booklet

2R

Set A

~5

...

...



Table of Content


2. Organizing the Data Entry Process3. Data Manager’s Responsibilities 4. Necessary Material for Data Entry 5. Control Receipts of Survey Material6. Organize Material for Data Entry7. Organize Material for Scoring8. The Process of (Reliability) Scoring


Scoring


Test

Adm

inis

trati

on

Codebook Adaptation

Rece

ivin

g M

ate

rial

Data

Su

bm

issi

on


ScoringSampling

and PreparingTest Administration


ReliabilityScoring

MainScoring

Scoring


Step 6: Reliability Scoring

Step 5: Main Scoring (I)Reliabilit

y

Scoring

Sheet

Step 7: Return “R” Booklets

Step 8: Main Scoring (II)







Main ScoringStep 5: Main Scoring (I)

202

~218

Booklet

1

Set A

Booklet

1R

Set A

Booklet

1

Set A

~218

Booklet

1

Set A

Booklet

1R

Set A

Booklet

1

Set B

Scoring Team A

Scoring Team B


Reliability ScoringStep 6: Reliability Scoring

202

Scoring Team A

Scoring Team B

Booklet

1R

Set B

~50

Reliability

Scoring

Sheet

1R Set B

~50

Booklet

1R

Set A

~50

Reliability

Scoring

Sheet

1R Set A

~50


101

Booklet

1R

Set B

Reliability Scoring

101

1011001010

3974

2

Scoring Team A

Reliability

Scoring

Sheet

1R Set B


Return „R“ Booklets for Main Scoring

Step 7: Return “R” Booklets

Sco

rin

g T

eam

A

Sco

rin

g T

eam

B

Booklet

1R

Set A

~50

Reliability

Scoring

Sheet

1R Set A

~50~218

Booklet

1

Set A

Booklet

1R

Set A

Booklet

1

Set A

Booklet

1R

Set B

~50

Reliability

Scoring

Sheet

1R Set B

~50~218

Booklet

1

Set A

Booklet

1R

Set A

Booklet

1

Set B


Main ScoringStep 8: Main Scoring (II)

202

Scoring Team A

Scoring Team B

Booklet

1R

Set B

~50

Booklet

1R

Set A

~50


Scoring Team B

101

Documenting Main Scorer

202

Booklet

1R

Set B

202

1


Table of Content


2. Organizing the Data Entry Process3. Data Manager’s Responsibilities 4. Necessary Material for Data Entry 5. Control Receipts of Survey Material6. Organize Material for Data Entry7. Organize Material for Scoring 8. The Process of (Reliability) Scoring





Test

Adm

inis

trati

on

Codebook Adaptation

Rece

ivin

g M

ate

rial

Data

Su

bm

issi

on

ScoringSampling

and PreparingTest Administration


CompleteStudent

Information

EnterBackground Data

EnterAchievement and

Reliability Data



Thank You for Your Attention.Thank You for Your Attention.

60

Data Management Seminar, 8-11th July 2008, Hamburg Data Entry Overview and Changes from the Field Trial.

Documents

data documentation data

school data

teacher data

data management seminar

occupational data

hamburg data entry overview

unit data manager

coder higher data entry