Wilson Coding Protocol Page 1 Development of Coding Protocol Coding protocol: essential feature of meta-analysis Goal: transparent and replicable description.

Wilson Coding Protocol Page 1

Development of Coding Protocol

Coding protocol: essential feature of meta-analysis Goal: transparent and replicable

description of studies extraction of findings


Topics for Coding

Eligibility criteria and screening form Development of coding protocol Hierarchical nature of data Assessing reliability of coding Training of coders Common mistakes


Study Eligibility Criteria

Flow from research question Identify specifics of:

Defining features of the program/policy/intervention Eligible designs; required methods Key sample features Required outcomes Required statistical data Geographical/linguistic restrictions, if any Time frame, if any

Also explicitly states what is excluded


Study Eligibility Screening Form

Develop a screening form with criteria Complete form for all studies retrieved as potentially

eligible Modify criteria after examining sample of studies

(controversial) Double-code eligibility Maintain database on results for each study screened Example



Goal of protocol Describe studies Differentiate studies Extract findings (effect sizes if possible)

Coding forms and manual Both important



Iterative nature of development Structuring data

Data hierarchical (findings within studies) Coding protocol needs to allow for this complexity Analysis of effect sizes needs to respect this structure Flat-file (example) Relational hierarchical file (example)


Example of a Flat File

ID Paradigm ES1 DV1 ES2 DV2 ES3 DV3 ES4 DV422 2 0.77 323 2 0.77 331 1 -0.1 5 -0.05 5 -0.2 1136 2 0.94 340 1 0.96 1182 1 0.29 11

185 1 0.65 5 0.58 5 0.48 5 0.068 5186 1 0.83 5204 2 0.88 3229 2 0.97 3246 2 0.91 3274 2 0.86 3 -0.31 3 0.79 3 1.17 3295 2 7.03 3 6.46 3 . 3 0.57 .626 1 0.87 3 -0.04 3 0.1 3 0.9 3

1366 2 0.5 3

Note that there is only one record (row) per study

Multiple ESs handled by having multiplevariables, one for each potential ES.


Example of a Hierarchical Structure

ID PubYear MeanAge TxStyle100 92 15.5 2

7049 82 14.5 1

OutcomeID ESNum Type TxN CgN ES

100 1 1 24 24 -0.39100 2 1 24 24 0100 3 1 24 24 0.09100 4 1 24 24 -1.05100 5 1 24 24 -0.44

7049 1 2 30 30 0.347049 2 4 30 30 0.787049 3 1 30 30 0

Note that a single record in the file above is “related” to five records in the file to the right

Study Level Data File

Effect Size Level Data File


Example of a More Complex MultipleFile Data Structure

ID PubYear MeanAge TxStyle100 92 15.5 2

7049 82 14.5 1

Study Level Data File Outcome Level Data FileID OutNum Constrct Scale

100 1 2 1100 2 6 1100 3 4 2

7049 1 2 47049 2 6 3

ID OutNum ESNum Months TxN CgN ES100 1 1 0 24 24 -0.39100 1 2 6 22 22 0100 2 3 0 24 24 0.09100 2 4 6 22 22 -1.05100 3 5 0 24 24 -0.44100 3 6 6 22 21 0.34

7049 1 2 0 30 30 0.787049 1 6 12 29 28 0.787049 2 2 0 30 30 0

Effect Size Level Data FileNote that study 100 has 2 records in the outcomes data file and 6 outcomes in the effect size data file, 2 for each outcome measured at different points in time (Months)


Advantages & Disadvantages of Multiple Flat Files Data Structure Advantages

Can “grow” to any number of ESs Reduces coding task (faster coding) Simplifies data cleanup Smaller data files to manipulate

Disadvantages Complex to implement Data must be manipulated prior to analysis Must be able to select a single ES per study for any analysis

When to use Large number of ESs per study are possible


Concept of “Working” Analysis Files

Study Data File

Outcome Data File

ES Data File

Composite Data File

createcompositedata file

select subset of ESs of interest to current analysis,e.g., a specific outcome atposttest

verify that there is only asingle ES per study

yes

Working Analysis File

Permanent Data Files

Average ESs, further selectbased explicit criteria, orselect randomly

no


Example: SPSS ES Data File


Example: SPSS ES+Outcome Data File


Example: SPSS ES+Outcome+Study Data File


Example: Creating Subset for Analysis


Example: Final Working File fora Single Analysis


Concept of “Working” Analysis Files

Study Data File

Outcome Data File

ES Data File

Composite Data File

createcompositedata file

select subset of ESs of interest to current analysis,e.g., a specific outcome atposttest

verify that there is only asingle ES per study

yes

Working Analysis File

Permanent Data Files

Average ESs, further selectbased on explicit criteria, orselect randomly

no


What about Sub-Samples?

What if you are interested in coding ESs separately for different sub-samples, such as, boys and girls, or high-risk and low-risk youth, etc? Just say “no”!

Often not enough of such data for meaningful analysis Complicates coding and data structure

Well, if you must, plan your data structure carefully Include a full sample effect size for each dependent measure

of interest Place sub-sample in a separate data file or use some other

method to reliable determine ESs that are statistically dependent


Coding Mechanics

Paper Coding (see Appendix E) include data file variable names on coding form all data along left or right margin eases data entry

Coding into a spreadsheet Coding directly into a computer database


Coding Directly into a Computer Database Advantages

Avoids additional step of transferring data from paper to computer

Easy access to data for data cleanup Data base can perform calculations during coding process (e.g.,

calculation of effect sizes) Faster coding

Disadvantages Can be time consuming to set up

the bigger the meta-analysis the bigger the payoff Requires a higher level of computer skill


Example of Database with Forms


Assessing Reliability of Coding

Inter-rater reliability and double coding Intra-rater reliability


Training Coders

Regular meetings (develops normative understandings) Annotate coding manual “Specialist” coders


Common Mistakes

Not understanding or planning the analysis prior to coding

Underestimating time, effort, and technical/statistical demands

Using a spreadsheet for managing a large review Variable names not on coding forms Not breaking apart difficult judgments


Common Mistakes

Over-coding—Trying to extract more detail than routinely reported


Comments on Managing the Bibliography Major activity Information you need to track

source of reference (e.g., PsychLit, Dissertation Abs.) retrieval status

retrieved, requested from ILL, etc. eligibility status

eligible not eligible relevant review article

coded status

Word processor not up to the task Spreadsheets are cumbersome Use a database of some form

Wilson Coding Protocol Page 1 Development of Coding Protocol Coding protocol: essential feature of meta-analysis Goal: transparent and replicable description.

Documents

wilson coding protocol

studies coding protocol

outcomes data file

effect size data file

coding task faster coding

coding eligibility criteria

possible coding forms

flat file note