Conducting Randomized Evaluations in Schools Lisa Markman Associate Director Education Research Section Princeton University.

Conducting Randomized Evaluations in Schools

Lisa MarkmanAssociate DirectorEducation Research SectionPrinceton University

Education Research Section

• Promotes the use of research in education decision-making.

• Conducts and promotes high quality education research through the use of experimental and quasi-experimental research designs

No Child Left Behind, Schools and Research

• No Child Left Behind Act (2001) requires programs and practices to be based on “scientifically based research”.

• Term “scientifically based research” appears throughout the law over 100 times.

• Applies to curricular interventions, professional development programs, psychosocial interventions and supplemental education services.

What is Scientifically Based Research ?

The term scientifically based research means research that involves the application of rigorous, systematic, and objective procedures to obtain reliable and valid knowledge relevant to education activities and programs

(According to NJ Dept. of Ed)

Necessary Components of SBR

• Employs systematic, empirical methods that draw on observation or experiment

• Involves rigorous data analysis that are adequate to test the stated hypotheses and justify the general conclusions drawn

• Relies on measurement or observable methods that provide reliable and valid data

• Is evaluated using experimental or quasi-experimental designs with appropriate control or comparison groups (with a preference for random assignment)

• Reports results with sufficient detail and clarity• Has been subject to external (peer) review

(According to NJ Dept. of Ed)

Evaluations in Education

Undesirable Features of Many Evaluations

• No comparison group or comparison group is not comparable to treatment group

• Only “successfully” treated schools/students are included in the analysis

• The evaluators are not independent of the developers of the program

Desirable features of Evaluations

• Participants are randomly assigned to “treatment” status

• All initial participants are included in the analysis

• The evaluators are independent of the developers of the program

Recipe for an Ideal Measurement

Step 1: Randomly assign students.

Step 2: Obtain PERFECT test.

Step 3: Test students in fall.

Step 4: Test students in spring.

Step 5: Set non-school parameters back to original conditions.

Step 6: Repeat Steps 3 & 4.

Step 7: Impact =

(Step 4 – Step 3) Year 2 - (Step 4 – Step 3) Year 1

The “2nd Best” OptionRandomization

Randomly assign some students, classes, school districts, or states to the reform in question; the others are randomly assigned to the status quo (or another reform).

Why Randomize ?

To obtain an appropriate control group

Allows one to answer what would happen in the absence of the intervention

(creates a credible counterfactual)

Common Concerns About Randomized Evaluations

• It is too disruptive

• Unethical to deny service to some children

• Legal obligations require providing services to all children

• It does not answer all questions

• Fear of “random”

Is It Too Disruptive ?

• It can be.

• Good researchers work with teachers, schools and school districts to understand school culture, and school needs.

Example of a Randomized Research Evaluation

• To determine the effectiveness of a computerized math curriculum

• 3 Urban School districts

• Randomize on the class level by school

Teacher Period 1 Period 2 Period 3 Period 4

A Alg 1 Alg 1A Alg 1  

B Alg 1 Alg 1A   Alg 1A

C   Alg 1A  Alg I  

D  Alg I   Alg 1 Alg 1A

Randomized Evaluation of A Computerized Math Curriculum

Current School Schedule

Teacher Period 1 Period 2 Period 3 Period 4

A Alg 1 Alg 1A Alg 1  

B Alg 1 Alg 1A   Alg 1A

C   Alg 1A  Alg I  

D  Alg I   Alg 1 Alg 1A

Teacher Period 1 Period 2 Period 3 Period 4

A Regular Regular LAB  

B LAB LAB   Regular

C   Regular  Regular  

D  Regular   Regular LAB

Randomized Evaluation of A Computerized Math Curriculum

Current School Schedule

Below is a sample schedule that would be returned to each school after random assignment

It’s Unethical/Illegal to Deny Service

• We don’t know if program/reform actually works

• Ways around completely excluding all students

• All children receive “something” or can contrast two programs

It Doesn’t Answer All Questions

• True

• Benefit of Mix-Methods Approach

The Fear of “Random”

• Usually can not treat all students

• Way to organize groups to get meaningful information

To Sum….

• Need more high-quality impact evaluations to help guide everyday decisions

• Conducting evaluations does not have to be super disruptive

• Schools should make sure that researchers work with them

• It’ll take cooperation from all to improve the quality of education research on impacts

Information for Librarians

• What Works Clearinghouse

• Education Databases

What Works Clearinghouse• Created as a resource for decision makers in education. Provides

databases and reports regarding the effectiveness scientifically based programs and practices.

• Current topics – – Adult Literacy, – Character Education, – Delinquency, Disorderly and Violent behavior, – Dropout Prevention, – English Language Learners, – Math (Elementary and Middle School)– Peer Assisted Learning– Reading, – Early Childhood

• http://www.whatworks.ed.gov

Examples of Education Databases

• National Center for Education Statistics (NCES) – National Assessment of Educational Progress (NAEP) – High School and Beyond (HS&B)– The Common Core of Data (CCD)

– http://nces.ed.gov

• National Longitudinal Study of Youth (NLSY)– NLSY97– NLSY79

– http://www.bls.gov/nls

– Check websites for a list of all databases