Some Lessons Learned in Conducting Scientific Research on Behavior Disorders within Applied School Contexts By Hill M. Walker Ph.D. Institute on Violence.

Some Lessons Learned in Conducting Scientific Research on Behavior Disorders

within Applied School Contexts

ByHill M. Walker Ph.D.

Institute on Violence and Destructive BehaviorCollege of Education, University of Oregon

and The School Behavior Disorders GroupOregon Research Institute

Eugene, Oregon

The Fifth Annual Institute of Education Sciences (IES)Research Conference

June, 2010 – National Harbor, MD

Topics to Cover

• Pressures Driving Demands for Scientific Research Outcomes in School Settings

• Single-Case vs. Randomized Controlled Trials as Vehicles for Conducting Scientific Research in Schools

• Integrating Single Subject and RCT Designs in a 3-Stage Research and Development Process

• Some Conditions Necessary for a Randomized Control Trial to be Successful in Schools Contexts

Factors Driving Interest in EBPs

• National Legislation – No Child Left Behind– Individuals with Disabilities Education

Act (2004 Reauthorization)

• Court Mandates

• Threats to School Security

Factors (continued)

• Public Demand for Return on Investments in Federal Research

• Growing Interest in Prevention

– Especially as an Alternative to Specialized Accommodation and Special Education

• Development of Quality Standards by Professional Organizations

Development of innovations in better serving at-risk students has ramped up substantially in

the past decade

1. 3-tiered public health prevention model applied to school contexts

2. Advent in use of response to intervention approaches for screening, identification and treatment

3. Strong interest by psychologists in conducting school-based research on conduct disorders

4. Priority of adapting promising programs for routine usage in school practices

Where are Schools Re: New Evidence-Based Practices?

• Until the past decade, schools have been slow to adopt proven or promising EBPs.

Impact1. Vast majority of K-12 students have not

accessed effective interventions.

2. Emerging pressures on educators to adopt and implement best and preferred practices that have a solid evidence base.

Bridging the Gap Between Research and Practice

• Time lag between the availability of an EBP and its adoption and effective use on a broad scale within routine contexts can be vast.

-Cure for scurvy (Rogers, 1995).

-Time lag in mental health is estimated to be 20 years.

-Time lag is at least this long in K-12 education.

• Typical barriers to adoption-Cost

-Difficulty accessing-Philosophical objections-Resistance to change-Innovation takes too much time and effort

Changed Landscape of Prevention Science and

Public Demand for Reliable Evidence on What Works • Increasing demand for large-scale prevention studies that

demonstrate positive outcomes.

– Typically, RCTs are the preferred method of demonstrating such effects.

– Other types of methods and evidence are regarded as less rigorous.

• Prevention approaches and intervention studies that meet this standard are highly regarded and lead to many positive benefits.

• Blueprint Programs Example

Continuing Challenges for Advocates ofSingle Subject vs. Randomized Control Research Designs

• Need to demonstrate positive impact on individuals while also addressing population-based questions relating to public safety, health, education, social policy and so forth.

Sidman (2006) notes that:

• Influencing large groups is not where applied behavior analysis has made its greatest impact and progress.

• Not all problems are subject to the same kinds of solutions and approaches.

• Need to address the challenge of achieving widespread behavior change simultaneously.

Sidman calls for: • Alternative approaches that can accomplish broad,

population-based changes and adoption of new innovations.

• Examples are:

– Epidemiology

– Actuarial-based decision-making

– Scaling up promising interventions to evaluate their efficacy and effectiveness

– Large-scale studies of policy decisions

Referencing Outcomes in Behavioral Disorders to Macro-Societal Issues and Problems

(1) Can the BD field more directly target issues and problems of urgent societal concern?– School safety– Bullying, mean-spirited teasing, sexual harassment– Delinquency– Gang membership– School dropout– Youth violence– Drug abuse

Referencing Outcomes in Behavioral Disorders to Macro-Societal Issues and Problems (continued)

(2) Can the BD field implement outcome measures of these problems as part of documenting intervention effects?

(3) Examples of Longitudinal Initiatives that have done so:(a) Perry Preschool Program(b) Hawkins & Catalano 12- to 20-year intervention

follow-up study(c) Fast Track(d) Olds Nurse Home Visitation Model

Randomized Control Trials and Critical Errors

• RCTs defined--Randomized Controlled Trials are studies that randomly assign individuals to an intervention group or to a control group, in order to measure effects of the intervention.

• RCTs are considered the surest way to avoid making serious errors on heath, education and policy issues that can affect the public health and safety.– Pre/post only and control-comparison studies often produce

erroneous conclusions. – Random assignment allows one to evaluate whether the

intervention itself, or some other set of factors, causes the observed outcomes.

Aldous Huxley

• “The Single Greatest Tragedy of Science is the Cold-Blooded Slaying of a Beautiful Theory by an Ugly Fact.”

– Randomized Control Trials are often the Cruel Means by Which this Tragedy Occurs.

Seminal RCT-Derived Medical and Educational Findings

Medical: vaccines for polio, measles, and hepatitis B interventions for hypertension and high cholesterol cancer (Hodgkins, leukemia, multiple myeloma)

Educational: reduced class sizes in grades K-3 reading improvement and phonics high quality early childhood development programs

Use of RCTs in Educational Contexts

• Researchers have had considerable difficulty in selling RCTs to school administrators.

– As a rule, educational decision-makers will insist on use of wait-list control group designs in which:

(1) usual care controls receive the same intervention within one school year.

(2) long-term followup assessments are thus not possible.

– When parents and/or teachers discover they are in a control group condition, participant dropout is much more likely.

– Teachers in control conditions are often motivated to enhance options for target participants to compensate for their not having access to an intervention perceived as beneficial.

School Context• Factors that Influence Educator Adoption of

New Practices:– Fits seamlessly into ongoing school routines– Consistent with school and educator values– Universal versus targeted interventions– Solves a high priority problem or issue– Time and effort costs are reasonable– Teacher perceives s/he has the skills and

resources to apply practice effectively

Single Case vs. Randomized Control Trials

in Demonstrating Efficacy-Effectiveness of Interventions Single Case Methodology•Analysis of level and slope changes between phases

•Target single individuals

•Continuous recording of measures

Randomized Control Trials•Comparison of group averages and variability

•Random assignment to treatment & control groups

•Pre, Post, Follow-up administration of measures

Single Case Methodology

-Allows daily inspection of co-variation of Independent and Dependent Variables

-Visual inspection of intervention effects across phases

-Strong internal validity

Randomized Control Trials

-Allows subgroup analyses of variations in subject group performance

-Allows calculation of effect size estimates of magnitude of treatment effect

-Strong external validity

Example of the Sensitivity and Value

of Using a Single Subject Design in

Understanding Teacher FS Implementation

First Step Fidelity Study

(Rob Horner and Colleagues)

po

Amus

0%

20%

40%

60%

80%

100%

0 5 10 15 20 25

Card Fidelity Problem Behavior

Observation number

Joe

0%

20%

40%

60%

80%

100%

0 5 10 15 20 25

First Step Teacher Phase First Step Teacher Phase Plus Coach Feedback

0%

20%

40%

60%

80%

100%

0 5 10 15 20 25

Edward

Star shows coach present, feedback after observation

Teacher Fidelity of Implementation and Student Problem Behavior during First Step and First Step Plus Coach Feedback

Three-Stage Research and Development Model

• Research Settings – Experimental Classroom– Regular Classroom and Playground Settings– Field Test Settings

• Target Behavior Disorders– Acting Out– Peer Aggression– Social Withdrawal– Academic Survival Skills

Each Completed Behavior Management Package Contains:

(1) Program Materials

(2) Consultant Manual

(3) Teacher Manual and/or

(4) Playground Supervisor Manual

Intervention Delivery Process• Local Program Coordinator—Masters package content and procedures,

recruits behavioral coaches to implement program, maintains liaison-coordinator role.

• Behavioral Coach—Screens and recruits target participants, teachers and

classrooms, sets up intervention program, transfers control to teacher or playground supervisor.

• Classroom Teacher-Playground Supervisor—Assumes control of

program’s operation after behavioral coach transfers it, works with coach to trouble shoot program.

• Target Student—Participates in and cooperates with program during

implementation.• Parents—provide support and home rewards throughout program’s

operation.

RECESS• Reprogramming Environmental Contingencies for Effective Social Skills: An

Early Intervention for Aggressive Young Children

• Age-Grade Range: K-3

• Key Components:

– Social Skills Training

– Praise, Points and Feedback

– Group Rewards at School – Individual Rewards at Home

– Response Cost

– Daily Debriefing and Coaching

• Duration: 3 months

• Target Setting: Playground

Key Lessons Learned for Conducting Successful RCTs in School Settings

For an RCT to be successful in today’s schools, we believe certain conditions should be in place as listed below:

(1) The host school district(s) should be at a point where it is receptive to the innovation being tested and is open to a collaborative partnership in a trial test of it.

(2) Key educational decision makers need to be on board in supporting the study and be able to clearly see its benefits.

Key Lessons Learned for Conducting Successful RCTs in School Settings (continued)

(3) There has to be a local champion or champions who are in a powerful position to support and advocate for the intervention.

(4) The host site and the investigative team need to be willing to negotiate the terms of the RCT so that its strength and integrity are not compromised.

(5) The intervention approach or strategy needs to be well manualized to facilitate training of key staff and to allow a high quality implementation.

Key Lessons Learned for Conducting Successful RCTs in School Settings (continued)

(6) It is essential that staff from the host school districts be hired, whenever possible, in conducting the study so as to motivate district investment in and ownership of the study.

(7) It is of the utmost importance that all RCT study staff display good citizenship and maintain excellent communications and working relationships with host district decision makers and key staff.

Key Lessons Learned for Conducting Successful RCTs in School Settings (continued)

(8) It may become increasingly important to compare the intervention with another active intervention to enable policy-makers, both inside and outside the host district, to make informed decisions about its adoption potential at the conclusion of the study.