Using systematic reviews to inform practice

Using systematic reviews to inform

practice Statewide School-based

OT/PT ConferenceOctober 28, 2005

Steven M. Cope, Sc.D., OT

Evidence-based medicine is the integration of best research evidence with clinical expertise and patient values

Sackett et al. (2000)

An evidence-based occupational therapy practice uses research evidence together with clinical knowledge and reasoning to make decisions about interventions that are effective for a specific client(s)

Law & Baum (1998)

Although there is an undeniable art to pediatric physical therapy, the heart of our practice should be the scientific basis of our interventions. The challenge is to integrate art and science in making clinical decisions that allow us to provide our patients and families with optimal care.

M.J. Barry

Evidence-based practice is…

based on a single patient based on clinical judgement and

patient values (client-centered) guided by research rather than

dictated by it

Being an evidence-based practitioner means…

You value the importance of scientific literature as a foundation for clinical decision making;

You frequently ask yourself, “what evidence exists in the literature to support or refute the intervention I am about to provide?”

You believe your interventions will be more effective when research evidence is integrated

Assumptions of EBP

Scientific research provides an important basis for verifying the effectiveness of our interventions

The effectiveness of interventions are improved under an evidence-based approach

Life-long learning is important to effective clinical practice

Diagnostic and technical skills increase with experience; however, clinical effectiveness deteriorates with time unless current knowledge is used to modify practice patterns

Barriers to EBP

Time for individual study and group discussion (lack of)

Access to scientific literature (lack of) Interpretation of published findings

(difficult to understand) Attitudes towards EBP may be

negative

Evidence-based practice: Getting started

Pose a researchable question Search literature for best evidence Do critical appraisal of study’s

validity Integrate evidence into clinical

decision Evaluate clinical effectiveness

Pose a Researchable Question

Identify clinical problem Example: Children with cerebral palsy have

spasticity and underlying muscle weakness leading to functional movement difficulties

Identify intervention(s)-Strength training

Relative outcomes-strength, spasticity, and functional movement

Patient characteristics-children with spastic CP

The Research Question

Is strength training [intervention] effective at increasing strength and functional movement [relevant outcomes] in children with spastic CP [patient characteristics]?





Search Literature for “Best Evidence”:

What does this mean? Research on subjects whose

characteristics are similar to your patient’s;

Research on interventions that match the one you want to provide;

Research on outcomes that are of interest and apply to both you and the patient;

Research that is credible and believable;





Critical Appraisal

Design (I-V) Sample Size (A-C) Internal Validity (1-3) External Validity (a-c)

DesignLevels of Evidence

I Randomized Controlled (clinical) Trial

II Nonrandomized Controlled Trial

III One Group Pre-Post Test Study

IV Single Subject Designs V Case Reports, Anecdotes,

Expert Opinion

Sample Size

A: n ≥ 20 per group B: n < 20 per group

Internal Validity

1: High internal validity No alternate explanation for outcome

2: Moderate internal validity Attempt to control for lack of

randomization biases 3: Low internal validity

Two or more serious alternative explanations for outcome

Threats to Internal Validity

Hawthorne effect Maturation Testing effect Experimenter bias Co-intervention effects Errors with data No Randomization Attrition

External Validity

a: High external validity Participants represent population, and

treatments represent current practice b: Moderate external validity

Between high and low c: Low external validity

Heterogeneous sample without being able to understand whether effects were similar for all diagnoses or treatments do not represent current practice

Critical Appraisal

Strongest evidence IA1a

Weakest evidence VB3b





Integrate Evidence into Clinical Decision

The evidence reviewed should help you decide: 1) do I start an intervention I’m not currently using; 2) do I stop doing an intervention I am currently doing? 3) continue?

Was the evidence you found best evidence? Does the evidence support or refute the

intervention you want to provide? For which outcomes? Which patients responded well?

How much evidence exists? How much evidence is needed to make this

decision?





Evaluate Clinical Effectiveness

Reflection/assessment after performing the intervention

Did your patient experience a positive change?

Systematic Review: What is it?

A summary of several research articles on one topic presented in table and narrative formats

Qualitative rather than quantitative analysis

Subjective interpretation Time consuming process Strongest form of evidence

Systematic Review #1:Research Question

What is the effectiveness of sensori-motor and motor learning handwriting interventions on improving writing legibility and speed in children with handwriting difficulties?

Search strategies

Key terms (handwriting, writing, written communication, treatment, OT, intervention, …)

Data bases (PubMed…) Entrance criteria (level of evidence,

diagnosis, age, years since publication…)

Critical Appraisal

Two or more people read articles independently and reported relevant information on a form for discussion

Group discussion to develop consensus on key decisions (journal club)

Use AACPDM methodology to present results (table and narrative format)

Results

Sixteen studies located Eight eliminated from review

because they didn’t meet entrance criteria

Eight studies selected for review

Systematic Review of Handwriting Interventions

TABLE 1: Description of Interventions and Participants

Study Intervention Participants LOE n Age 1. Oliver (1990)

Sensorimotor and practice 30 min/wk

Unable to learn skills in typical classroom

IIIB3c 24 Range 5-7 yrs

2. Lockhart et al. (1994)

Sensorimotor and practice 5 hours (10 wks)

Sensorimotor difficulties and writing problems

IVB2a 4 Range 9-11 yrs

3. Peterson et al. (2003)

Sensorimotor Teaching-learning model 10 hours (10 wks)

Enrollment in 1st grade in federally funded school-based health center for economically disadvantaged children

IA2a 59

Mean 7.1 yrs

4. Case-Smith (2002)

Handwriting practice Sensorimotor 9 hours (7 mos)

Received special education and OT services

IIB3a 38 Range 7-10 yrs

5. Sudsawad et al. (2002)

Kinesthetic training Handwriting practice 3 hours (6 days)

Kinesthetic deficit, appropriate attention span Exclusion: on medication to improve attention span

IB1c 45

Range 6-7 yrs Mean 6.11 yrs

6. Denton et al. (2005)

Sensorimotor Therapeutic practice 10 hours (5 wks)

No known physical problems that affect handwriting, no exceptional educational needs

IA2c 38

Range 6-11 yrs

7. Jongmans Study 1 (2003)

Task-oriented intervention Therapeutic practice 9 hours (3 mos)

Referred to a specialist team, not receiving PT or OT, dysgraphic score on BHK

IB1b 7 Mean 7.92 yrs

8. Jongmans Study 2 (2003)

Task-oriented intervention Therapeutic practice 18 hours (6 mos)

Attending special ed school, not receiving PT or OT, dysgraphic score on BHK

1B1b 24 Mean 8.94 yrs

9. Schilling et al. (2003)

Sensorimotor 30 hours (6 wks)

ADHD diagnosis, on medication for attention

IVB1a 3 9 yrs

Table 2TABLE 2: Evidence for Effect of Handwriting Interventions

ICF Dimension

Outcome Statistical results favor tx

Trend favoring tx

Trend oppose tx

No change

Impairment Performance components Kinesthetic acuity Kinesthetic perception and memory

II-W4 II-W4

I-S5 I-S5

Activity Limitation

Writing readiness Quality of handwriting Speed Legibility Space Line Size Form

II-W2, II-W8

I-S3, II-W4 I-S3

I-S3 I-S3

III-W7 I-S6, II-M9

I-S6

IV-W1 II-W2, I-S3, III-W7, II-W8 I-S6

Participation Restriction

Related School Functions Teacher rating Social validity

II-W4 II-M9

II-W4 I-S5

Note. ICF = International Classification of Functioning, Disability, and Health; Roman numeral refers to level of evidence (I = randomized control, V = case study); W, M, S = weak, moderate, strong internal validity respectively; superscripted numbers refer to study (see Table 1); colored items indicate findings that were judged as not clinically important.

Systematic Review Discussion

4 studies used a combination of sensorimotor and motor learning-based intervention; however, in these studies, emphasis was on sensorimotor-based intervention;

1 study used only sensorimotor intervention; 2 studies used only motor learning (practice); 2 studies compared sensorimotor with motor

learning (practice);

Discussion continued

Children in all the studies ranged from 5-11 years of age and were WNL for cognitive function;

All the children receiving intervention were identified as having handwriting difficulties;

Interventions ranged from 3 hours to 30 hours and from 6 days to 7 months. Most sessions were 30 minutes long;

Discussion continued

3 articles were randomized controlled trials and represented strong evidence; the remaining articles were either non-randomized or lacked control groups;

4 of 9 of the studies showed significant improvement for handwriting and an additional 3 demonstrated trends toward improvement;

Studies that looked at quality of handwriting legibility showed that children achieved modest benefits;

Studies that looked at handwriting speed showed no change in this variable;

Long term effects were not studied; Outcomes measured primarily focused on activity

level (handwriting), but all levels were represented;

Clinical Bottom Line

We do know that handwriting is an area that can improve with intervention;

The evidence to date shows some benefit of intervention, however, the studies have not determined what the benefits from these interventions are long-term. Future studies should examine the relative effectiveness of sensorimotor and motor learning approaches and the long-term effects of each intervention

Clinical Bottom Line

This systematic review of handwriting interventions reveals a relatively small number of studies, only a few with rigorous designs;

After reviewing the scientific evidence we are still not sure of the best method for improving handwriting. More studies need to be done that isolate sensorimotor and motor learning interventions to determine if one approach is more effective than the other.

Credits

This review was completed in May 2004 by Rula LaLicata, Stephanie Beilke, Mary Lassanske, Lisa Villardita, Nicole Rosalez, and Steve Cope; it was updated in May 2005 by Cathy Payne and Steve Cope

What do we know about the effectiveness of strengthening exercises for children with spastic cerebral palsy?

Systematic Review #2Research Question

Focused Research Questions

What is the effect of strengthening exercises on force production in children with spastic CP?

Are there adverse effects to strengthening spastic muscles?

What is the effect of strengthening exercises on functional outcome in children with spastic CP?

Method of Review

Literature search was performed using on-line databases: Pubmed, PEDro, CINAHL;

Each study selected for review was read and critically appraised by two people;

Reviewers collaborated on classification of the article in terms of threats to internal validity, level of evidence rating, statistical significance, and clinical importance;

Method of Review cont.

Information and findings from all articles were summarized in table and narrative format according to AACPDM methodology

Results

The literature search yielded 15 articles specifically addressing strengthening exercise and spastic CP;

6 of the 15 articles were eliminated because they did not meet entrance criteria;

9 articles were reviewed

Dimension Outcome Statistical Trend No Effectresults favor tx favoring tx

Impairment Spasticity EEI* Flexibility (ROM) Submax HR Strength: Lower Extremity Upper ExtremityFunctional Limitation Gait Analysis: Velocity Cadence Distance Step length Stride length Knee flex @ heel strike Functional Tests: L LSUT* R LSUT* MAS – STS* score Min. chair height (cm) 10-m walk test (s) 2-min walk test (m) 50-m dash (s) 12-min test (m) Timed stair test (s) GMFM* Total Dimension 5 Dimension D Dimension E Societal Limitation Perceived competence SPPA* SPPC*

III1-S III3-W III6 -S III5-W, III9-S III9-S III9-S

III2-S, III4-M, III5-W I8-S, III4-M I8-S III6-S, III9-S, I8-SIII7-S III9-S

III3-W, III6-S III4-M III5-W, I8-SIII3-W, III6-S III4 -MIII3-WIII3-WIII4-M III6-SIII2-S

III4-MIII4-MIII4-MIII4-MIII4-M III4-M III7-SIII7-S I8-S

III5-W I8-S III6 -SIII6 -S I8-S I8-S

III9-S III9-S

Summary of Studies

Abbreviations for Table EEI – Energy Expenditure Index LSUT – Lateral Step-Up Test MAS-STS – Motor Assessment Scale – Sit to Stand GMFM – Gross Motor Function Measure : SPPA – Self-Perception Profile for Adolescent

1 Fowler, et.al. (2001) 2 Damiano, et.al. (1995) 3 Eagleton, et.al. (2004) 4 Blundell, et.al. (2003) 5 MacPhail & Kramer, (1995) 6 Damiano & Abel, (1998) Colored = no clinical

importance 7 O’Connell & Barnhart, (1995) Italics = small clinical importance 8 Dodd, et.al. (2003) Regular font = clinically important 9 Darrah, et.al. (1999)

Abbreviations cont.

Level of Evidence

I = Randomized controlled trial

III = One group pre-post test

Ratings of Internal Validity

S = Strong, M = Moderate, W = Weak

What is the effect of strengthening exercises on force production in children

with spastic cerebral palsy?

Evidence from the studies reviewed showed that for the majority of studies, children, and muscles studied, children with CP did improve their strength, primarily in their lower extremities

Are there adverse effects to strengthening spastic muscles?

Only one study specifically examined the effects of muscle strengthening on spasticity, and it provided evidence that spasticity was not increased

What is the effect of strengthening exercises on functional outcome in

children with spastic cerebral palsy?

There is evidence to suggest that function improves after a strengthening program

Function was operationally defined in most studies as a performance on the GMFM and various parameters derived from gait analysis

Additional outcome measures

Other impairment level outcomes were measured such as ROM, Heart Rate (HR), and energy expenditure

Results indicate that strengthening did have a positive effect on energy expenditure but no effect on ROM and HR

Suggestions for Future Research

A study design with other types of CP other than spastic (i.e. dystonic and athetoid);

It remains unknown if strengthening programs can be used (and if they are safe) for very young children and toddlers with spastic CP;

The best type of strengthening exercise and the most effective had high intensity and duration;

The effects on upper extremities are not known; The long-term effects of a strengthening

program on strength and function are not known

Clinical Bottom Line This review revealed a reasonable number of

moderate quality studies; There is clear evidence that strengthening programs

are effective for increasing strength in children with spastic CP;

There is also evidence to suggest that kids not only get stronger, but also have an increase in functional skills;

There has only been one randomized study done, and there are still many unanswered questions which suggests that this area of research is in its early phases;

More research with rigorous designs and larger sample sizes is needed

Credits

This systematic review was completed in May, 2005 by Christina Keller, DPT and Steve Cope, Sc.D., OT

Systematic Review #3

How effective are sensory-based interventions that are used in school settings for improving time on task in children with ADHD?

Purpose

The purpose of this project was to systematically review the scientific literature investigating the effectiveness of treatments used in the classroom for improving attention and behavior in children with ADHD and PDD (including autism).

Research Question: What is the effectiveness of interventions using sensory-based approaches in improving attention span and behavior in children with ADHD or PDD in the classroom setting?

Inclusion Criteria Population studied was ADHD or

PDD Interventions that affect classroom

function and likely to be provided by an occupational therapist

Level III of evidence or higher Peer-review journal only 1992-present

Procedures Terms used for research: ADHD,

treatment, OT treatment, time on task, school-based therapy, therapy, rehabilitation, weighted vests, OT, PT, PDD, autism, sensory integration, sensory modulation, attention and time;

Data bases used: Ebscohost, OT Search, OCLC First Search, Proquest-Cinahl, OT Seeker, PsychInfo, ERIC, and PubMed;

Articles were read individually and then reviewed and discussed by all group members to identify internal and external validity, level of evidence, and clinical and statistical significance;

Procedures cont.

Information and findings from each article were summarized in table and narrative format according to the AACPDM methodology.

Results

8 articles addressing interventions that affect classroom function in children with ADHD or PDD from 1992 to present were located:- 2 randomized control trials (level I)- 1 nonrandomized control trial (level III)- 4 single subject design (level IV)- 1 case study (level V)

Treatments Therapy Balls as seats in

classroom Interactive Metronome/Video

Game Weighted Vests Hug Machine Chewy Sensory Integration

Clinical Populations

Diagnoses - 63 ADHD - 7 PDD - 17 Autism - 3 Autistic-like

tendencies

Age Range 2 to 13 years

Outcomes Measured Behavior Issues Social Skills Attention Sensory Motor Academic & Cognitive Skills Hyperactivity Peer/Adult Interaction Mastery of Play Approach to New Activities Touch Non-engaged Behavior

Table 1 Study Characteristics

Study Level of Evidence

Treatment Population N Age

Schilling et al (2003)

IVB2b Therapy balls

ADHD 3 (2 M, 1 F) 9.8– 9.11 yrs.

Shaffer et al (2001)

IA3b Interactive Metronome, Video Game

ADHD 56 M 6.0-12.5 yrs.

VandenBerg (2001)

IIIB2c Weighted Vests

ADHD 4 (2 M, 2 F) 5.9-6.10 yrs.

Edelson et al. (1999)

IB2b Hug Machine

Autism 12 (9 M, 3 F) 4.0-13.0 yrs.(M=7.58yrs.)

Fertel-Daly (2001)

IVB2b Weighted Vests

PDD 5 (3 M, 2 F) 2.0-4.0 yrs.

Case-Smith & Bryan

(1999)

IVB2b OT w/ SI Autism 5 (5 M) 4.0–5.3 yrs.

Scheerer (1992)

VB3c Chewy Autistic Like Tendencies

3 M 5.0-10.0 yrs.

Linderman & Stewart

(1999)

IVB2c SI PDD 2 M 3.9 & 3.3 yrs.

Results Table IStudy Outcome Measure Dimensio

nResult

sClinical

ImportanceStat.Valid

LOE

Schilling et al

(2003)

Behavior ISB P + No NR IVB2b

Classroom Performance

LWP P +/-(1), + (2)

No NR

VandenBerg (2001)

Attention Time on Task (min)

AL + Yes P<.05 IIIB2c

Edelson et al. (1999)

Hyperactivity RHS (CPRS)

P + No WG P<.01 IB3b

+/- No BG P>.05

Fertel-Daly

(2001)

Attention Focused Attention

(sec)

AL +(5) No NR IVB2b

Distractions

(number)

AL +(4), +/-(1)

No NR

Duration and Type

of Self-Stimulator

y Behaviors

AL + No NR

Results Table IIDimension Outcome Results favor tx Trend favoring

txResults not favoring tx

No effect

Participation Restrictions

Behavior IIIg IVa, Vc, IVd , IIIg

Hyperactivity Ie Ie

Social IVd,

Approach to new activities

IVd,

Touch IVd,

Mastery of Play IIIg IIIg

Non-Engaged Behaviors

IIIg

Activity Limitations

Academic & Cognitive Skills

Ib Ib

Attention IIIf IVh, Ib Ib IVh

Adult Interactions

IIIg IIIg

Peer Interactions IIIg

Impairment Sensory Motor Ib IVd, Ib, Ib Ib

A= Schilling et al (2003), B = Shaffer et al (2001), C = Scheerer (1992), D = Linderman & Stewart (1999), E = Edelson et al. (1999), F = VandenBerg (2001), G = Case-Smith & Bryan (1999), H = Fertel-Daly (2001)

Bold=not clinically important

Conclusion There are two well design research studies

and a limited number of articles found on this topic;

Some findings were positive although many were judges as having limited clinical importance;

Statistical significance was difficult to determine due to lack of recorded data;

Majority of outcomes were categorized as participation restrictions with some activity limitations.

Conclusion The interventions varied greatly in

terms of type, intensity, and duration but all focused on improving the attention through the sensory systems;

While these types of treatments are commonly used, there is limited evidence that supports their effectiveness of improving classroom functioning;

Long term effects of treatment were not reported.

Clinical Bottom Line There is a small number of clinical

trials investigating the effectiveness of sensory-based interventions for attention; many are poorly designed with inconclusive results and questionable clinical importance;

More well designed studies are needed to confirm the effectiveness of treatments used to improve classroom functioning in children with ADHD or PDD.

Credits

This review was completed in May of 2004 by Juliet Bartels, Michelle DoBas, Tammy Juern, and Steve Cope

Using systematic reviews to inform practice

Documents

clinical knowledge

clinical decisions

clinical expertise

clinical judgement

barryevidencebased practice

clinical decision makingyou

practice patternsbarriers

otevidencebased medicine