Roles of Common Factors & “Therapist “Effects” in Therapy Outcomes Session #0931 G.S. (Jeb) Brown, Ph.D. Center for Clinical Informatics.

Roles of Common Factors &

“Therapist “Effects” in

Therapy Outcomes

Session #0931

G.S. (Jeb) Brown, Ph.D.

Center for Clinical Informatics

Common factors

• The effectiveness of all treatments is due, in some part, to factors common to all treatments.

• Contact with a helping, caring professional fosters hope and expectancy.

• We have come to accept the potency of “placebo effects”, and insist that the effectiveness bona fide treatments exceeds that of placebo treatments

• So far, so good. Who can argue with this?

Randomized double-blind placebo controlled drug trials

• Double blind placebo controlled drug studies provide an exemplar for estimating the role of common factors.

• Traditionally, the drug is interpreted as the difference between placebo and the active drug.

• Meta-analysis of multiple studies of antidepressants lets us estimate the relative importance of common factors (placebo effects) versus drug effects.

Meta-analyses and placebo

• Meta-analysis involves the use of a statistical techniques to combine results from multiple studies in order in an effort to generalize findings.

• Meta-analysis of multiple studies of antidepressants let us estimate the relative importance of common factors (placebo effects) versus drug effects. 1-3

Antidepressant effect(25%)

Placebo effect

Natural course of theillness

Drug effect accounted for 25% of measured improvement

Evidenced based psychotherapy

• For several decades psychotherapy researchers have attempted to design randomly controlled trails (RCT) to investigate the effectiveness of specific methods of psychotherapy.

• Study design analogous to pharmacy trials, except that designing credible “placebo treatments” is much more problematic.

• Various treatment methods are being touted as “evidenced based” by citing the number of RCTs providing evidence that the treatment exceeded placebo (or some other treatment).

Psychotherapy “brands”

• The advocacy for the use of specific therapies is analogous to the advertising of brands of antidepressant medication.

• Calls for wide spread use of “evidence based treatments” in psychotherapy is analogous to the FDA’s insistence that a drug may not be marketed for the treatment of depression until at least two studies have shown superiority to placebo.

• Advocates and practitioners of various “evidence based treatments” have a vested interest in discouraging the use of “unproven” treatments.

Brand differentiation

• Advocates of psychotherapy brands insist on the uniqueness of their therapy and the need to adher to specific treatment procedures

• Research methodology requires the use of manuals and other techniques to standardize treatments

• Treatment effectiveness presumed to be dependent on the correct application of the “active ingredients” in the psychotherapy method.

The Dodo Bird Effect

Rosenzweig S. (1936) Some implicit common factors in diverse methods of psychotherapy: “At last the Dodo said, ‘Everybody has won and all must have prizes.’”

Am J Orthopsychiatry 6:412-5.

The Dodo Bird Lives!

Wampold BE, Mondin GW, Moody M, et al. (1997). A meta-analysis of outcome studies comparing bona fide psychotherapies: Empirically, “All must have prizes.” Psychol Bull 122:203-15.

Luborsky, L., Rosenthal, R., Diguer, L., et al. 2002 The dodo bird verdict is alive and well--mostly. J. Psychotherapy Integration Vol 12(1) 32-57

Meta-analysis & common factors

• Over two decades of meta-analytic studies have served to reinforce Rosenzweig’s 1936 observation that different methods of psychotherapy tend to produce comparable outcomes… the “Dodo Bird Effect”

• Lack of evidence for specific treatment effects bolster the argument that almost all of the effects of psychotherapy are due to factors common to all psychotherapies. 5-11

Real world example

• Human Affairs International (HAI) collected outcome data from a large number of clinicians between 1996 and 1998.

• Clinicians were asked to specify the primary method of psychotherapy (or medication management only)

• Analyses revealed no significant differences in the outcome or mean number of sessions across all treatment methods, including medication management.

Treatment & outcomeHAI data

-0.20

-0.10

0.00

0.10

0.20

0.30

0.40

InsightOriented(n=135)

SolutionFocused(n=647)

Behavioral(n=67)

CognitiveBehavioral(n=1226)

Family (n=152) MedicationManagement

(n=514)

Other (n=129) Unknown(n=85)R

esi

du

al E

ffect

Siz

e

Mean Value of Residual Change after Controlling for Severity(positive values = greater than average change; 90% confidence interval)

Fidelity and practicality

• Propagation of evidenced based treatment methods requires some method of measuring fidelity to the treatment.

• BIG PROBLEM!

• If clinicians reports using Cognitive Behavioral Therapy, how can we have any confidence that what that clinician did with a specific patient was comparable to the “Cognitive Behavioral Therapy” in the RCTs?

Goldilocks Effect

• Clinicians tend to be eclectic and flexible in their choice of treatment methods.

• Clinicians and patients tend to try different treatments until they find something that is “just about right” for them.

• Patients and clinicians tend to adjust the number and frequency of sessions depending on the patient’s level of distress and rate of improvement.

• Result: All treatments appear to have similar outcomes and patients with good outcomes tend to use fewer session than patients with poor outcomes.

Recommended reading

Rigorous review and analysis of controlled studies on psychotherapy outcome.

Conclusion: much more variance resides with the clinician than with the treatments.

Therapists effects

• Wampold and others argue that researchers have ignored the individual therapist as a source of variance.11, 16-24

• The person of the therapist is necessary to delivery the treatment, and personal characteristics of the therapist modify the effect of the treatment.

• Factors contributing to therapists effects may include elements clinical skill and knowledge as well as personality traits.

RCT and ANOVA – brief history

• Some of the earliest applications of randomized control group design and analysis of variance were in agriculture and education. 12,13

• RCT methodology later adopted by medicine and eventually psychotherapy research. 11,14

• Simple ANOVA is appropriate only if the individual farmer, teacher or clinician has little or no impact on the effectiveness of the farming, teaching or treatment method!

HLM & therapist effects

• Hierarchical Linear Modeling (HLM) is an advance in statistical methodology that permits us to model variance at the clinicians level and as well as the treatment level.

• An rapidly growing body of published research points to the conclusion that therapist effects almost certainly exceed specific treatment effects by a large margin.

Variance due to the clinician

• Published research making use of HLM points to the conclusion that the clinician accounts for much more of the variance in psychotherapy outcomes that treatment method per se. 11, 17-21

• Analyses of PacifiCare Behavioral Health’s massive database database on patient outcomes confirms significant variance in psychotherapy outcomes at the clinician level. 24,25

PacifiCare Behavioral Health ALERT System

• Initiated an outcomes management program in 1998 using 30 item patient self report questionnaires administered at regular intervals in treatment.

• ALERT System used to capture data and monitor patient outcomes in real time.

• Currently over 7,000 clinicians are contributing outcome data on a regular basis.

• Probably largest database on mental health outcomes in the world.

PBH research collaboration

• PBH actively sought the involvement of leading psychotherapy outcomes researchers from leading academic institutions.

• External researchers actively involved in design of the measurement system and ongoing analysis of the data.

• PBH encouraged publication of findings in academic journals.

The (almost) Bell CurvePBH data

Solo clinicians with sample sizes => 20

0%

5%

10%

15%

20%

25%

-0.2 -0.1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

Effect Size

% o

f c

linic

ian

s

Where is the variance?

Unexplained 74% (externalstressors, health condition, socialsupports, etc.)

Acuity (intake score) 18%

Diagnosis, age & sex 2%

Clinician 6%

Treatment method 1%

Clinicians & treatments

% of variance due to therapists in the real world

• Analysis of PacifiCare Behavioral Health (PBH) data reveals 6% of variance due to therapist. 25

• Patients on medication have a higher % of variance due the therapist than those receiving psychotherapy alone.

• Huh??

Therapists and meds

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

meds

nomeds

Outcomes (residualized scores) of 15 therapists for patients with concurrent medication or no medication 25

Test scores and medicationPBH data

Percentage of cases on medication as a function of intake score

0%10%20%30%40%50%60%70%80%

0-20

21-3

0

31-4

0

41-5

0

51-6

0

61-7

0

71-8

0

81-9

0

90-1

20

Which treatment is best? Goldilocks Effect: Clients tend to get the treatment that is just about right for them.

Treatment, severity and outcome

-0.50

0.00

0.50

1.00

1.50

2.00

0-20 21-30

31-40

41-50

51-60

61-70

71-80

81-90

90-120

Intake score

Effe

ct s

ize

Psychotherapyonly

Psychotherapyand medication

Normal functioning Severe symptoms

Cross validation analysis

• Psychotherapists in PBH network ranked based on all cases from 1999-2002 if sample size =>30; N=116.

• If a therapist’s mean residualized final score < 0 then clinician rated “Highly effective”; else clinician rated “Less effective”.

• Outcomes evaluated in the 2003-2004 cross validation period for a new sample of cases.

Cross validation resultsClinician cross validation results - 2003 to 2004

Therapists assessed on at least 30 cases between 1999-2002. Highly effective clinicians had mean Change Index Score > 0

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0-50 51-120

Intake scores (mean split)

Effe

ct s

ize

Highly effective clinicians-psychotherapy only

Highly effective clinicians-psychotherapy and medication

Less effective clinicians-psychotherapy only

Less effective clinicians-psychotherapy and medication

Risk of not using HLM

• Wampold and colleagues at the University of Wisconsin recently reanalyzed data from the National Institute of Mental Health’s Treatment of Depression Collaborative Research Program (TDCRP) study using HLM. 26-28

• Prior published reports found significant differences between two methods of psychotherapy as well as between placebo and antidepressant medication.

• Reanalysis of psychotherapy data using HLM revealed that 0% of the variance was due to the psychotherapy methods, while 8% was attributable to the therapists. 27

Psychiatrist effects

• Wampold and colleagues also used HLM to reanalyze the results antidepressant and placebo legs of the TDCRP study. 28

• Included the 9 individual psychiatrists as a variable.

• Outcome measured by change on patient self report measure (Beck Depression Inventory).

• 9.1% of the variance due to the psychiatrist; only 3.4% due to the medication.

• Top 3 psychiatrists had a better outcome with placebo than bottom 3 had with the antidepressant.

Placebo & therapist effects

• Hypothesis: Placebo/common factor effects are mediated by the clinician/patient relationship.

• Common factors tend to account for much more of the variance than specific treatment effects.

• If the effects of common factors are medicated by the clinician/patient relationship, then we would naturally find much of the variance in outcomes would be due to the clinician.

• The human factor matters!

• DUH!

What’s a clinician to do?

• If a wide variety of treatments appear to be equally efficacious, can a therapist do to achieve the best outcomes possible for their patients?

• A growing body of research supports the use of repeated administrations of patient self report outcome questionnaires to monitor response to treatment. 29-36

• Routine measurement and and early identification of patients with a poor response to treatment has been shown to reduce treatment failures.

Therapeutic alliance

• A large body of evidence suggests that the relationship and working alliance between the clinicians and patient is an important factor in the outcome. 39-45

• Routine use of a session rating/therapeutic alliance scale may permit clinicians to identify and repair problems in the working alliance.

Outcomes informed care

• “Meta-method” designed to improve outcomes across all patients and diagnoses, regardless of treatment method.

• Routine use of patient self report questionnaires to track symptom severity and therapeutic alliance.

• Use of feedback mechanisms to alert clinicians to patients at risk for poor outcomes.

• Performance feedback to clinicians, including comparison to outcomes to those of clinicians treating similar patients.

• Preferential referrals to highly effective clinicians.

2 case studies

• Resources for Living (RFL) provides telephonic EAP services, data collected over the phone at time of service; clinicians receive real time feed back on trajectory of improvement and working alliance (SIGNAL system)

• Accountable Behavioral Healthcare Alliance (ABHA) is a managed behavioral healthcare organization servicing Oregon Health Plan members in 5 rural county area

Case history # 1: RFL

• Began using the 4 item Outcome Rating Scale and Session Rating Scale in 2002

• Administered telephonically as part of telephonic counseling sessions.

• Baseline data collected for 5 months

• Baseline data used to create trajectory of change graphs

• Real time feedback provided to counselors via SIGNAL System

0.00

0.10

0.20

0.30

0.400.50

0.60

0.70

0.80

0.90

2nd

quar

ter 2

002

3rd

quar

ter 2

002

4th

quar

ter 2

002

1st q

uarte

r 200

3

2nd

quar

ter 2

003

3rd

quar

ter 2

003

4th

quar

ter 2

003

1st q

uarte

r 200

4

2nd

quar

ter 2

004

3rd

quar

ter 2

004

Eff

ec

t s

ize

RFL Signal System: results

Baseline period

Training and feedback

Case history # 2: ABHA

• Began utilizing the 4 item Oregon Change Index (OCI) in 2004.

• OCI administered at every session in outpatient and day treatment settings.

• OCIs collected at over 80% of all sessions.

• Collected baseline data for 18 months, began giving feedback in mid 2005.

• Updated Excel based Active Case Report contains outcome data on all cases seen within the last 6 weeks is emailed to the clinicians weekly.

ABHA results

ABHA Effect Size by QuarterClients with scores in clinical range at intake

-0.05

0.05

0.15

0.25

0.35

0.45

0.55

0.65

0.75

0.85

1st 04 2nd 04 3rd 04 4th 04 1st 05 2nd 05 3rd 05 4th 05 1st 06

Effe

ct S

Ize

Benchmark from clinical trials: .83

Implications for clinicians

• Good news: The clinician matters!!!!!!

• All treatments (including medications!?) are only as effective as the clinicians delivering the treatment.

• Clinicians have an ethical responsibility to assess and improve their personal effectiveness as clinicians… they cannot rely on the treatments alone to be curative.

• Effective clinicians deliver high value services and are worth more money!!!

Implications for administrators & policy makers• Exclusive focus on the effectiveness of treatments

rather than the value of the clinicians limits the potential to improve outcomes.

• Use of effective clinicians tends to lower costs.

• Administrators and policy makers have an obligation to consumers to assure that they have access to effective clinicians.

• Failure to monitor outcomes at the clinician level places consumers at risk.

References

1. Kirsch, I & Sapirstein, G. 1998. Listening to Prozac but hearing placebo: A meta analysis of antidepressant medication. Prevention & Treatment. 1, Article 0002a, No Pagination Specified

2. Kirsch, I. 2000. Are drug and placebo effects in depression additive? Biological Psychiatry 47, 733-73.

3. Kirsch, I, Moore, TJ, Scoboria, A, Nicholls, SS. 2002. The emperor's new drugs: An analysis of antidepressant medication data submitted to the U.S. Food and Drug Administration. Prevention & Treatment. 5(1), No Pagination Specified

4. Rosenzweig S. 1936. Some implicit common factors in diverse methods of psychotherapy: “At last the Dodo said, ‘Everybody has won and all must have prizes.’” Am J Orthopsychiatry 6:412-5.

5. Shapiro DA & Shapiro D. 1982. Meta-analysis of comparative therapy outcome studies: A replication and refinement. Psychol Bull 92:581-604.

References (continued)

6. Robinson LA, Berman JS, Neimeyer RA. 1990. Psychotherapy for treatment of depression: A comprehensive review of controlled outcome research. Psychol Bull 108:30-49.

7. Wampold BE, Mondin GW, Moody M, et al. 1997. A meta-analysis of outcome studies comparing bona fide psychotherapies: Empirically, “All must have prizes.” Psychol Bull 122:203-15.

8. Ahn H, Wampold BE. 2001. Where oh where are the specific ingredients? A meta-analysis of component studies in counseling and psychotherapy. J Counsel Psychol 48:251-7.

9. Chambless DL, Ollendick TH. 2001. Empirically supported psychological interventions: Controversies and evidence. Annual Rev Psychol 52:685-716.

10. Luborsky, L., Rosenthal, R., Diguer, L., et al. 2002. The dodo bird verdict is alive and well--mostly. J. Psychotherapy Integration Vol 12(1) 32-57

References (continued)

11. Wampold BE. 2001. The great psychotherapy debate: Models, Methods, and Findings. Mahwah NJ: Lawrence Erlbaum Associates. Wampold BE, Mondin GW, Moody M, et al. 1997. A meta-analysis of outcome studies comparing bona fide psychotherapies: Empirically, “All must have prizes.” Psychol Bull 122:203-15.

12. McCall, WA. 1923 How to experiment in education. New York: McmIllan.

13. Fisher, RA. 1935 The design of experiments. Edinburgh: Oliver and Boyd.

14. Gehan, E. & Lemark, NA. 1994. Statistics in medical research: Developments in clinical trials. New York: Plenum Press.

15. Martindale C. 1978. The therapist-as-fixed-effect fallacy in psychotherapy research. J Consult Clin Psychol 46:1526-30.

References (continued)

16. Luborsky L, Crits-Christoph P, McLellan T, et al. 1986. Do therapists vary much in their success? Findings from four outcome studies. Am J Orthopsychiatry 56:501-12.

17. Crits-Christoph P, Baranackie K, Kurcias JS, et al. 1991. Meta-analysis of therapist effects in psychotherapy outcome studies. Psychother Res 1:81-91.

18. Crits-Christoph P, Mintz J. 1991. Implications of therapist effects for the design and analysis of comparative studies of psychotherapies. J Consul Clin Psychol 59:20-6.

19. Wampold BE. 1997. Methodological problems in identifying efficacious psychotherapies. Psychother Res 7:21-43,

20. Elkin I. 1999. A major dilemma in psychotherapy outcome research: Disentangling therapists from therapies. Clin Psychol Sci Prac 6:10- 32.

References (continued)

21. Wampold BE, Serlin RC. 2000. The consequences of ignoring a nested factor on measures of effect size in analysis of variance designs. Psychol Methods 4:425-33.

22. Huppert JD, Bufka LF, Barlow DH, et al. 2001. Therapists, therapist variables, and cognitive-behavioral therapy outcomes in a multicenter trial for panic disorder. J Consul Clin Psychol 69:747-55.

23. Okiishi J, Lambert MJ, Nielsen SL, et al. 2003. Waiting for supershrink: An empirical analysis of therapist effects. Clin Psychol Psychother 10:361-73.

24. Brown GS, Jones ER, Lambert MJ, et al. 2005. Identifying highly effective psychotherapists in a managed care environment. Am J Managed Care 11(8):513-20.

25. Wampold BE, Brown GS. 2005. Estimating variability in outcomes due to the therapist: A naturalistic study of outcomes in managed care. J Consul Clin Psychol. 73(5): 914-923.

References (continued)

26. Elkin, I, Shae, T, Watkins, JT., et al. 1989. National Institute of Mental Health Treatment of Depression Collaborative Research Program: General effectiveness of treatments. Archive of General Psychiatry. 46: 971-982.

27. Kim DM, Wampold BE, Bolt DM. 2006. Therapist effects and treatment effects in psychotherapy: Analysis of the National Institute of Mental Health Treatment of Depression Collaborative Research Program. Psychother Res. 16(2): 161-172.

28. McKay, KM, Imel, ZE & Wampold, BE. In press. Psychiatrist effects in the pharmacological treatment of depression. J. Affective Disorders.

29. Hannan C, Lambert MJ, Harmon C et al. 2005. A lab test and algorithms for identifying clients at risk for treatment failure. J Clin Psychol 61(2):155-63.

30. Lambert MJ, Harmon C, Slade K et al. 2005. Providing feedback to psychotherapists on their patients progress: Clinical results and practice suggestions J Clin Psychol 61(2):165-74.

References (continued)

31. Harmon C, Hawkins, Lambert MJ et al. 2005. Improving outcomes for poorly responding clients: The use of clinical support tools and feedback to clients. J Clin Psychol 61(2):175-85.

32. Brown GS, Jones DR. 2005. Implementation of a feedback system in a managed care environment: What are patients teaching us? J Clin Psychol 61(2):187-98.

33. Claiborn CD, Goodyear EK. 2005. Feedback in psychotherapy. J Clin Psychol 61(2):209-21.

34. Lueger RJ. 1998. Using feedback on patient progress to predict the outcome of psychotherapy. J Clin Psychol 54:383-93.

35. Lambert MJ, Whipple JL, Smart DW, et al. 2001. The effects of providing therapists with feedback on patient progress during psychotherapy: Are outcomes enhanced? Psychother Res 11(1):49-68.

References (continued)

36. Lambert MJ, Whipple JL, Vermeersch DA, et al. 2002. Enhancing psychotherapy outcomes via providing feedback on client progress: A replication. Clin Psychol Psychother 9:91-103.

37. Whipple JL, Lambert MJ, Vermeersch DA, et al. 2003. Improving the effects of psychotherapy: The use of early identification of treatment failure and problem-solving strategies in routine practice. J Counsel Psychol 50(1):59-68.

38. Lambert MJ, Whipple JL, Hawkins EJ, et al. 2003. Is it time for clinicians to routinely track patient outcome? A meta-analysis. Clin Psychol Sci Prac 10:288-301.

39. Bachelor, A., & Horvath, A. (1999). The therapeutic relationship. In M.A. Hubble, B.L. Duncan, and S.D. Miller (eds.). The Heart and Soul of Change: What Works in Therapy. Washington, D.C.: APA Press, 133-178.

40. Blatt, S. J., Zuroff, D.C., Quinlan, D.M., & Pilkonis, P. (1996). Interpersonal factors in brief treatment of depression: Further analyses of the NIMH Treatment of Depression Collaborative Research Program. J Consul Clin Psychol. 64, 162-171.

References (continued)41. Bordin, E. S. (1979). The generalizability of the psychoanalytic

concept of the working alliance. Psychotherapy: Theory, Research, and Practice, 16, 252-260.

42. Burns, D., & Nolen-Hoeksema, S.  (1992). Therapeutic empathy and recovery from depression in cognitive-behavioral therapy: A structural equation model. J Consul Clin Psychol. 60, 441-449.

43. Connors, GJ, DiClemente, CC., Carroll, KM, et al. 1997 The therapeutic alliance and its relationship to alcoholism treatment participation and outcome. J Consul Clin Psychol, 65(4), 588-598.

44. Horvath, A. O., & Symonds, B. D. (1991). Relation between working alliance and outcome in psychotherapy: A meta-analysis. J Consul Clin Psychol. 38, 139-149.

45. Krupnick, J., Sotsky, SM, Simmens, S et al. 1996. The role of the therapeutic alliance in psychotherapy and pharmacotherapy outcome: Findings in the National Institute of Mental Health Treatment of Depression Collaborative Research Project. J Consul Clin Psychol. , 64, 532-539.

About the presenter

G.S. (Jeb) Brown is a licensed psychologist with a Ph.D. from Duke University. He served as the Executive Director of the Center for Family Development from 1982 to 19987. He then joined United Behavioral Systems (an United Health Care subsidiary) as the Executive Director for of Utah, a position he held for almost six years. In 1993 he accepted a position as the Corporate Clinical Director for Human Affairs International (HAI), at that time one of the largest managed behavioral healthcare companies in the country.

In 1998 he left HAI to found the Center for Clinical Informatics, a consulting firm specializing in helping large organizations implement outcomes management systems. Client organizations include PacifiCare Behavioral Health/ United Behavioral Health, Department of Mental Health for the District of Columbia, Accountable Behavioral Health Care Alliance, Resources for Living and assorted treatment programs and centers throughout the world.

Dr. Brown continues to work as a part time psychotherapist at behavioral health clinic in Salt Lake City, Utah. He does measure his outcomes.

http://[email protected]

1821 Meadowmoor Rd.Salt Lake City, UT 84117

Voice 801-541-9720