Assessment of Reliable Change: Methods and Assumptions Michael Basso, Ph.D. Associate Professor and Director of Clinical Training Department of Psychology—University.

Assessment of Reliable Change: Methods and Assumptions

Michael Basso, Ph.D.Associate Professor and Director of Clinical Training

Department of Psychology—University of Tulsa

Clinical Associate Professor

Department of Psychiatry—University of Oklahoma

Objectives• Provide background concerning methods

of assessing reliable change• Describe assumptions and applications of

reliable change scores• Illustrate use of reliable change scores

Assessment of Clinical Change

• Two Basic Approaches– Assessment of Group Differences Across Time– Assessment of Individual Differences Across Time

Assessment of Group Differences Across Time

• Assessment of statistically reliable change– “Does the treatment yield significant benefits for groups of

patients?”

» i.e., do average scores at T1 and T2 come from different distributions

– This approach describes the average rate of change over groups primarily

– It is accomplished with repeated measure ANOVA– Problem: You could have a statistically significant difference

with a very small effect size, but it might not be a clinically meaningful change

• Assessment of Clinically Meaningful Change– “Did the patient’s change in performance at T2 exceed base

rates of change?”

» i.e., did the individual show change that exceeded expectations based on measurement error, practice effects, and regression to the mean?

– This method describes the base rate of change

» Change that is exceeds the base rate is not normal, and is therefore clinically meaningful

– Our focus is on the assessment of clinically meaningful change in individuals, but this method can be applied to group data as well

Assessment of Individual Differences Across Time

• How do you establish the base-rate of change?

• Bear into consideration that:– It would be improbable to obtain the exact same score twice

– There is no perfect test-retest correspondence because of

» measurement error

» regression to the mean

» practice effects

Assessment of Clinical Change for Individuals

• Reliable change Index scores– “Does change exceed what would be expected based on measurement

error alone?”– This method is based on Reliability of measurement– It is used for typical performance tests

» i.e., attitude, personality, psychopathology, etc.

• Standardized Regression-Based Change Scores– “Does change in scores exceed expectations based on T1 (baseline)

scores?”– This method is based on a validity coefficient (i.e., what T2 score is

predicted by the T1 score)– It is used for maximal performance tests

» i.e., IQ, neuropsychological, etc.

Two Methods of Assessing Base Rates of Change

• Elaborated by Jacobson and Truax (1991)– Based on the standard error of the difference

» Which in turn is based on the reliability coefficient– This reflects the sampling distribution of difference scores– it implies the magnitude of differences between two test scores that

vary by chance alone

• Assumptions– Error components are mutually independent and independent of true

pretest and posttest scores– Error is normally distributed with a mean of 0– SE of error is equal for all subjects– These assumptions are questionable in clinical settings (cf. Maassen,

2004)

Reliable Change Index Scores

Standard Normal Curve—Distribution of Difference Scores

-4 -3 -2 -1 0 1 2 3 4

34% 34%

14% 14%2% 2%

50th 84th 98th16th2nd

• To use the RCI, you must compute the SE of difference between two scores

• SEdiff=(2(SD(1-rxx)1/2)2)1/2

• Then, compute a confidence interval for change scores• for 95% confidence, you multiply 1.96 * SEdiff• for 90% confidence, you multiply 1.60 * Sediff• Does the raw score change between T2 and T1 exceed the

confidence interval?– If so, it represents change that exceeds the base rate expected based on

measurement error» Thus, clinically meaningful change has occurred

– If not, then the change is consistent with the base rate expected based on measurement error» Thus, no clinically meaningful change has occurred

Reliable Change Index Scores--Method

• Ferguson, Robinson, & Splaine (2002)– SF-36 in 200 patients who had undergone a Coronary Artery

Bypass Grafting (CABG) surgery– SF-36 contains 8 scales

» Physical Functioning» Role Functioning Physical» Bodily Pain» General Health» Vitality» Social Functioning» Role Functioning-Emotional» Mental Health

Reliable Change Index Scores—An Example

• Ferguson, Robinson, & Splaine (2002)– Physical Functioning

» Reliability=.93 (from normative sample of 2474)» Mean of normative sample=84.15» SD of normative sample=23.28

» SEdiff=(2(SD(1-rxx)1/2)2)1/2

• SEdiff=(2(23.28(1-.93)1/2 ) 2) )1/2=9.85

• 95% CI: (SEdiff)*1.96=19.32

• T1 Mean=40.97

• T2 Mean=63.39

• Mean Diff=22.42

• The mean difference exceeds 19.32

• Thus, clinically meaningful change has occurred as a result of surgery

Reliable Change Index Scores—An Example

• Ferguson, Robinson, & Splaine (2002)– Mental Health

» Reliability=..84 (from normative sample of 2474)» Mean of normative sample=75.01» SD of normative sample=21.40

» SEdiff=(2(SD(1-rxx)1/2)2)1/2

• SEdiff=(2(21.40(1-..84)1/2 ) 2) )1/2=10.92

• 95% CI: (SEdiff)*1.96=21.40

• T1 Mean=72.08

• T2 Mean=71.84

• Mean Diff=-0.24

• The mean difference fails to exceed 21.40

• Thus, no clinically meaningful change has occurred as a result of surgery

Reliable Change Index Scores—An Example

• Elaborated by Charter (1996)• Used primarily for maximal performance tests• The RCI of Jacobsen and Truax is used for

typical performance tests– It assumes that errors between test scores at baseline and time

2 are uncorrelated– This assumption is untenable in maximal performance tests

because of practice effects

Standardized Regression Based Change Scores

• Based on the standard error of prediction– SEpred=SDY2((1-rY1Y2

2)1/2)

• The SE reflects the sampling distribution of predicted scores• It implies the range of scores that might be expected at time two that may

be expected from the baseline score and prediction error• This method requires you to compute the estimated true score

– Y2True=M+((rY1Y2)(Y1-M))

• The T2 score is prone to error, and this formula permits an unbiased estimate of the true score

• The SEpred is used to compute a confidence interval around the estimated true score

Standardized Regression Based Change Scores

Standard Normal Curve—Distribution of Standard Error of Prediction Around

Estimated True Score

-4 -3 -2 -1 0 1 2 3 4

34% 34%

14% 14%2% 2%

50th 84th 98th16th2nd

• To use the SRB, you must compute the estimated true T2 score

• Compute the confidence interval around this estimated true T2 score

• For 95% confidence, you multiply 1.96 * SEpred• For 90% confidence, you multiply 1.60 * SEpred• Does the obtained T2 score fall outside the confidence

interval around the estimated true score for T2?– If so, it represents change that exceeds the base rate expected based

on measurement error, regression to the mean, and practice» Thus, clinically meaningful change has occurred

– If not, then the change is consistent with the base rate expected based on measurement error, practice, and regression to the mean» Thus, no clinically meaningful change has occurred

Standardized Regression Based Change Scores--Method

• Basso, Carona, Lowery, & Axelrod (2002)– WAIS-III re-tested in a group of control subjects over a 3-6 month interval– FSIQ

» Test-Retest Reliability=.90 » T1 Mean T1=109.4 (11.6)» T2 Mean T2=115.0 (12.1)

» SEpred=SDY2((1-rY1Y22)1/2)

• SEpred=(12.1((1-.902) 1/2))=5.29

• 95% CI: (SEdiff)*1.96=10.36• Mean Diff=5.60

– The mean difference fails to exceed the 95% CI– No individual had a score exceeding the 95% CI

• To apply the SRB, the T2 True Score is estimated– If the obtained score falls within the CI around the T2 True score, then no

clinically meaningful change has occurred

Standardized Regression Based Change Scores--An Example

• Basso, Carona, Lowery, & Axelrod (2002)– An example application:– T1 obtained score=104– T2 obtained score=116– Estimated True T2 Score

» YTrue=M+((rY1Y2)(Y1-M))

» YTrue=100+(.90)(104-100)=103.6

» 116 exceeds 10.36 points from 103.6

» Thus, meaningful change has occurred

Standardized Regression Based Change Scores--An Example

• Basso, Carona, Lowery, & Axelrod (2002)– An example application:– T1 obtained score=103– T2 obtained score=106– Estimated True T2 Score

» YTrue=M+((rY1Y2)(Y1-M))

» YTrue=100+(.90)(106-100)=105.4

» 105 falls within 10.36 points of 106

» Thus, no meaningful change has occurred

Standardized Regression Based Change Scores--An Example

Questions?