Hypothesis Testing and Dynamic Treatment Regimes S.A. Murphy, L. Gunter & B. Chakraborty ENAR March 2007.

Hypothesis Testing and Dynamic Treatment Regimes

S.A. Murphy, L. Gunter & B. Chakraborty

ENAR

March 2007

2

Outline

• Dynamic treatment regimes• Constructing and addressing questions regarding

an optimal dynamic treatment regime• Why and when non-regular?• A Solution• Simulation Results.

3

Dynamic treatment regimes are individually tailored treatments, with treatment type and dosage changing according to patient outcomes. Operationalize clinical practice.

k Stages for one individual

Observation available at jth stage

Action at jth stage

4

k Stages

History available at jth stage

“Reward” following jth stage (rj is a known function)

Primary Outcome:

5

Goal:

Construct decision rules that input information in the history at each stage and output a recommended decision; these decision rules should lead to a maximal mean Y.

The dynamic treatment regime is the sequence of decision rules:

6

In the future we employ the actions determined by the decision rules:

An example of a simple decision rule is: alter treatment at time j if

otherwise maintain on current treatment; Sj is a summary of the history, Hj.

7

Data for Constructing the Dynamic Treatment Regime:

Subject data from sequential, multiple assignment, randomized trials. At each stage subjects are randomized among alternative options.

Aj is a randomized action with known randomization probability.

binary actions with P[Aj=1]=P[Aj=-1]=.5

8

Sequential, Multiple Assignment Randomized Studies

• CATIE (2001) Treatment of Psychosis in Schizophrenia

• STAR*D (2003) Treatment of Depression• Tummarello (1997) Treatment of Small Cell Lung

Cancer (many, for many years, in this field)• Oslin (on-going) Treatment of Alcohol

Dependence• Pellman (on-going) Treatment of ADHD

9

Sequential Multiple Assignment Randomization

Initial Txt Intermediate Outcome Secondary Txt

Relapse

Responder R Prevention

Low-levelMonitoring

Switch toTx C

Tx A

Nonresponder RAugment withTx D

R

Responder Relapse

R Prevention

Low-levelMonitoring

Tx B

Switch toTx C

Nonresponder R

Augment withTx D

10

Constructing and Addressing Questions Regarding an Optimal Dynamic

Treatment Regime

11

Regression-based methods for constructing decision rules

•Q-Learning (Watkins, 1989) (a popular method from computer science)

•A-Learning or optimal nested structural mean model (Murphy, 2003; Robins, 2004)

•The first method is an inefficient version of the second method when each stages’ covariates include the prior stages’ covariates and the actions are centered to have conditional mean zero.

12

(k=2)

Dynamic Programming

13

Approximate for S', S vector summaries of the history and

A Simple Version of Q-Learning –binary actions

• Stage 2 regression: Use least squares with outcome, Y, and covariates to obtain

• Set

• Stage 1 regression: Use least squares with outcome, and covariates to obtain

14

Decision Rules:

15

Why non-regular?

16

Non-regularity

17

When do we have non-regularity?

18

A Soft-Max Solution

19

A Soft-Max Solution

20

Distributions for Soft-Max

21

Regularized Q-Learning (binary actions)

• Set

• Stage 1 regression: Use least squares with outcome,

and covariates to obtain

22

Interpretation of λ

Future treatments are assigned with equal probability, λ=0

Optimal future treatment is assigned, λ=∞

Future treatment =1 is assigned with probability

Estimator of Stage 1 Treatment Effect when

23

Interpretation of λ

24

Proposal

25

Proposal

26

Proposal

27

Simulation

28

P[β2TS2=0]=1 β1(∞)=β1(0)=0

Test Statistic Nominal Type 1 based on Error=.05 .045

.047

.034*

.024*

(1)Nonregularity results in low Type 1 error

(2)Additional smoothing due to use of is useful.

29

P[β2TS2=0]=1 β1(∞)=β1(0)=.1

Test Statistic Power based on

.15

.14

.10

.09

(1)The low Type 1 error rate translates into low power

30

Test Statistic Power based on

.05

.13

.12

.12

(1) Averaging over the future is not a panacea

P[β2TS2=0]=0 β1(∞)=.125, β1(0)=0

31

Test Statistic Type 1 Error=.05 based on

.57

.16

.05

.05

(1) The price is that the null hypothesis is altered.

P[β2TS2=0]=.25 β1(∞)=0, β1(0)=-.25

32

Discussion

• We replace the hypothesis test concerning a non-regular parameter, β1(∞) by a hypothesis test concerning a near-by regular parameter β1(λ*).

• This is work in progress—limited theoretical results are available.

• If you let increase with the sample size you again end up with a non-regular problem (convergence to limiting distribution is locally non-uniform).

33

Discussion

• Robins (2004) proposes several conservative confidence intervals for β1.

• Ideally to decide if the two stage 1 treatments are equivalent, we would evaluate whether the choice of stage 1 treatment influences the mean outcome resulting from the use of the dynamic treatment regime. We did not do this here.

• Constructing “evidence-based” regimes is of great interest in clinical research and there is much to be done by statisticians.

34

This seminar can be found at:http://www.stat.lsa.umich.edu/~samurphy/

seminars/ENAR0307.ppt

Email me with questions or if you would like a copy!

[email protected]