Prior Elicitation in Bayesian Clinical Trial Design Peter F. Thall Biostatistics Department M.D. Anderson Cancer Center SAMSI intensive summer research.

Prior Elicitation in Bayesian Clinical Trial Design

Peter F. ThallBiostatistics Department

M.D. Anderson Cancer Center

SAMSI intensive summer research program on

Semiparametric Bayesian Inference: Applications in Pharmacokinetics and

PharmacodynamicsResearch Triangle Park, North Carolina

July 13, 2010

Disclaimer

To my knowledge, this talk has nothing to do with semiparametric Bayesian inference, pharmacokinetics, or pharmacodynamics.

I am presenting this at Peter Mueller’s behest.

Blame Him!

Outline ( As time permits )

1. Clinical trials: Everything you need to know

2. Eliciting Dirichlet parameters for a leukemia trial

3. Prior effective sample size

4. Eliciting logistic regression model parameters for Pr(Toxicity | dose)

5. Eliciting values for a 6-parameter model of

Pr(Toxicity | dose1, dose2)

6. Penalized least squares for {Pr(Efficacy),Pr(Toxicity)}

7. Eliciting a hyperprior for a sarcoma trial

8. Eliciting two priors for a brain tumor trial

9. Partially informative priors for patient-specific dose finding

Clinical Trials

Definition: A clinical trial is a scientific experiment with human subjects.

1. Its first purpose is to treat the patients in the trial. 2. Its second purpose is to collect information that may be

useful to evaluate existing treatments or develop new, better treatments to benefit future patients.

Other, related purposes of clinical trials:3. Generate data for research papers 4. Obtain $$ financial support $$ from pharmaceutical

companies or governmental agencies5. Provide an empirical basis for drug or device approval

from regulatory agencies such as the US FDA

Medical Treatments

Most medical treatments, especially drugs or drug combinations, have multiple effects.

Desirable effects are called efficacy► Shrinkage of a solid tumor by > 50%► Complete remission of leukemia► Dissolving a cerebral blood clot that caused an ischemic stroke► Engraftment of an allogeneic (matched donor) stem cell transplant

Undesirable effects are called toxicity ► Permanent damage to internal organs (liver, kidneys, heart, brain)► Immunosuppression (low white blood cell count or platelet count)► Cerebral bleeding or edema (accumulation of fluid)► Graft-versus-host disease (the engrafted donor cells attack the

patient’s organs)► Regimen-related death due to any of the above

Scientific Method

Advice from Ronald Fisher Don’t waste information

Advice From Peter Thall Don’t waste prior information when

designing a clinical trial

Standard Statistical Practice Ignore Fisher’s advice and just run your favorite

statistical software package. And be sure to record lots and lots of p-values.

A Chemotherapy Trial in Acute Leukemia

Complete Remission (CR)

Yes No

Yes 1 2

No 3 4

TOX IC ITY

Model12 , 3, 4 ) ~ Dirichlet(a1, a2, a3, a4) ≡ Dir(a)

p(a) ∝ 1a1-1 2

a2-1 3a3-1 4

a4-1, a+ = a1+a2+a3+a4 = ESSTOX = 12 ~Be(a1+a2, a3+a4)

CR = 13 ~ Be(a1+a3, a2+a4) E(TOX) = (a1+a2 )/a+ E(CR) = (a1+a3 )/a+

4 = 1 – 1 – 2 – 3

If possible, use Historical Data to establish a prior: CR and Toxicity counts from 264 AML Patients Treated

With an Anthracycline + ara-C

CR No CR

Toxicity 73

(27.7%)

63 (23.9%) 136

(51.5%)

No

Toxicity

101

(38.3%)

27

(10.2%)

128

(48.5%)

174

(65.9%)

90

(34.1%)

264

P(CR | Tox) = 73/136 = .54P(CR | No Tox) = 101/128 = .79

CR and Tox are Not Independent

S = “Standard” treatmentE = “Experimental” treatment

S ~ Dir (73,63,101,27) aS,+ = ESS = 264 (“Informative”)

Set E = S with aE,+ = 4

E ~ Dirichlet (1.11, .955, 1.53, .409) (“Non-Informative”)

Dirichlet Priors and Stopping Rules

Stop the trial if

1) Pr(S,CR + .15 < E,CR | data) < .025 (“futility”), or

2) Pr(S,TOX + .05 < E,TOX | data) > .95 (“safety”)

But what if you don’t have historical data?!!

An Easy Solution:To obtain the prior onS

1) Elicit the prior marginal outcome probability means E(TOX) = (a1+a2 )/a+ and E(CR) = (a1+a3 )/a+

2) Assume independence and solve algebraically for (1, 2, 3, 4) = (a1, a2, a3, a4)/ a+

3) Elicit the effective sample size ESS = a+ that the elicited values E(TOX) and E(CR) were based on

4) Solve for (a1, a2, a3, a4)

Sensitivity Analysis of Association in the desirable case wherePr(CR) ↑ 0.15 from .659 to .809 and Pr(TOX) = .516

i.e. there is no increase in toxicity.

p11p00

p10p01

True E Probability of Stopping the Trial Early

Sample Size

(25%,50%,75%)

.007 (.027,.489,.782,.102) >.99 4 7 14

.138 (.227,.289, .582,.102) .56 14 44 56

1.28 (.427,.089,.382,.102) .16 56 56 56

52.6 (.510,.006,.299,.185) .16 56 56 56

Oops!!

If you don’t have historical data . . .

A slightly smarter wayto obtain prior(S

1) Elicit the prior means E(TOX) = (a1+a2 )/a+ and E(CR) = (a1+a3 )/a+

2) Elicit the prior mean of a conditional probability, like Pr(CR | Tox) = 1/(1 + 2), which has mean a1/(a1 + a2), and solve for (1, 2, 3, 4) = (a1, a2, a3, a4)/ a+ . That is, do not assume independence. 3) Elicit the effective sample size ESS = a+ that the values E(TOX) and E(CR) were based on

4) Solve for (a1, a2, a3, a4)

Rocket Science!!

Example

Elicited prior values

E(TOX) = (a1+a2 )/a+ = .30

E(CR) = (a1+a3 )/a+ = .50 E{ Pr(CR | Tox) } = E{ 1/(1 + 2)} = a1/(a1 + a2) = .40

ESS = a+ = 120

(a1, a2, a3, a4) = (14.4, 21.6, 45.6, 38.4)

(1, 2, 3, 4) = (a1, a2, a3, a4)/ a+ = (.12, .18, .38, .32)

A Fundamental question in Bayesian analysis: How much information is contained in the

prior?

Priorp(θ)(((

(((

(((

(((

Determining the Effective Sample Size of a

Parametric Prior (Morita, Thall and Mueller, 2008)

The answer is straightforward for many commonly used models

E.g. for beta distributions

Be (1.5,2.5)

Be (16,19)

Be (3,8)

ESS = 16+19 = 35

ESS = 3+8 = 11

ESS = 1.5+12.5 = 5

But for many commonly used parametric Bayesian models it is not obvious how to determine the ESS of the prior.

E.g. usual normal linear regression model

22

10

210

210

inverse~ normal, bivariate~,

,,

)( ,)(

YVarXYE

Intuitive Motivation

Saying Be(a, b) has ESS = a+b implicitly refers to the wel known fact that

θ ~ Be(a, b) and Y | θ ~ binom(n, θ)

θ | Y,n 〜 Be(a +Y, b +n-Y) which has ESS = a + b + n

So, saying Be(a,b) has ESS = a + b implictly refers to an

earlier Be(c,d) prior with very small c+d = and solving

for m = a+b – (c+d) = a+b – for a very small > 0

General Approach

1) Construct an “-information” prior q0(θ), with same

means and corrs. as p(θ) but inflated variances

2) For each possible ESS m = 1, 2, ... consider a

sample Ym of size m

3) Compute posterior qm(θ|Ym) starting with prior q0(θ)

4) Compute the distance between qm(θ|Ym) and p(θ)

5) The interpolated value of 5) The interpolated value of mm minimizing the distance minimizing the distance

is the ESS.is the ESS.

A Phase I Trial to Find a Safe Dose forAdvanced Renal Cell Cancer (RCC)

Patients with renal cell cancer, progressive after treatment with Interferon

Treatment = Fixed dose of 5-FU + one of 6 doses of Gemcitabine: {100, 200, 300, 400, 500, 600} mg/m2

Toxicity = Grade 3,4 diarrhea, mucositis, or hematologic (blood) toxicity

Nmax = 36 patients, treated in cohorts of 3

Start the with1st cohort treated at 200 mg/m2

Adaptively pick a “best” dose for each cohort

Continual Reassessment Method (CRM, O’Quigley et al.

1990) with a Bayesian Logistic Regression Model

1) Specify a model for (xj, = Pr(Toxicity| , dose xj) and prior on

2) Physician specifies pTOX* = a target Pr(Toxicity)3) Treat each successive cohort of 3 pats. at the “best”

dose for which E[(xj, | data] is closest to pTOX*4) The best dose at the end of the trial is selected

exp( +xj )(xj, = 1 + exp( +xj )

using xj = log(dj) - {j=1,…klog(dj)}/k , j=1,…,6.

Prior: ~ N(, 2), ~ N(,

2)

Elicit the mean toxicity probabilities at two doses. In the RCC trial, the elicited prior values were

E{(200,)} = .25 and E{(500,)} = .75

1) Solve algebraically for = -.13 and = 2.40

2) = = 2 ~ N(-.13, 4), ~N(2.40, 4) which gives prior ESS = 2.3

Alternatively, one may specify the prior ESS and solve for = =

CRM with Bayesian Logistic Regression Model

Plot of ESS as a function of

ESS{ p(,)|}

0.1 0.2 0.3 0.4 0.5 0.7 1 2 3 4 5 10

ESS 928 232 103 58.0 37.1 18.9 9.3 2.3 1.0 0.58 0.37 0.09

For cohorts of size 1 to 3, =1 is still too small since it

gives prior ESS = 9.3

These ESS values are OK, so = 2 to 5 is OK. ?

These values give a prior with far more

information than the data in a typical phase I trial.

Prior of = Prob(tox | d = 200, )

p()

=0.1

=0.5=2.0

=10.0

ESS=928ESS=928

ESS=37.1ESS=37.1ESS=2.3ESS=2.3

ESS=0.09ESS=0.09

Why not just set = = = a very large number, so ESS = a = a very large number, so ESS = a tiny number, and have a very “non-informative” prior ?tiny number, and have a very “non-informative” prior ?

Example: A “non-informative” prior is ~ N(-.13,100) and ~ N(2.40,100), i.e. =10.0 ESS = 0.09.

But this prior has some very undesirable properties :

Prior Probabilities of Extreme

Values

Dose of Gemcitabine (mg/m2)

100 200 300 400 500 600

Pr{(x,)<.01} .45 .37 .33 .31 .31 .31

Pr{(x,)>.99} .30 .30 .32 .35 .38 .40

This says you believe, a priori, that

1) Pr{(x,< .01} = Prob(toxicity is virtually impossible) = .31 to .45

2) Pr{(x,> .99} = Prob(toxicity is virtually certain) =

.30 to .40

Making Making === = too large (a so-called “non too large (a so-called “non

informative” prior) gives a pathological prior. informative” prior) gives a pathological prior. What What is “too large” numerically is not obvious without is “too large” numerically is not obvious without

computing the corresponding ESS.computing the corresponding ESS.

Dose-Finding With Two Agents(Thall, Millikan, Mueller, Lee, 2003)

Study two agents used together in a phase I clinical trial, with

dose-finding based on (x,) = probability of toxicity for a

patient given the dose pair x = (x1, x2)

Find one or more dose pairs (x1, x2) of the two agents used

together for future clinical use and/or study in a randomized

phase II trial

Elicit prior information on (x,) with each agent used alone

Single Agent Toxicity Probabilities :

1 x11= x1= Prob(Toxicity | x1, x2=0, 1)

2 x22= x2= Prob(Toxicity | x1=0, x2, 2)

0

20

0

40

0

60

0

80

0

1,0

00

1,2

00

1,4

00 0

400

600

900

0

10

20

30

40

50

60

70

80

P(tox)

Cyclophosphamide

Gemcitabine

Hypothetical Dose-Toxicity Surface

Probability Model

x2=0 1 x11= 1 x1 / ( 1 + 1 x1

) = exp(1)/{1+exp(1)}

x1=0 2 x22= 2 x2 / (1 +2 x2

) = exp(2) / {1+ exp(2)}

where j = log(j)+j log(xj) for j=1,2

12), where 111and222have elicited informative priors and the interaction parameters23,3have non-informative priors.

Single-Agent Prior Elicitation Questions

1. What is the highest dose having negligible (<5%) toxicity?

2. What dose has the targeted toxicity * ?

3. What dose above the target has unacceptably high (60%) toxicity?

4. At what dose above the target are you nearly certain (99% sure) that toxicity is above the target (30%) ?

Resulting Equations for the Hyperparameters

Denote g() = / (1+) so (x,)} = g(x). Denote the doses given as answers to the questions by

{ x(1), x(2), x(3) = x*, x(4) }, and zj = x(j) / x*.

Assuming ~ Ga(a1 , a2 ) and ~ Ga(b1 , b2 ), solve the following equations for (a1 , a2 , b1 , b2 ) :

1. Pr{ g(z1) < .05 } = 0.99

2. E(z*)) = a1 a2 E(1) = * / (1 - * )

3. E(z3) = a1 a2 E(z3

) = 0.60 / 0.40 = 1.5

4. Pr{ g(z4) > * } = 0.99

The answers to the 4 questions for each single agent

Randy Millikan, MD

An Interpretation of this Prior

The ESS of p(θ) = p(θ1, θ2, θ3) is 1.5

Since informative priors on θ1 and θ2 and a vague prior on θ3 were elicited, it is useful to determine the prior ESS of each subvector :

ESS of marginal prior p(θ1) is 547.3 for (x1,0 | 11)}

ESS of marginal prior p(θ2) is 756.3 for (0,x2 | 22)}

ESS of marginal prior p(θ3) is 0.01 for the interaction

parameters θ3 = (33)

This illustrates 4 key features of prior ESS

1. ESS is a readily interpretable index of a prior’s informativeness.

2. It may be very useful to compute ESS’s for both the entire parameter vector and for particular subvectors

3. ESS values may be used as feedback in the elicitation process

4. Even when standard distributions are used for priors, it may NOT be obvious how to define a prior’s ESS.

For indices a=0,1 and b=0,1, and x = standardized dose,

a,b (x, ) = Pr(YE = a , YT = b | x, )

= Ea(1-E)1-a T

b(1-T)1-b + (-1)a+b E(1-E)T(1-T) (e-1)/(e+1)

with marginals

logit T(x,) = TxT

logit E(x,) = E xE,1 x2E,2 The model parameter vector is (TTEE,1 E,2 ,

Probability Model for Dose-Finding Based on Bivariate Binary Efficacy (Response) and Toxicity Indicators YE and YT

(Thall and Cook, 2004)

Establishing Priors

1) Elicit mean & sd of T(x,) & E(x,) for several values of x. 2) Use least squares to solve for initial values of the

hyperparameters in prior( | ) 3)Each component of is assumed normally distributed, r ~ N(r, r), so = (1,1,…, p,p)4) mE,j = prior mean and sE,j = prior sd of E(xj, mT,j = prior mean and sT,j = prior sd of T(xj,5) # elicited values > dim() find the vector that minimizes

the objective function

Penalty term to keep the ’s on the same numerical domain, c = .15

A trial of allogeneic stem cell transplant patients: Up to 12 cohorts of 3 each (Nmax = 36) were treated to determine

a best dose among {.25, .50, .75, 1.00 } mg/m2 of Pentostatin® as prophyaxis for graft-versus-host disease.

E = drop from baseline of at least 1 grade in GVHD at week 2 T = unresolved infection or death within 2 weeks.

Example: Elicited Prior for the illustrative application in Thall and Cook (2004)

ESS() = 8.9 (equivalent to 3 cohorts of patients!!)

ESS(E) = 13.7, ESS(T) = 5.3, ESS() = 9.0

A Slightly Smarter Way to Think About Priors

Fix the means

and use ESS contour plots to choose

Example:

A Strategy for Determining Priors in the Regression Model

To obtain desired overall ESS = 2.0 and ESSE = ESST = ESS = 2.0, one may inspect the ESS plots to choose the variances of the hyperprior. One combination that gives this is

Eliciting the Hyperprior for a Hierarchical Bayesian Model in a Phase II Trial (Thall, et at. 2003)

A single arm trial of Imatinib (Gleevec, STI571) in sarcoma, accounting for multiple disease subtypes.

i = Pr( Tumor response in subtype i )

Prior: logit(i) | ~ i.i.d Normal( ), i=1,…,k

Hyperprior: ~ N( -2.8, 1), ~ Ga( 0.99, 0.41 )

Stopping Rule: Terminate accrual within the ith subtype if

Pr( i > 0.30 | Data ) < 0.005

“Data” refers to the data from all 10 subtypes.

But where did these numbers come from?

Eliciting the Hyperprior

Denote Xi = # responders out of mi patients in subtype i.

1) I fixed the mean of at logit(.20) = -1.386, to correspond to mean prior response rate midway between the target .30 and the uninteresting value .10.2) I elicited the following 3 prior probabilities :

Pr( 1 > 0.30 ) = 0.45

Pr( 1 > 0.30 | X1 / m1 = 2/6) = 0.525

Pr( 1 > 0.30 | X2 / m2 = 2/6) = 0.47

Prior Correlation Between Two Sarcoma Subtype Response

Probabilities 1 and 2

Two Priors for a Phase II-III Pediatric Brain Tumor Trial

A two-stage trial of 4 chemotherapy combinations :S = carboplatin + cyclophosphamide + etoposide + vincristine

E1 = doxorubicin + cisplatinum + actinomycin + etoposide

E2 = high dose methotrexate

E3 = temozolomide + CPT-11

Outcome (T,Y) is 2-dimensional : T = disease-free survival timeY = binary indicator of severe but non-fatal toxicity

Both p(T | Y,Z,) and p(Y | Z,) account for patient covariates:

Age, I(Metastatic disease), I(Complete resection)

I(Histology=Choriod plexus carcinoma)

Probability Model

1) T| Z,Y, j ~ lognormal with variance T2 and

mean T,j(Z,Y,) = T,j + T(Z,Y)

T,j = effect of trt j on T, after adjusting for Z and Y

For j=0 (standard trt), T = (T,0 , T)

2) logit{Pr(Y=1 | Z, j)} = Y,j + Y Z

Y,j = effect of trt j on Y, after adjusting for Z

For j=0 (standard trt), Y = (Y,0 , Y)

Toxicity Probability as a Function of Age Elicited from Three Pediatric Oncologists

Probability Model for Toxicity

logit{Pr(Y=1 | Z, , j=0)} = Y,0 + Y,1 Age1/2 + Y,2 log(Age)

was determined by fitting 72 different fractional polynomial functions and picking the one giving the smallest BIC.

Estimated linear term with posterior mean subscripted by the posterior sd is

This determined the prior of Y

64 Elicited EFS Probabilities

How do you use these 64

probabilities to solve for 10

hyperparameters?!!

Johannes Wolff, MD

T = (T,j , T, T) has prior

Regard each prior mean EFS prob as a func of

Use nonlinear least squares to solve for

by minimizing E(T) = (0.44, -0.41, 0.56, -0.53) with common

variance 0.152

and log(T) ~ N(-0.08, 0.142)

Prior for T

YE = indicator of Efficacy

YT = indicator of Toxicity

d = assigned dose

Z = vector of baseline patient covariates

Model the marginals

E(d, Z) = Prob(E if d is given to a patient with covs Z)

T(d, Z) = Prob(T if d is given to a patient with covs Z)

Use a copula to define the joint distribution :

a,b = Pr(YE=a, YT=b) is a function of E(d, Z) and T(d, Z)

A Phase I/II Dose-Finding Method Based on E and T that Accounts for Covariates

E = link{ E(d,Z) } & T = link{ T(d,Z) }

where E(d,Z) & T(d,Z) are functions of

[ dose effects ] + [ covariate effects ]

+ [ dose-covariate interactions ]

a,b = Pr(YE=a, YT=b) = func(E, T ,for a, b = 0 or 1

Model for E(d,Z) and T(d,Z)

For the trial:

E(x, Z) = f(x,E) + EZ + x EZ

For the historical treatment j :

E( j, Z) = E,j + E,HZ + E,jZ

Linear Terms of the Model for E(,Z)

Dose effect Covariate effects Dose-Covariate Interactions

Historical trt effect Historical trt-covariate interactions

For the trial:

T(x, Z) = f(x,T) + TZ + x TZ

For the historical treatment j :

T( j, Z) = T,j + T,HZ + T,jZ

Linear Terms of the Model for T(,Z)

Dose effect Covariate effects Dose-Covariate Interactions

Historical trt effects Historical trt-covariate interactions

In planning the trial, historical data are used to estimate patient covariate main effects :

Prior(T) = Posterior(T,H | Historical data)

Prior(E) = Posterior(E,H | Historical data)

The estimated covariate effects are incorporated

into the model for E(d,Z) and T(d,Z) used to

plan and conduct the trial

Using Historical Data

For a reference patient Z*, elicit prior means of T(xj, Z*) and E(xj, Z*) at each dose xj to establish prior means of the dose effect parameters

Assume non-informative priors on dose effects and dose-covariate interactions

Use prior variances to tune prior effective

sample size (ESS) in terms of E and T

Establishing Priors

Control the prior ESS to make sure that the data

drives the decisions, rather than the prior on

the dose-outcome parameters

Application

A dose-finding trial of a new “targeted” chemo-chemo-protective agent (CPA)protective agent (CPA) given with idarubicin + cytosine arabinoside (IDA) for untreated acute myelogenous leukemia (AML)patients age < 60

Historical data from 693 AML patients

Z = (Age, Cytogenetics)

where Cytogenetics = (Poor, Intermediate, Good)

Inv-16 or t(8:21) -5 or -7

Application

Efficacy = Alive and in Complete Remission at day 40 from the start of treatment

Toxicity = Severe (Grade 3 or worse) mucositis, diarrhea, pneumonia or sepsis within 40 days from the start of treatment

Doses and Rationale

The CPACPA is hypothesized to decrease the risk of IDA-induced mucositis and diarrhea and thus allow higher doses of IDA

Fixed CPACPA dose = 2.4 mg/kg and ara-C dose = 1.5 mg/m2 daily on days 1, 2, 3, 4

IDA dose = 12 (standard), 15, 18, 21 or 24 mg/m2

daily on days 1, 2, 3 (five possible IDA doses)

Interactive

E( j, Z) = E,j + EZ + E,jZ

T( j, Z) = T,j + TZ + T,jZ

Additive

E( j, Z) = E,j + EZ

T( j, Z) = T,j + TZ

Reduced

E( j, Z) = E + EZ

T( j, Z) = T + TZ

Models for the linear terms used to fit the historical data

No treatment-covariate interactions

No differences between the

historical treatment effects

Model Selection for Historical Data

Posteriors of E(, Z) and T(, Z) based on

Historical Data from 693 Untreated AML Patients

1) Choose each patient’s most desirable dose based on his/her Z

2) No dose acceptable for that Z : No dose acceptable for that Z :

Do Not TreatDo Not Treat

3) At the end of the trial, use the fitted model to pick ( d | Z ) for future patients

Dose-Finding Algorithm

The trial’s entry criteria may change dynamically dynamically during the trial :

1) Different patients may receive different doses at the same point in the trial

2) Patients initially eligible may be ineligible (no acceptable dose) after some data have been observed

3) Patients initially ineligible may become eligible after some data have been observed

Hypothetical Trial Results :

Recommended Idarubicin Doses by Z

AGE Cyto Poor Cyto Int Cyto Good

18 – 33 1818 2424 2424

34 – 42 1818 2121 2424

43 – 58 1515 1818 2121

59 – 66 1212 1515 1818

> 66 NoneNone 1212 1515

Currently being used to conduct a 36-patient trial to select among 4 dose levels of a new cytotoxic agent for relapsed/refractory Acute Myelogenous Leukemia

Y = (CR, Toxicity) at 6 weeks

Z = (Age, [1st CR > 1 year], Number of previous trts)

Marina Konopleva, MD, PhDis the PI

Bibliography

[1] Morita S, Thall PF, Mueller P. Determining the effective sample size of a parametric prior. Biometrics. 64:595-602, 2008.

[2] Morita S, Thall PF, Mueller P. Evaluating the impact of prior assumptions in Bayesian biostatistics. Statistics in Biosciences. In press.

[3] Thall PF, Cook JD. Dose-finding based on efficacy-toxicity trade-offs. Biometrics, 60:684-693, 2004.

[4] Thall PF, Simon R, Estey EH. Bayesian sequential monitoring designs for single-arm clinical trials with multiple outcomes. Statistics in Medicine 14:357-379, 1995.

[5] Thall PF, Wathen JK, Bekele BN, Champlin RE, Baker LO, Benjamin RS. Hierarchical Bayesian approaches to phase II trials in diseases with multiple subtypes. Statistics in Medicine 22: 763-780, 2003.

[6] Thall PF, Wooten LH, Nguyen HQ, Wang X, Wolff JE. A geometric select-and-test design based on treatment failure time and toxicity: Screening chemotherapies for pediatric brain tumors. Submitted for publication.

[7] Thall PF, Nguyen H, Estey EH. Patient-specific dose-finding based on bivariate outcomes and covariates. Biometrics. 64:1126-1136, 2008.