Disability weights measurement in the Global Burden of Disease Study 2010

Disability weights measurement in the Global Burden of Disease Study 2010

Joshua A Salomon

Harvard School of Public Health

Global Health Metrics and Evaluation Conference

June 18, 2013Supported by funding from the Bill and Melinda Gates Foundation

Salomon – Disability weights - 2 2

Global Burden of Disease Study

• Global Burden of Disease (GBD) Study aims to measure impact of disease and injury in terms of losses in population health

• GBD 1990 study launched in 1991, updated by WHO in 2000s• GBD 2010 study, undertaken from 2007-2012, provides first

comprehensive overhaul since 1996

• GBD quantifies • Magnitude of different health problems in units of disability-adjusted

life years (DALYs)• Overall population health in units of healthy life expectancy (HALE)

Salomon – Disability weights - 3 3

Disability weights

• Disability weights provide the bridge between mortality and non-fatal outcomes in DALYs and in healthy life expectancy

• To measure health impact of non-fatal outcomes, GBD needs weights for all unique sequelae, which capture the major health consequences of all of the causes in the study

• Disability weights quantify severity of outcomes as percentage reductions from perfect health, which are multiplied by years lived in each sequela to give years lived with disability (YLD) e.g., If weight for blindness were 0.20, then 5 years lived with

blindness would be equivalent to dying one year prematurely

Salomon – Disability weights - 4

Intervention 2:Prevent 1y of deafness for

2000 individuals

Intervention 1:Extend life by 1y in

1000 healthy individuals

Disability weights in the 1996 GBD revision

Person trade-off: which would you choose?

• Expert panel used ‘person trade-off’ to assign values to 22 indicator conditions

Salomon – Disability weights - 5

1

Disability weights

2 3 4 5 6 7

Class 1:• Vitiligo on face

Class 4:• Below-knee amputation• Deafness

Class 7:• Active psychosis• Quadriplegia

Disability weights in the 1996 GBD revision

• Expert panel used ‘person trade-off’ to assign values to 22 indicator conditions

• These 22 conditions used as operational definitions of 7 disability classes

Salomon – Disability weights - 6

1

Disability weights

2 3 4 5 6 7

Rheumatoid arthritis cases

Average disability weight=0.2*0.07 + 0.4*0.18 + 0.4*0.30=0.21

Disability weights in the 1996 GBD revision

• Expert panel used ‘person trade-off’ to assign values to 22 indicator conditions

• These 22 conditions used as operational definitions of 7 disability classes

• Remaining conditions allocated across classes to compute average weights

Salomon – Disability weights - 7 7

Disability weights measurement study goals

• Derive weights for all 220 health states capturing nonfatal outcomes from 291 disease and injury causes in GBD 2010

• Address criticisms of previous approaches by: Focusing on valuations from community respondents… … in a diverse range of settings … using suitable measurement methods

• Specific research aims Develop valid and reliable data collection tools for population-

based surveys Empirical examination of variation in weights

Salomon – Disability weights - 8

• Background• Study design and methods• Key findings• Interpretations, limitations, conclusions

Disability weights measurement in GBD 2010

Salomon – Disability weights - 9 9

Study components

• Population-based household surveys• Face-to-face interviews in Tanzania,

Bangladesh, Indonesia, Peru• Telephone interview in random sample of

US households• Focus on paired comparisons for 108

health-states• Key objectives include comparative analysis

across diverse settings and benchmarking Internet survey against community samples

• Open-access Internet surveys• Available in English, Spanish and Mandarin• Key objectives are to fill in gaps with

remaining sequelae and to anchor scale for paired comparison responses

Bangladesh

Tanzania

Salomon – Disability weights - 10

Web survey

Web survey included 16,328 respondents from 167 countries

Salomon – Disability weights - 11

Web survey included 16,328 respondents from 167 countries

Web survey

1 - 9

10 - 49

50 - 99

100 - 499

500+

Salomon – Disability weights - 12 12

Measurement methods: paired comparisons

• Primary mode of eliciting responses is paired comparison• Respondents hear (or read) two descriptions of hypothetical people,

each with a randomly selected condition• Respondents indicate which person is healthier

• Paired comparison questions chosen for relative ease of comprehension, administration and analysis• Literacy and numeracy not essential• Health comparisons not tied to external “calibrators” such as risk• Appealing intuitive basis and established strategies for analysis

Salomon – Disability weights - 13 13

Framing paired comparisons

• Basis for all comparison are lay descriptions of sequelae, which highlight major functional consequences and symptoms associated with each sequela

• Must be brief: restricted to <35 words based on pretest results• Must use simple, non-clinical vocabulary

• Prologue to paired comparison questions orients respondents to focus on functioning

A person’s health may limit how well parts of his body or his mind work. As a result, some people are not able to do all of the things in life that others may do, and some people are more severely limited than others.

I am going to ask you a series of questions about different health problems. In each question I will describe two different people … Imagine they have the same number of years left to live, and will experience the health problems that I describe for the rest of their lives. I will ask you to tell me which person you think is healthier overall, in terms of having fewer physical or mental limitations on what they can do in life…

Salomon – Disability weights - 14

Paired comparison example

The first person has vision problems that make it difficult to see and recognize faces of family or friends across a room.

The second person has severe back and leg pain, which causes difficulty dressing, sitting, standing, walking, and lifting things. The person sleeps poorly and feels worried.

Imagine that both people will have these problems for the rest of their lives. Who would you say is healthier overall, the first person or the second person?

• Respondents in household surveys each answered 15 of these questions, with pairs of sequelae drawn at random from the universe of 108 × 108 possible pairs

Salomon – Disability weights - 15 15

Analyzing paired comparison data

Bangladesh Tanzania• Simple ordering of

outcomes based on “winning” proportions, as in various websites using paired comparisons to rank large pools of competitors

http://kittenwar.com/

Salomon – Disability weights - 16 16

Analyzing paired comparison data

Bangladesh Tanzania

Won 78% of 511 battles Won 77% of 539 battles

Winningest

Lost 80% of 846 battles Lost 80% of 1872 battles

Losingest

• Simple ordering of outcomes based on “winning” proportions, as in various websites using paired comparisons to rank large pools of competitors

http://kittenwar.com/

Salomon – Disability weights - 17 17

Analyzing paired comparison data

Bangladesh Tanzania• Simple ordering of

outcomes based on “winning” proportions, as in various websites using paired comparisons to rank large pools of competitors

• More sophisticated algorithms incorporate information on the relative strength of the opponent (e.g. Elo rating system in Chess)

• Statistical approaches enable maximum likelihood estimation of underlying scores

Salomon – Disability weights - 18 18

Statistical modeling of paired comparisons data

• Conceptual foundation for statistical modeling of paired comparisons data comes from Thurstone (1927), through Luce (1959), McFadden (1974) and others

• Intuitively, a pair of health states that are similar in severity are likely to produce greater disagreement over which is healthier than a pair of states that are different in severity

• Consider a population in which: 70% of people think depression is worse than blindness 90% of people think blindness is worse than arthritis

• Reasonable to conclude that depression and blindness are nearer on some unobserved scale than blindness and arthritis

Salomon – Disability weights - 19

-4 -2 0 2 4 6 8

blindness

90%

Salomon – Disability weights - 20

-4 -2 0 2 4 6 8

depression blindness

90%70%

Salomon – Disability weights - 21

-4 -2 0 2 4 6 8

depression blindness

90%

Salomon – Disability weights - 22

-4 -2 0 2 4 6 8

depression blindnessarthritis

90%

Salomon – Disability weights - 23

-4 -2 0 2 4 6 8

depression blindnessarthritis

Salomon – Disability weights - 24 24

Statistical modeling of paired comparisons data

• Statistical model formalizes this intuition Each health state i has an unobserved actual health level Xi as perceived by an

individual rater The rater makes a choice in the paired comparison based on which of the

health states is regarded as ‘healthier’State 1 is chosen over State 2 if X1 > X2

• Thurstone model assumes that Xi is normally distributed, implying that the choice probabilities are also normal

• We can model response probabilities using probit regression Requires arbitrary identifying assumptions Assumes that each health state has the same variance around the disability

weight

Salomon – Disability weights - 25

From paired comparisons to disability weights

Analysis of paired comparisons

Analysis of “population health equivalence”

responses

Rescaling of paired comparison results

47

Salomon – Disability weights - 26

From paired comparisons to disability weights

Analysis of paired comparisons

Analysis of “population health equivalence”

responses

Rescaling of paired comparison results

• Pooled data from all household surveys, CATI and web survey

• Estimated probit model with separate variance for each survey

Salomon – Disability weights - 27

From paired comparisons to disability weights

Analysis of paired comparisons

Analysis of “population health equivalence”

responses

Rescaling of paired comparison results

• Some Web survey respondents randomly assigned to “population health equivalence” (PHE) questions, for subset of 30 sequelae

• PHE provides separate estimates for these sequelae that are anchored on (0,1) disability scale

49

Salomon – Disability weights - 28

From paired comparisons to disability weights

Analysis of paired comparisons

Analysis of “population health equivalence”

responses

Rescaling of paired comparison results

• Probit scale mapped to (0,1) disability scale by regressing probit coefficients on PHE anchors

• Non-constant variance accommodated by regression on logit-transformed PHE values, then back-transforming values into (0,1) space

50

Salomon – Disability weights - 29

• Background• Study design and methods• Key findings• Interpretations, limitations, conclusions

Disability weights measurement in GBD 2010

Salomon – Disability weights - 30

Results: paired comparison responses

• Probabilities of responses on paired comparisons (‘Who is healthier?’) summarized in heat maps.

Best Worst

Best

Worst

First health state in pair

Second health state in pair

Salomon – Disability weights - 31

• Probabilities of responses on paired comparisons (‘Who is healthier?’) summarized in heat maps.

High agreement in choices between very healthy vs. unhealthy outcomes (>90%)

… or vice versa (<10%)

Split responses for similar outcomes (~50%)

Results: paired comparison responses

Salomon – Disability weights - 32

Measurement error in paired comparisons

• In the household survey, we assessed test-retest reliability by randomly assigning 20% of respondents to have the same pair repeated as 1st and 15th comparison

• Consistency of responses ranged between 60% and 70%

• Consistency of 2 coin flips would be 50%

• Kappa values range from 20% to 40%

Percent of respondents with consistent retest responses

36

Salomon – Disability weights - 33

• Probabilities of responses on paired comparisons (‘Who is healthier?’) summarized in heat maps.

High agreement in choices between very healthy vs. unhealthy outcomes (>90%)

… or vice versa (<10%)

Split responses for similar outcomes (~50%)

Results: paired comparison responses

Salomon – Disability weights - 34

Results: paired comparisons across surveys

Salomon – Disability weights - 35 35

Results: probit values across surveys

Salomon – Disability weights - 36

Results: comparison of household and web surveys

• Web respondents comprise non-random, highly educated, self-selected sample

Educational attainment in HH & web samples

Salomon – Disability weights - 37

Results: comparison of household and web surveys

Tanzania (N=2,613)

Web (N=3,417)

• Web respondents comprise non-random, highly educated, self-selected sample

• But, response probabilities are virtually indistinguishable from those in household surveys

Salomon – Disability weights - 38

Results: comparison of household and web surveys

• Web respondents comprise non-random, highly educated, self-selected sample

• But, response probabilities are virtually indistinguishable from those in household surveys

• And estimated weights from probit regressions are very highly correlated

Salomon – Disability weights - 39

Results: new disability weights

Salomon – Disability weights - 40

Results: new disability weights

Salomon – Disability weights - 41

Results: new disability weights

Salomon – Disability weights - 42

Results: new disability weights

Salomon – Disability weights - 43

Results: new disability weights

Salomon – Disability weights - 44

Results: comparison to previous disability weights

Salomon – Disability weights - 45

Framing challenge: drug use disorders

…uses heroin daily and has difficulty controlling the habit. When the effects wear off, the person feels severe nausea, agitation, vomiting and fever. The person has a lot of difficulty in daily activities.

Salomon – Disability weights - 46

Framing challenge: drug use disorders

…uses heroin daily and has difficulty controlling the habit. When the effects wear off, the person feels severe nausea, agitation, vomiting and fever. The person has a lot of difficulty in daily activities.

Salomon – Disability weights - 47

Framing experiment: caffeine addiction

Alternative descriptions for caffeine addiction

…drinks several cups of coffee a day in order to increase energy and stay alert. When the effects wear off, the person feels tired and irritable and sometimes gets headaches

…takes medication several times a day in order to increase energy and stay alert. When the effects wear off, the person feels tired and irritable and sometimes gets headaches

…uses an addictive substance several times a day in order to increase energy and stay alert. When the effects wear off, the person feels tired and irritable and sometimes gets headaches

…uses an addictive drug several times a day in order to increase energy and stay alert. When the effects wear off, the person feels tired and irritable and sometimes gets headaches

…uses an illegal, addictive drug several times a day in order to increase energy and stay alert. When the effects wear off, the person feels tired and irritable and sometimes gets headaches

Salomon – Disability weights - 48

Framing experiment: caffeine addiction

Alternative descriptions for caffeine addictionDisability weights

…drinks several cups of coffee a day in order to increase energy and stay alert. When the effects wear off, the person feels tired and irritable and sometimes gets headaches

0.018

…takes medication several times a day in order to increase energy and stay alert. When the effects wear off, the person feels tired and irritable and sometimes gets headaches

0.064

…uses an addictive substance several times a day in order to increase energy and stay alert. When the effects wear off, the person feels tired and irritable and sometimes gets headaches

0.067

…uses an addictive drug several times a day in order to increase energy and stay alert. When the effects wear off, the person feels tired and irritable and sometimes gets headaches

0.136

…uses an illegal, addictive drug several times a day in order to increase energy and stay alert. When the effects wear off, the person feels tired and irritable and sometimes gets headaches

0.198

Salomon – Disability weights - 49

…takes daily medication and has difficulty going without it. When the effects wear off, the person feels severe nausea, agitation, vomiting and fever. The person has a lot of difficulty in daily activities.

…uses heroin daily and has difficulty controlling the habit. When the effects wear off, the person feels severe nausea, agitation, vomiting and fever. The person has a lot of difficulty in daily activities.

Framing challenge: drug use disorders

Salomon – Disability weights - 50

…takes daily medication and has difficulty going without it. When the effects wear off, the person feels severe nausea, agitation, vomiting and fever. The person has a lot of difficulty in daily activities.

…uses heroin daily and has difficulty controlling the habit. When the effects wear off, the person feels severe nausea, agitation, vomiting and fever. The person has a lot of difficulty in daily activities.

Framing challenge: drug use disorders

Salomon – Disability weights - 51

• Background• Study design and methods• Key findings• Interpretations, limitations, conclusions

Disability Weights Measurement Study

Salomon – Disability weights - 52

Interpretations

Largest empirical effort to date to measure weights for health outcomes across range of populations shows:• Feasible to collect this sort of information in virtually any population• Simple data collection tools can be combined with straightforward analytic

techniques to yield meaningful weights• Weights appear highly consistent across diverse cultural settings and

respondent characteristics

New disability weights provide critical resource for assessment of burden of disease, healthy life expectancy and intervention cost-effectiveness

Salomon – Disability weights - 53

Limitations

• Selection of countries to provide diversity, not as random sample of world’s population

• Web survey heavily skewed toward North America, Australia, Western Europe, with few respondents from Africa or Middle East

• Responses depend on validity of lay descriptions Are some key consequences omitted? Does inclusion of labels in selected instances (e.g. drug use disorders)

bias disability weights for these conditions?

• Future work should prioritize Exploring use of standardized health-state classifications More empirical data on distribution of sequelae across health-states

Salomon – Disability weights - 54

Lancet 2012; 380: 2129-2143.