Disability weights measurement in the Global Burden of Disease Study 2010

Disability weights measurement in the Global Burden of Disease Study 2010

Joshua A Salomon

Harvard School of Public Health

Global Health Metrics and Evaluation Conference

June 18, 2013Supported by funding from the Bill and Melinda Gates Foundation

Salomon – Disability weights - 2 2

Global Burden of Disease Study

• Global Burden of Disease (GBD) Study aims to measure impact of disease and injury in terms of losses in population health

• GBD 1990 study launched in 1991, updated by WHO in 2000s• GBD 2010 study, undertaken from 2007-2012, provides first

comprehensive overhaul since 1996

• GBD quantifies • Magnitude of different health problems in units of disability-adjusted

life years (DALYs)• Overall population health in units of healthy life expectancy (HALE)


Disability weights

• Disability weights provide the bridge between mortality and non-fatal outcomes in DALYs and in healthy life expectancy

• To measure health impact of non-fatal outcomes, GBD needs weights for all unique sequelae, which capture the major health consequences of all of the causes in the study

• Disability weights quantify severity of outcomes as percentage reductions from perfect health, which are multiplied by years lived in each sequela to give years lived with disability (YLD) e.g., If weight for blindness were 0.20, then 5 years lived with

blindness would be equivalent to dying one year prematurely

Salomon – Disability weights - 4

Intervention 2:Prevent 1y of deafness for

2000 individuals

Intervention 1:Extend life by 1y in

1000 healthy individuals

Disability weights in the 1996 GBD revision

Person trade-off: which would you choose?

• Expert panel used ‘person trade-off’ to assign values to 22 indicator conditions


1

Disability weights

2 3 4 5 6 7

Class 1:• Vitiligo on face

Class 4:• Below-knee amputation• Deafness

Class 7:• Active psychosis• Quadriplegia



• These 22 conditions used as operational definitions of 7 disability classes


1

Disability weights

2 3 4 5 6 7

Rheumatoid arthritis cases

Average disability weight=0.2*0.07 + 0.4*0.18 + 0.4*0.30=0.21



• These 22 conditions used as operational definitions of 7 disability classes

• Remaining conditions allocated across classes to compute average weights


Disability weights measurement study goals

• Derive weights for all 220 health states capturing nonfatal outcomes from 291 disease and injury causes in GBD 2010

• Address criticisms of previous approaches by: Focusing on valuations from community respondents… … in a diverse range of settings … using suitable measurement methods

• Specific research aims Develop valid and reliable data collection tools for population-

based surveys Empirical examination of variation in weights


• Background• Study design and methods• Key findings• Interpretations, limitations, conclusions

Disability weights measurement in GBD 2010


Study components

• Population-based household surveys• Face-to-face interviews in Tanzania,

Bangladesh, Indonesia, Peru• Telephone interview in random sample of

US households• Focus on paired comparisons for 108

health-states• Key objectives include comparative analysis

across diverse settings and benchmarking Internet survey against community samples

• Open-access Internet surveys• Available in English, Spanish and Mandarin• Key objectives are to fill in gaps with

remaining sequelae and to anchor scale for paired comparison responses

Bangladesh

Tanzania


Web survey

Web survey included 16,328 respondents from 167 countries


Web survey included 16,328 respondents from 167 countries

Web survey

1 - 9

10 - 49

50 - 99

100 - 499

500+


Measurement methods: paired comparisons

• Primary mode of eliciting responses is paired comparison• Respondents hear (or read) two descriptions of hypothetical people,

each with a randomly selected condition• Respondents indicate which person is healthier

• Paired comparison questions chosen for relative ease of comprehension, administration and analysis• Literacy and numeracy not essential• Health comparisons not tied to external “calibrators” such as risk• Appealing intuitive basis and established strategies for analysis


Framing paired comparisons

• Basis for all comparison are lay descriptions of sequelae, which highlight major functional consequences and symptoms associated with each sequela

• Must be brief: restricted to <35 words based on pretest results• Must use simple, non-clinical vocabulary

• Prologue to paired comparison questions orients respondents to focus on functioning

A person’s health may limit how well parts of his body or his mind work. As a result, some people are not able to do all of the things in life that others may do, and some people are more severely limited than others.

I am going to ask you a series of questions about different health problems. In each question I will describe two different people … Imagine they have the same number of years left to live, and will experience the health problems that I describe for the rest of their lives. I will ask you to tell me which person you think is healthier overall, in terms of having fewer physical or mental limitations on what they can do in life…


Paired comparison example

The first person has vision problems that make it difficult to see and recognize faces of family or friends across a room.

The second person has severe back and leg pain, which causes difficulty dressing, sitting, standing, walking, and lifting things. The person sleeps poorly and feels worried.

Imagine that both people will have these problems for the rest of their lives. Who would you say is healthier overall, the first person or the second person?

• Respondents in household surveys each answered 15 of these questions, with pairs of sequelae drawn at random from the universe of 108 × 108 possible pairs


Analyzing paired comparison data

Bangladesh Tanzania• Simple ordering of

outcomes based on “winning” proportions, as in various websites using paired comparisons to rank large pools of competitors

http://kittenwar.com/



Bangladesh Tanzania

Won 78% of 511 battles Won 77% of 539 battles

Winningest

Lost 80% of 846 battles Lost 80% of 1872 battles

Losingest

• Simple ordering of outcomes based on “winning” proportions, as in various websites using paired comparisons to rank large pools of competitors

http://kittenwar.com/



Bangladesh Tanzania• Simple ordering of

outcomes based on “winning” proportions, as in various websites using paired comparisons to rank large pools of competitors

• More sophisticated algorithms incorporate information on the relative strength of the opponent (e.g. Elo rating system in Chess)

• Statistical approaches enable maximum likelihood estimation of underlying scores


Statistical modeling of paired comparisons data

• Conceptual foundation for statistical modeling of paired comparisons data comes from Thurstone (1927), through Luce (1959), McFadden (1974) and others

• Intuitively, a pair of health states that are similar in severity are likely to produce greater disagreement over which is healthier than a pair of states that are different in severity

• Consider a population in which: 70% of people think depression is worse than blindness 90% of people think blindness is worse than arthritis

• Reasonable to conclude that depression and blindness are nearer on some unobserved scale than blindness and arthritis


-4 -2 0 2 4 6 8

blindness

90%


-4 -2 0 2 4 6 8

depression blindness

90%70%


-4 -2 0 2 4 6 8

depression blindness

90%


-4 -2 0 2 4 6 8

depression blindnessarthritis

90%


-4 -2 0 2 4 6 8

depression blindnessarthritis


Statistical modeling of paired comparisons data

• Statistical model formalizes this intuition Each health state i has an unobserved actual health level Xi as perceived by an

individual rater The rater makes a choice in the paired comparison based on which of the

health states is regarded as ‘healthier’State 1 is chosen over State 2 if X1 > X2

• Thurstone model assumes that Xi is normally distributed, implying that the choice probabilities are also normal

• We can model response probabilities using probit regression Requires arbitrary identifying assumptions Assumes that each health state has the same variance around the disability

weight


From paired comparisons to disability weights

Analysis of paired comparisons

Analysis of “population health equivalence”

responses

Rescaling of paired comparison results

47





responses


• Pooled data from all household surveys, CATI and web survey

• Estimated probit model with separate variance for each survey





responses


• Some Web survey respondents randomly assigned to “population health equivalence” (PHE) questions, for subset of 30 sequelae

• PHE provides separate estimates for these sequelae that are anchored on (0,1) disability scale

49





responses


• Probit scale mapped to (0,1) disability scale by regressing probit coefficients on PHE anchors

• Non-constant variance accommodated by regression on logit-transformed PHE values, then back-transforming values into (0,1) space

50



Disability weights measurement in GBD 2010


Results: paired comparison responses

• Probabilities of responses on paired comparisons (‘Who is healthier?’) summarized in heat maps.

Best Worst

Best

Worst

First health state in pair

Second health state in pair



High agreement in choices between very healthy vs. unhealthy outcomes (>90%)

… or vice versa (<10%)

Split responses for similar outcomes (~50%)



Measurement error in paired comparisons

• In the household survey, we assessed test-retest reliability by randomly assigning 20% of respondents to have the same pair repeated as 1st and 15th comparison

• Consistency of responses ranged between 60% and 70%

• Consistency of 2 coin flips would be 50%

• Kappa values range from 20% to 40%

Percent of respondents with consistent retest responses

36



High agreement in choices between very healthy vs. unhealthy outcomes (>90%)

… or vice versa (<10%)

Split responses for similar outcomes (~50%)



Results: paired comparisons across surveys


Results: probit values across surveys


Results: comparison of household and web surveys

• Web respondents comprise non-random, highly educated, self-selected sample

Educational attainment in HH & web samples



Tanzania (N=2,613)

Web (N=3,417)


• But, response probabilities are virtually indistinguishable from those in household surveys




• But, response probabilities are virtually indistinguishable from those in household surveys

• And estimated weights from probit regressions are very highly correlated


Results: new disability weights










Results: comparison to previous disability weights


Framing challenge: drug use disorders

…uses heroin daily and has difficulty controlling the habit. When the effects wear off, the person feels severe nausea, agitation, vomiting and fever. The person has a lot of difficulty in daily activities.





Framing experiment: caffeine addiction

Alternative descriptions for caffeine addiction

…drinks several cups of coffee a day in order to increase energy and stay alert. When the effects wear off, the person feels tired and irritable and sometimes gets headaches

…takes medication several times a day in order to increase energy and stay alert. When the effects wear off, the person feels tired and irritable and sometimes gets headaches

…uses an addictive substance several times a day in order to increase energy and stay alert. When the effects wear off, the person feels tired and irritable and sometimes gets headaches

…uses an addictive drug several times a day in order to increase energy and stay alert. When the effects wear off, the person feels tired and irritable and sometimes gets headaches

…uses an illegal, addictive drug several times a day in order to increase energy and stay alert. When the effects wear off, the person feels tired and irritable and sometimes gets headaches


Framing experiment: caffeine addiction

Alternative descriptions for caffeine addictionDisability weights

…drinks several cups of coffee a day in order to increase energy and stay alert. When the effects wear off, the person feels tired and irritable and sometimes gets headaches

0.018

…takes medication several times a day in order to increase energy and stay alert. When the effects wear off, the person feels tired and irritable and sometimes gets headaches

0.064

…uses an addictive substance several times a day in order to increase energy and stay alert. When the effects wear off, the person feels tired and irritable and sometimes gets headaches

0.067

…uses an addictive drug several times a day in order to increase energy and stay alert. When the effects wear off, the person feels tired and irritable and sometimes gets headaches

0.136

…uses an illegal, addictive drug several times a day in order to increase energy and stay alert. When the effects wear off, the person feels tired and irritable and sometimes gets headaches

0.198


…takes daily medication and has difficulty going without it. When the effects wear off, the person feels severe nausea, agitation, vomiting and fever. The person has a lot of difficulty in daily activities.




…takes daily medication and has difficulty going without it. When the effects wear off, the person feels severe nausea, agitation, vomiting and fever. The person has a lot of difficulty in daily activities.





Disability Weights Measurement Study


Interpretations

Largest empirical effort to date to measure weights for health outcomes across range of populations shows:• Feasible to collect this sort of information in virtually any population• Simple data collection tools can be combined with straightforward analytic

techniques to yield meaningful weights• Weights appear highly consistent across diverse cultural settings and

respondent characteristics

New disability weights provide critical resource for assessment of burden of disease, healthy life expectancy and intervention cost-effectiveness


Limitations

• Selection of countries to provide diversity, not as random sample of world’s population

• Web survey heavily skewed toward North America, Australia, Western Europe, with few respondents from Africa or Middle East

• Responses depend on validity of lay descriptions Are some key consequences omitted? Does inclusion of labels in selected instances (e.g. drug use disorders)

bias disability weights for these conditions?

• Future work should prioritize Exploring use of standardized health-state classifications More empirical data on distribution of sequelae across health-states


Lancet 2012; 380: 2129-2143.

Disability weights measurement in the Global Burden of Disease Study 2010

Health & Medicine

salomon disability weights

study disability weights

disability classes

units of disability

disability yld

average weights

population health gbd

health states