REVIEW ARTICLES A systematic review identifies valid comorbidity indices derived from administrative health data Marko Yurkovich a,b , J. Antonio Avina-Zubieta a,b , Jamie Thomas c , Mike Gorenchtein d , Diane Lacaille a,b, * a Division of Rheumatology, Department of Medicine, University of British Columbia, Vancouver, British Columbia, Canada b Milan Ilich Arthritis Research Centre, 5591 No. 3 Rd, Richmond, British Columbia, Canada V6X 2C7 c Faculty of Pharmaceutical Sciences, University of British Columbia, Vancouver, British Columbia, Canada d Faculty of Medicine, University of Limerick, Castletroy, Co. Limerick, Ireland Accepted 3 September 2014; Published online 31 October 2014 Abstract Objectives: To conduct a systematic review of studies reporting on the development or validation of comorbidity indices using admin- istrative health data and compare their ability to predict outcomes related to comorbidity (ie, construct validity). Study Design and Setting: We conducted a comprehensive literature search of MEDLINE and EMBASE, until September 2012. After title and abstract screen, relevant articles were selected for review by two independent investigators. Predictive validity and model fit were measured using c-statistic for dichotomous outcomes and R 2 for continuous outcomes. Results: Our review includes 76 articles. Two categories of comorbidity indices were identified: those identifying comorbidities based on diagnoses, using International Classification of Disease codes from hospitalization or outpatient data, and based on medications, using pharmacy data. The ability of indices studied to predict morbidity-related outcomes ranged from poor (C statistic 0.69) to excellent (C statistic O0.80) depending on the specific index, outcome measured, and study population. Diagnosis-based measures, particularly the Elix- hauser Index and the Romano adaptation of the Charlson Index, resulted in higher ability to predict mortality outcomes. Medication-based indices, such as the Chronic Disease Score, demonstrated better performance for predicting health care utilization. Conclusion: A number of valid comorbidity indices derived from administrative data are available. Selection of an appropriate index should take into account the type of data available, study population, and specific outcome of interest. Ó 2015 Elsevier Inc. All rights reserved. Keywords: Systematic review; Comorbidity; Multimorbidity; Administrative data; Claims data; Mortality; Health care utilization 1. Introduction Administrative databases are being increasingly used for research purposes. They play an important role in epi- demiologic, quality of care, pharmacovigilance, and out- come studies. These databases provide complementary information to randomized controlled trials because of their real-life setting, large samples, long follow-up duration, and their ability to provide population-based samples, free of selection bias. These data, however, have some limita- tions including lack of clinical, lifestyle, and demographic data and because of the observational nature, which can introduce biases. These biases include selection and chan- neling bias, as well as confounding by indication. These limitations can be minimized by careful adjustment in sta- tistical analyses. In observational studies, the outcomes of interest are often influenced by concurrent or preexisting comorbid- ities. Comorbidity may be defined as the total burden of ill- nesses unrelated to the principal diagnosis [1]. It is important to adequately adjust for comorbidities in studies in which comorbidities could act as confounders. Given the Conflict of interest: None. Funding: This research was funded by a peer reviewed grant from the Canadian Arthritis Network (11-01-RIPP-02). At the time of research, M.Y. was supported by a summer studentship from the Canadian Rheuma- tology Association. D.L. holds the Mary Pack Chair in Arthritis Research, funded by The Arthritis Society and the University of British Columbia. * Corresponding author. Milan Ilich Arthritis Research Centre, 5591 No. 3 Rd, Richmond, British Columbia, Canada V6X 2C7. Tel.: 604- 207-4019; fax: 604-207-4059. E-mail address: [email protected](D. Lacaille). http://dx.doi.org/10.1016/j.jclinepi.2014.09.010 0895-4356/Ó 2015 Elsevier Inc. All rights reserved. Journal of Clinical Epidemiology 68 (2015) 3e14
12
Embed
A systematic review identifies valid comorbidity indices ...€¦ · REVIEWARTICLES A systematic review identifies valid comorbidity indices derived from administrative health data
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Journal of Clinical Epidemiology 68 (2015) 3e14
REVIEWARTICLES
A systematic review identifies valid comorbidity indices derivedfrom administrative health data
Marko Yurkovicha,b, J. Antonio Avina-Zubietaa,b, Jamie Thomasc, Mike Gorenchteind,Diane Lacaillea,b,*
aDivision of Rheumatology, Department of Medicine, University of British Columbia, Vancouver, British Columbia, CanadabMilan Ilich Arthritis Research Centre, 5591 No. 3 Rd, Richmond, British Columbia, Canada V6X 2C7
cFaculty of Pharmaceutical Sciences, University of British Columbia, Vancouver, British Columbia, CanadadFaculty of Medicine, University of Limerick, Castletroy, Co. Limerick, Ireland
Accepted 3 September 2014; Published online 31 October 2014
Abstract
Objectives: To conduct a systematic review of studies reporting on the development or validation of comorbidity indices using admin-istrative health data and compare their ability to predict outcomes related to comorbidity (ie, construct validity).
Study Design and Setting: We conducted a comprehensive literature search of MEDLINE and EMBASE, until September 2012. Aftertitle and abstract screen, relevant articles were selected for review by two independent investigators. Predictive validity and model fit weremeasured using c-statistic for dichotomous outcomes and R2 for continuous outcomes.
Results: Our review includes 76 articles. Two categories of comorbidity indices were identified: those identifying comorbidities basedon diagnoses, using International Classification of Disease codes from hospitalization or outpatient data, and based on medications, usingpharmacy data. The ability of indices studied to predict morbidity-related outcomes ranged from poor (C statistic �0.69) to excellent (CstatisticO0.80) depending on the specific index, outcome measured, and study population. Diagnosis-based measures, particularly the Elix-hauser Index and the Romano adaptation of the Charlson Index, resulted in higher ability to predict mortality outcomes. Medication-basedindices, such as the Chronic Disease Score, demonstrated better performance for predicting health care utilization.
Conclusion: A number of valid comorbidity indices derived from administrative data are available. Selection of an appropriate indexshould take into account the type of data available, study population, and specific outcome of interest. � 2015 Elsevier Inc. All rightsreserved.
Keywords: Systematic review; Comorbidity; Multimorbidity; Administrative data; Claims data; Mortality; Health care utilization
1. Introduction
Administrative databases are being increasingly used forresearch purposes. They play an important role in epi-demiologic, quality of care, pharmacovigilance, and out-come studies. These databases provide complementary
Conflict of interest: None.
Funding: This research was funded by a peer reviewed grant from the
Canadian Arthritis Network (11-01-RIPP-02). At the time of research,
M.Y. was supported by a summer studentship from the Canadian Rheuma-
tology Association. D.L. holds the Mary Pack Chair in Arthritis Research,
funded by The Arthritis Society and the University of British Columbia.
* Corresponding author. Milan Ilich Arthritis Research Centre, 5591
0895-4356/� 2015 Elsevier Inc. All rights reserved.
information to randomized controlled trials because of theirreal-life setting, large samples, long follow-up duration,and their ability to provide population-based samples, freeof selection bias. These data, however, have some limita-tions including lack of clinical, lifestyle, and demographicdata and because of the observational nature, which canintroduce biases. These biases include selection and chan-neling bias, as well as confounding by indication. Theselimitations can be minimized by careful adjustment in sta-tistical analyses.
In observational studies, the outcomes of interest areoften influenced by concurrent or preexisting comorbid-ities. Comorbidity may be defined as the total burden of ill-nesses unrelated to the principal diagnosis [1]. It isimportant to adequately adjust for comorbidities in studiesin which comorbidities could act as confounders. Given the
� A number of comorbidity indices are available foruse in studies with administrative health data, inorder to control for the overall burden ofcomorbidities.
� To guide researchers and health policy makers inselecting the index most appropriate for their pur-pose, this systematic review describes the concep-tual and methodological differences among thevarious indices and compares their ability to pre-dict outcomes related to comorbidity (i.e. constructvalidity).
� The review reveals that a number of comorbidityindices demonstrate validity in predictingmortality.
� A diagnosis-based index, such as the Quan- or vanWalraven- EI or Romano-CCI, is recommended instudies where the outcome of interest is mortality.
� For studies evaluating healthcare utilization, wheremedication data is available, a medication-basedindex, such as the RxRisk-V, is recommended.
4 M. Yurkovich et al. / Journal of C
large number of comorbidities that may be relevant to agiven outcome, controlling for individual comorbiditiesmay not be practical for methodological reasons, includingloss of power. It may also be necessary to control for theoverall burden of comorbidity, rather than the individualeffect of each comorbidity.
For that purpose, a number of comorbidity indices havebeen developed to measure and weigh the overall burden ofcomorbidities. Some of these instruments have been devel-oped exclusively for use with administrative data, such asthe Elixhauser Index (EI) [2], whereas others have beendeveloped in other contexts but adapted for use with admin-istrative data, such as the Charlson Comorbidity Index(CCI) [3]. These comorbidity indices have been widelyused in studies using administrative data to control forthe overall burden of comorbidities.
However, given the large number of indices available inthe literature and the conceptual and methodological differ-ences among them, researchers and health policy makerswishing to control for comorbidity need guidance in select-ing the index most appropriate for their specific study.Although previous studies have compared the validity ofcomorbidity indices, they were limited by not systematicallyreviewing all indices available or by not explaining theconceptual and methodological differences between indices[4e6]. Our systematic review will guide scientists’ choiceby reviewing all the indices available, explaining their con-ceptual andmethodological differences, and comparing their
construct validity. Because there is no ‘‘gold standard’’ incomorbidity measurement, indices are often validated bymeasuring how well they are able to predict outcomesrelated to comorbidity, such as mortality or health care utili-zation (ie, construct validity) [7e9].
Accordingly, our aim was to conduct a systematic re-view with the following objectives: (1) to identify thedifferent instruments used in administrative data studiesto measure comorbidity, (2) to compare the instruments atthe conceptual level, that is, to describe how each indexwas developed and/or adapted for use with administrativedata and what concept the index aimed to measure, and(3) to evaluate and compare their ability to predictcomorbidity-related outcomes.
2. Methods
2.1. Search strategy
A methodological literature search was conducted as ofSeptember 2012, using theOvidplatform to searchMEDLINE(MEDLINE(R) In-Process & Other Non-Indexed Citations,Ovid MEDLINE(R) Daily, and Ovid OLDMEDLINE(R)from 1946) and EMBASE (from 1980). The year limits weredictated by the scope of the databases. We searched for andcombinedwith theBooleanoperator ‘‘OR’’ all relevant subjectheadings, using the ‘‘explosion’’ function where needed, andkeywords in titles and abstracts for the two concepts: ‘‘Admin-istrative data’’ and ‘‘Comorbidity index.’’ We combined thesetwo conceptswith theBoolean operator ‘‘AND.’’We excludedarticles that were solely abstracts, comments, conferenceproceedings, editorials, letters, or news. We included only ar-ticles published in English. The titles and abstracts of thearticles identified by this search were screened by one investi-gator (M.Y.) and selected for full-text review if relevant toour objectives. From this initial screen, a list of comorbidityindices potentially used in administrative data was identified.To ensure that we captured all relevant indices and theircorresponding validation studies, an additional literaturesearchwas performed using the same databases. This involvedsearching titles and abstracts for specific index names. Thesame screening process was applied to select articles poten-tially relevant to our objectives.
2.2. Study selection
Full-length articles of studies identified as potentially rele-vant to our objectives were independently reviewed by twoauthors (M.Y. and J.T.) to determine if they met the prespeci-fied inclusion criteria. Disagreements were settled byconsensus. For inclusion, studies had to have developed orvalidated a comorbidity index for use with administrativedata. Of note, we only included studies that related specif-ically to comorbidity indices and excluded studies thatfocused on the development or adaptation of risk scores orother groupers for risk adjustment. Adaptation of an index
5M. Yurkovich et al. / Journal of Clinical Epidemiology 68 (2015) 3e14
initially designed for use in a different context was permitted,such as adaptations of the CCI, which was initially designedfor medical chart review. Validation studies could be prospec-tive or retrospective and could include any patient population(adult or pediatric). We defined a validation study as one thatevaluated the ability of a comorbidity index to predict a spe-cific outcome (ie, construct validity) in a given populationsample. This could be achieved by reporting the C statistic(for dichotomous outcome variables) or the R2 (for linearoutcome variables). Alternatively, odds ratios (ORs), relativerisk, or hazard ratios (HR) (Cox) could be reported.
2.3. Data abstraction and reporting
Data were abstracted using a standardized data collec-tion form. Abstracted data included the comorbidity indicesevaluated, study population, type of administrative dataused to calculate the comorbidity score, outcome, and sta-tistics used to evaluate predictive ability. For index devel-opment studies, we collected information on the studypopulation, type of data used, and information on howthe index was developed or adapted.
For validation studies, we report on construct validity bypresenting results of the ability of indices to predict out-comes related to comorbidity, which we have labeled predic-tive ability. For dichotomous outcomes, such as mortality,the predictive ability was reported using the area under thecurve in a receiver operating characteristic curve, which isequivalent to the C statistic, a measure of model fit. The Cstatistic ranges from no predictive ability (when equaling0.50) to perfect prediction (when equaling 1.0). Consistentwith recommended guidelines [10], we considered C statis-tics of 0.7e0.8 as acceptable and �0.8 as excellent. For
Fig. 1. Flow chart of studies
continuous outcomes in linear regression models, the predic-tive ability was measured using the R2 value, which repre-sents the improvement in explained variance obtained byadding the comorbidity score to a baseline model. R2 valuesrange from 0 to 1, where 1 indicates that all the observedvariance in the outcome is explained by the model.
3. Results
3.1. Study selection
The primary literature search revealed 565 citations fortitle and abstract review. The second search, to identifyvalidation studies of the indices identified in the first search,identified 390 additional articles. Thus, a total of 955 arti-cles were selected for title and abstract review (Fig. 1). Ofthese, 37 were duplicates and 713 were not relevant to thestudy objectives, leaving 205 articles for full-text review.Of those, 18 studies did not involve the development orvalidation of an index, 55 involved an index not usingadministrative data, and 56 discussed a risk score ratherthan a comorbidity index. Therefore, these 112 studies wereexcluded, and a total of 76 articles were included in thefinal review.
Comorbidity indices identified were categorized intotwo groups: (1) those based on diagnoses from administra-tive data, using International Classification of Disease,Ninth or Tenth revision diagnostic coding system (ICD-9or ICD-10) and (2) those based on medications, using pre-scription data to identify comorbid conditions. The 76 arti-cles included 39 studies of diagnosis-based indices,including 35 related to the CCI and its adaptations, two spe-cifically to the EI, and two reporting study-specific
included in the review.
6 M. Yurkovich et al. / Journal of Clinical Epidemiology 68 (2015) 3e14
diagnosis-based indices. An additional 13 studies investi-gated medication-based indices such as the Chronic Dis-ease Score (CDS) and the RxRisk. The remaining 24articles compared the main indices identified.
3.2. Diagnosis-based indices
3.2.1. Charlson comorbidity indexThe CCI was created by Charlson et al. [3] in 1987. It
was developed using chart review to predict 1-year mortal-ity in a cohort of 604 patients admitted to a medical serviceat New York Hospital during 1 month in 1984. The CCIwas then validated in the same study using a cohort of685 breast cancer patients admitted to a Connecticut teach-ing hospital from 1962 to 1969. The final index is a list of19 conditions, with each condition assigned a weight of 1,2, 3, or 6 based on adjusted HR for each comorbid condi-tion derived from Cox proportional hazards regressionmodels. A total score is calculated from the sum of theweighted scores [3].
The CCI is the most widely used comorbidity index andhas been validated in patient populations with various diag-noses or undergoing various surgical procedures [11e34].Numerous adaptations of the CCI have been developedfor use with ICD-9 or ICD-10 codes in administrative data-bases [11,14,15,35e39] as described in the following para-graph. For each adaptation, the study populations andprimary end points used for development, along with a listof comorbid conditions included, are summarized inTable 1. The results of validation studies are summarizedin Table 1/Appendix A at www.jclinepi.com.
3.2.1.1. Deyo CCI. In 1992, Deyo et al. [35] adapted theCCI by identifying the ICD-9 codes corresponding to the19 original comorbid conditions. The codes for leukemiaand lymphoma were combined with the category ‘‘any ma-lignancy’’ leaving the Deyo CCI as a list of 17 comorbidconditions [35] (Table 1).
Eight studies have specifically evaluated the ability of theDeyo CCI to predict various outcomes [12,13,19,21,26,27,31,34,40,41]. The Deyo CCI’s ability to predict mortalityranged from poor to excellent, with C statistics ranging from0.64 to 0.86 for in-hospital mortality and 0.59e0.85 for 1-year mortality. A number of studies demonstrated that otherindices or risk scores performed better than the Deyo CCI inpredicting mortality [12,13,31,40,41], length of stay (LOS)[12,31], or costs [27]. The Deyo CCI demonstrated betterability to predict 1-year mortality when both prior inpatientand outpatient data were used to calculate the index [19]. In2004, the Deyo CCI was modified for use with ICD-10 codes,which performed similarly to the original ICD-9 version inpredicting in-hospital mortality [21].
3.2.1.2. Romano CCI. The Romano CCI, originallyknown as the Dartmouth-Manitoba CCI, adapted the CCIfor use with administrative data. The identification of
corresponding International Classification of Disease,Ninth Revision, Clinical Modification (ICD-9-CM) codeswas first done by Roos et al. [42] in 1989 and was subse-quently revised and modified by Romano et al. [36,37] in1993 to become what is known as the Romano CCI.Compared with the Deyo CCI, the Romano adaptation in-cludes broader definitions, encompassing more ICD-9-CMcodes, for peripheral vascular diseases, complicateddiabetes, and malignancy. Romano et al. evaluated theRomano and Deyo versions of the CCI in patients who un-derwent coronary artery bypass grafting (CABG) surgery inManitoba and lumbar discectomy in California (Table 1).Although no direct comparison of the two adaptationswas tested, both demonstrated similar ability to predict out-comes related to comorbidity, when evaluated in the samepopulation and using the same outcome. However, the riskestimate for each comorbidity varied widely across popula-tions. Accordingly, Romano et al. [36] recommended thatinvestigators use data from their own study population toreestimate the weights assigned to each comorbidity.
Three studies have modified and/or evaluated theRomano CCI for its ability to predict various outcomes[16,22,30]. Roos et al. created an augmented version ofthe Romano CCI, which demonstrated improved predictiveability compared with the original (Table 1/Appendix A atwww.jclinepi.com). However, the authors cautioned thatthis augmented index may factor in complications resultingin an overestimation of comorbidity [16]. In regressionanalyses, the Romano CCI was a poor predictor of postop-erative change in health-related quality of life scores [22].Romano CCI performed slightly lower than the other twoinstruments but still demonstrated acceptable predictiveability for 1-year mortality [30].
Three studies directly compared the Romano and Deyoadaptations of the CCI [14,17,26]. The scores derived fromboth adaptations in each study demonstrated substantial oralmost perfect agreement, indicating that the two comor-bidity classifications are similar [14]. Both adaptationsdemonstrated similar ability to predict mortality (Table 1/Appendix A at www.jclinepi.com); however, the Romanomethod was slightly superior for predicting mortality, andmodels with study-derived weights outperformed Charlsonweighted models [17]. ICD-10 adaptations of both theDeyo and Romano CCIs demonstrated acceptable predic-tive ability for 1-year mortality, with the Romano perform-ing slightly better [26].
3.2.1.3. D’Hoore CCI. D’Hoore et al. created a CCI adap-tation using only the first three digits of ICD-9 codingwithout CM, as many institutions outside the United Statesuse ICD-9 codes without CM (which includes proceduralcodes and additional morbidity details). Because codingof the tailing digits in ICD-9 codes can lead to inconsis-tencies, they claim to have created a simpler and morereliable adaptation [11]. The D’Hoore index demonstratedexcellent ability to predict in-hospital mortality in
Calgary Health RegionHospitalization data.ICD-9-CM (2001e2002) and ICD-10-CA (2002e2003)codes,c up to 16diagnoses.a ICD-10-CA, up to 25diagnoses and 20procedures.b
Outcomepredicted
Postoperative mortality(in-hospital or 6 wkafter discharge),postoperativecomplications, LOS,hospital charges
Identification of ICD-9-CMcodes for all 19CCIconditions; leukemiaand lymphomacombined with othermalignancies; use oforiginal CCI weights.
Use of the samecomorbid conditionsand weights as DeyoCCI; with broaderdefinitions for PVD,complicateddiabetes, andmalignancy.
Identification of ICD-9codes without CM forall 19 conditions oforiginal CCI; use oforiginal CCI weights.
Use of Deyo’s ICD codesincludes only the 5comorbiditiesassociated withmortality in theirstudy population;assigned study-derived weights.
Identification of ICD-10codes and enhancedICD-9-CM codesfor Deyo CCIcomorbidities; use oforiginal CCI weights.a
Assigned study-derivedweights to originalQuan CCI; includesonly 12 comorbiditiesassociated withmortality.b
Original CCI Deyo andRomano comorbid
conditions
D’Hoorecomorbidconditions
Ghali Quan
Comorbid conditionsincluded Weight
Comorbidconditions
Newweight
Comorbidconditions New weight
Myocardial infarct 1 X X X 1d Xa db 0Congestive heart failure 1 X X X 4 X X 2Peripheral vasculardisease
1 X X X 2 X d 0
Cerebrovascular disease 1 X X X 1 X d 0Dementia 1 X X d 0 X X 2Chronic pulmonarydisease
1 X X d 0 X X 1
Connective tissuedisease
1 X X d 0 X X 1
Ulcer disease 1 X X d 0 X d 0Mild liver disease 1 X X d 0 X X 2Diabetes 1 X X d 0 X d 0Hemiplegia 2 X X d 0 X X 2Moderate or severe renaldisease
2 X X X 3 X X 1
Diabetes with end-organdamage
2 X X d 0 X X 1
(Continued )
7M. Yurkovich et al. / Journal of Clinical Epidemiology 68 (2015) 3e14
Table 1. Continued
Original CCI Deyo andRomano comorbid
conditions
D’Hoorecomorbidconditions
Ghali Quan
Comorbid conditionsincluded Weight
Comorbidconditions
Newweight
Comorbidconditions New weight
Any tumor 2 X X d 0 X X 2Leukemia 2 d X d d d d d
Lymphoma 2 d X d d d d d
Moderate or severe liverdisease
3 X X d 0 X X 4
Metastatic solid tumor 6 X X d 0 X X 6AIDS 6 X X d 0 X X 4
Abbreviations: CCI, Charlson Comorbidity Index; CABG, coronary artery bypass grafting; CHF, congestive heart failure; ICD-9-CM, InternationalClassification of Disease, Ninth Revision, Clinical Modification; ICD-10-CA, International Classification of Disease and Related Health Problems,10th Revision, Canada; LOS, length of stay; PVD, peripheral vascular disease; AIDS, acquired immunodeficiency syndrome; MI, myocardialinfarction.
a Indicates original Quan CCI developed in 2005 [38].b Indicates updated Quan CCI developed in 2011 [39].c The Calgary Health Region has coded diagnostic data using ICD-10-CA since April 1, 2002.d Ghali divided MI into old and new MI, only new MI was found to be associated with in-hospital mortality in the study population.
8 M. Yurkovich et al. / Journal of Clinical Epidemiology 68 (2015) 3e14
populations with a principal diagnosis of myocardial infarc-tion, ischemic heart disease, and bacterial pneumonia butdemonstrated poor discrimination in stroke and congestiveheart failure populations [15]. The APACHE II scoreshowed better ability to predict in-hospital mortality thanD’Hoore CCI in a cohort of intensive care unit patients[25].
3.2.1.4. Ghali CCI. Using Deyo’s coding scheme, Ghaliet al. [14] created a study-specific index with a reducednumber of comorbidities, by including only comorbiditiesfound to be associated with in-hospital mortality(OR O 1.2), and with study-specific weights for each co-morbidity, derived from multiple logistic regression ana-lyses on the study sample used for index development.The Ghali CCI includes only five comorbidities. Whentested on the same sample, it performed better than theDeyo CCI in predicting in-hospital mortality. However,the performance of the Ghali CCI improved further whenthe coefficients from the original CCI were used insteadof the study-specific weights.
3.2.1.5. Quan CCI. In 2005, Quan et al. [38] identified theICD-10 codes corresponding to the Deyo CCI coding algo-rithm and also expanded the selection of codes for eachcomorbidity, using physicians to assess the face validityof the selected ICD-10 codes.
In 2007, Sundararajan et al. [21] compared the QuanCCI with two ICD-10 adaptations: one developed previ-ously by Sundararajan et al. [21] and one developed byHalfon et al. [43], which is not included in our reviewbecause the adaptation was not validated. In cohorts fromfour countries, the Quan CCI was a better predictor ofin-hospital mortality than the Sundararajan and Halfon ad-aptations [28]. Hanley et al. [32] compared the ability ofthe Quan CCI and adjusted clinical groups (ACGs) to pre-dict medication use and found that the ACGs predicted
better. Of note, ACGs use ICD-9 or ICD-10 codes todevelop a composite measure of patient illness burden, esti-mated from the mix of conditions experienced for a definedinterval.
In 2011, Quan et al. [14,39] created an updated version oftheir index and derived study-specific weights in a similarmethod to Ghali’s. The updated index includes 12 comorbid-ities with new weights assigned to each [Table 1] anddemonstrated slightly better ability to predict in-hospital,30-day, and 1-year mortality than the original Quan CCI[39] (Table 1/Appendix A at www.jclinepi.com).
3.2.1.6. CCI adaptations for administrative data. Sixstudies have developed or evaluated study-specific CCI ad-aptations for use with administrative data [18,20,23,24,29].Two study-specific CCIs demonstrated poor predictiveability for 30-day readmission [18,23]. Others demon-strated acceptable-to-excellent ability to predict mortality[23,24,33].
Martins and Blais developed a study-specific indexincluding 23 comorbidities, eight from the original Charl-son index and 13 identified as frequent comorbidities inthe study population. This index demonstrated superior per-formance for predicting in-hospital mortality comparedwith a CCI adaptation (specific adaptation not specified)[24].
Klabunde et al. created two new indices referred to asthe National Cancer Index (NCI)done using inpatientclaims and the other using outpatient claimsdby assigningweights to the comorbidities from the original Charlson in-dex based on a Cox proportional hazards model predicting2-year noncancer mortality. It was evaluated for its abilityto predict future treatment in both prostate and breast can-cer populations, demonstrating acceptable and excellentpredictive ability, respectively [20]. The index was revisedby combining the score from inpatient and outpatient dataand was further evaluated, compared with the original
9M. Yurkovich et al. / Journal of Clinical Epidemiology 68 (2015) 3e14
version, a uniform weights index and the CCI (specificadaptation not specified), for its ability to predict 2-yearnoncancer mortality in four cohorts with breast, prostate,colorectal, and lung cancer. The new NCI demonstratedsimilar predictive ability to the original NCI, and both ver-sions performed better than the CCI or the uniform weightindex in all four study populations [29] (Table 1/AppendixA at www.jclinepi.com).
3.2.1.7. Comparison of self-report vs. administrative data-derived CCI. Three studies compared the predictive abilityof CCI adaptations derived from self-reported data with thesame index derived from administrative data [44e46]. Twostudies [44,45] found that self-reported data and adminis-trative data adaptations had similar ability to predictvarious outcomes. Ronksley et al. [46] found that self-report of comorbid conditions had varying levels of agree-ment with those derived from administrative data, rangingfrom poor to substantial agreement depending on the co-morbid condition (k 5 0.14e0.79).
3.2.1.8. Comparison of chart review vs. administrativedata-derived CCI. In 2010, Leal and Laupland [47] con-ducted a systematic review of studies comparing CCI adap-tations derived from administrative data and chart review.They found that CCI scores calculated from administrativedata were consistently lower than those derived from chartreview, and agreement between the two sources was poor tofair (k ranging from 0.30 to 0.56) [47]. Two additionalstudies have compared chart review vs. administrativedata-derived CCIs [48,49]. One study, evaluating the QuanCCI, found that kappa agreement ranged greatly (from 0.02to 0.47) according to the comorbidity identified [44].Another study [49], evaluating a study-specific administra-tive data CCI using ICD-10 codes, found CCI scoresderived from the two sources to be well correlated(r 5 0.88, P ! 0.01) [49].
3.2.2. Elixhauser comorbidity indexElixhauser et al. developed a comorbidity index
comprised of a comprehensive set of 30 comorbiditiesdefined using ICD-9-CM codes from administrative data(Table 2/Appendix B at www.jclinepi.com). The EI comor-bidities were significant predictors of LOS and hospitalcharges. Many of the individual EI comorbidities wereassociated with in-hospital mortality, but as a group, the as-sociation was not significant [2]. One disadvantage of theoriginal EI is that it includes 30 dichotomous variables, rep-resenting each comorbidity, without a weighting system toprovide a single score.
Three studies validated the EI for its ability to predictmortality, with two providing a modification to the EI[38,50,51]. Additional studies have validated and comparedthe EI with other indices (Section 3.5). In predictingin-hospital mortality, all EI versions demonstratedacceptable-to-excellent predictive ability [38,50,51]. In
the Quan study, the enhanced ICD-9-CM version per-formed the best, followed by the ICD-10 version and theoriginal EI [38]. Van Walraven et al. developed a scoringsystem for the EI, using the regression coefficients for eachcomorbidity from a multivariate logistic regression modelpredicting in-hospital mortality. It demonstrated acceptableprediction of in-hospital mortality, with similar results formodels using the EI including all comorbidities and theEI including only the 21 comorbidities significantly associ-ated with mortality [51].
3.3. Other diagnosis-based indices
Two studies developed study-specific diagnosis-basedindices, which were not based on the EI or CCI [52,53]. Us-ing hospitalization and outpatient data from the Surveil-lance, Epidemiology, and End ResultsdMedicare linkeddatabase, Fleming et al. developed an index with 27 comor-bidity categories based on the prevalence of diseases spe-cifically in their study population, black males withprostate cancer [52]. Abildstrom et al. [53] created a co-morbidity index based on administrative data to predict30-day mortality after CABG. Their index performed simi-larly to the additive EuroSCORE registered in a clinicaldatabase.
3.4. Diagnosis-based indices comparison studies
Fourteen studies compared the EI to various CCI adap-tations [39,41,51,54e66]. A description of the cohortsand results for each study are found in Table 3/AppendixC at www.jclinepi.com. Overall, both the EI and CCIdemonstrated poor-to-excellent ability to predict variousoutcomes. When predicting in-hospital mortality, C statis-tics ranged from 0.632 to 0.878 and 0.608e0.860 for theEI and CCI, respectively. For 1-year mortality, resultsranged from 0.69 to 0.909 and 0.65e0.906, respectively.Nine studies [38,51,54,55,57,58,60,63,66] demonstratedthat various versions of the EI (including the original,Quan, and van Walraven adaptations) predicted mortalityoutcomes better than various adaptations of the CCI (Deyo,Romano, and Quan adaptations). In contrast, four studiesfound no difference [41,59,61,64]. Only one study foundthat a CCI adaptation (Romano CCI) predicted mortalitybetter than the EI [62]. Three studies examined the effectof using inpatient and/or outpatient data on predictingmortality and found that combining data from both sourcesresulted in higher C statistics [57,62,67]. Lieffers et al.augmented the EI by adding performance status andsubstituting clinical data instead of administrative recordsfor body weight. This augmented version of EI predicted2- and 3-year survival in patients with stages IIeIV colo-rectal cancer better than the Quan EI. Another study evalu-ated a combination of the van Walraven EI and the RomanoCCI and found the combined index predicted mortalitybetter than each individual index [64].
10 M. Yurkovich et al. / Journal of Clinical Epidemiology 68 (2015) 3e14
3.5. Medication-based indices
Medication-based indices use pharmacy data to identifycomorbidities by linking medications to specific diseasecategories. Fourteen studies examined the development orvalidation of these indices [68e80], as summarized inTable 2/Appendix B at www.jclinepi.com.
3.6. Chronic Disease Score
In 1992, von Korff et al. [68] created the CDS, usingmedications instead of diagnostic codes to identify comor-bidities. Using a population-based pharmacy database, apanel of experts evaluated patterns of use of selected med-ications to create disease categories, and weights were as-signed by consensus [68,76]. The original CDS included17 diseases and was validated against chart review andphysician rating of physical disease severity [68]. Its abilityto predict health outcomes was validated on two patientpopulations [68,69].
Clark et al. [70] modified the original CDS by updatingmedications, expanding the disease categories to 28, andweighting the disease categories based on regressionmodels. The CDS-2 ubiquitously replaced the originalCDS, and adaptations were subsequently made for applica-tion to specific populations, including the Pediatric ChronicDisease Score [71] and adaptations for diabetics by Joishet al. [77]. Other studies compared the predictive abilityof the CDS-2 based on outpatient pharmacy data comparedwith in-hospital prescription data [73], stratified based ontotal prescription number [72], and predicted surgical siteor nosocomial infections in hospitalized patient populations[73,78]. Fishman et al. [75] continued updating the CDS-2to include new medication classes and expand diseasecategories, ultimately developing a new but related instru-ment, RxRisk.
3.7. RxRisk and RxRisk-V
The RxRisk was developed as an all-age risk assessmentinstrument using outpatient pharmacy data to identifychronic diseases and predict future health care costs [75].The RxRisk included 57 adult and pediatric weighted dis-ease categories and associated drug classes. Validationwas based on a large general population sample using mul-tiple measures of predictive power. The RxRisk-V was asubsequent modification adapted to the Veterans HealthAdministration population [76].
3.8. Medication-Based Disease Burden Index
The final medication-based index is the Medication-Based Disease Burden Index (MDBI), developed as analternative to the original CDS to deal with the same issuesaddressed by Clark’s and Fishman’s revisions [79]. TheMDBI showed weak correlation with the CCI and CDS,moderate ability to predict 12-week death and readmission
[79], and a poorer ability to predict 6-month mortality thanthe RxRisk-V [80].
3.9. Cross-index comparison
Thirteen studies compared medication- and diagnosis-based indices [1,7,8,81e90]. Schneeweiss et al. conductedthree studies comparing medication- and diagnosis-basedindices for their ability to predict various outcomes (Table 4/Appendix A at www.jclinepi.com). Two studies found thefollowing performance ranking when predicting 1-yearmortality, long-term care admissions, and hospitalizations:Romano CCI � Deyo CCI O D’Hoore CCI O GhaliCCI O CDS-1 O CDS-2, but a different ranking whenpredicting physicianvisits or expenditures for physicianvisits:D’Hoore CCI O CDS-2 � Romano CCI O DeyoCCIO CDS-1O Ghali CCI [7,81]. Another study evaluateda study-specific adaptation of the RomanoCCI, which derivedits own study-specific weights and found that it predicted 1-year mortality better than the original Romano CCI and thatboth versions outperformed the CDS-1. However, the EIdemonstrated the best predictive ability of the four indicescompared [8].
Ten additional studies compared diagnosis- andmedication-based indices [1,82e90] (Table 4/Appendix Dat www.jclinepi.com). When predicting mortality out-comes, results were not consistent across studies. In onestudy [1], the predictive ability of the diagnosis-based index(Deyo CCI) was better than the medication-based index(CDS-1); yet, another study demonstrated the opposite[85] (RxRisk-V predicted mortality better than the DeyoCCI) and another found no difference between the sametwo indices [86]. Generally, medication-based indicesdemonstrated better ability to predict various health careutilization outcomes, including prescription medicationuse [87], total costs [84], disease burden [90], and hospital-izations [86]. However, the EI demonstrated better abilityto predict physician visits than the RxRisk-V [87]. Medica-tion- and diagnosis-based indices demonstrated similarability to predict hospital readmission and LOS [82], hospi-talization [1], spending [84], and costs [89].
4. Discussion
In this report, we summarize the published literature onthe development and validity of comorbidity indices used inadministrative data studies. The body of literature on thistopic is broad, as we identified a total of 76 primary articlesfor inclusion.
All indices identified could be grouped as either diagnosisbased, using ICD coding, or medication based, using phar-macy dispensing data. The main diagnosis-based indiceswere the EI and the various adaptations of the CCI for usewith administrative data. Medication-based indices includedversions of the CDS, which later became known as the
11M. Yurkovich et al. / Journal of Clinical Epidemiology 68 (2015) 3e14
RxRisk, and its adaptation for use in the veteran population,the RxRisk-V.
Of the diagnosis-based measures, we found that the EIconsistently outperformed the CCI in predicting both short-and long-term mortality. Of the main adaptations of theCCI, the Romano CCI demonstrated equal or better perfor-mance in its ability to predict various outcomes comparedwith the Deyo CCI, despite the fact that the Deyo CCI isthe more commonly used measure. Although both the EIand all administrative data adaptations of the CCI weredeveloped for use with inpatient hospitalization data,several studies found that using combination of both inpa-tient and outpatient data consistently improved the perfor-mance of the index studied. Furthermore, a number ofstudies examined adaptations of the CCI and EI, whichderived empirical weights based on the study-specific pop-ulation and outcome measure. Assigning study-specificweights for both EI and CCI adaptations tended to improvetheir predictive performance. Accordingly, for diagnosis-based indices, we recommend the use of the EI or theRomano CCI, particularly when predicting mortality out-comes. When available, we recommend calculating theseindices using a combination of both inpatient and outpatientdata and, when possible, deriving study-specific empiricallyderived weights for the index selected.
Of the medication-based indices, we found that the orig-inal version of the CDS developed by von Korff et al. [68]tended to outperform the CDS-2, developed by Clark et al.[70]. However, the later version known as the RxRisk-Vwas the most commonly used medication-based measureand demonstrated the best predictive ability. Thus, instudies using pharmacy dispensing data, we recommenduse of the RxRisk-V.
In studies comparing the predictive ability of indices, wefound that diagnosis-based measures were better predictorsof mortality outcomes than medication-based indices. Somestudies found that medication-based indices were betterpredictors of health care utilization and costs; however,other studies found that diagnosis-based measures werebetter at predicting such outcomes. Disadvantages ofdiagnosis-based indices include a wide variability in ICDcoding practices, underreporting of chronic conditions inthe secondary diagnosis fields, and difficulty in distinguish-ing between acute conditions present on admission fromsubsequent complications of care [6,82,91]. Furthermore,certain CCI adaptations can only be used with specificICD versions (eg, Deyo CCI with ICD-9-CM and the QuanCCI with ICD-10). Additionally, a number of country-specific ICD-10 versions exist [91], which may further limitthe application of the diagnosis-based indices. Hence, it isimportant, when choosing a CCI adaptation, to considerthe ICD version used in the administrative data of the studyand to select the CCI index accordingly. Pharmacy data arecredited as a more timely, complete, and reliable datasource than diagnosis-based data [76], but it is not readilyavailable in many jurisdictions. The major criticism of
medication-based indices lies in the back coding from pre-scription to diagnosis [7]. This limits the definition of co-morbidity to include only chronic diseases treated withprescription medication. Furthermore, medication-basedindices require continual updates to accommodate thedevelopment and reassignment of new medications for spe-cific indications.
We found that the predictive ability of comorbidityindices varied widely, ranging from poor (C statistic0.50e0.69) to good (C statistic 0.70e0.79) and excellent(C statistic O0.80), with similar variability observed whenR2 values were reported. Performance varied according tothe specific index, outcome measured, and study population.In 2000, Schneeweiss et al. [5] conducted the first review ofstudies evaluating the ability of comorbidity indices to pre-dict comorbidity-related outcomes, using administrativedata. They concluded that comorbidity indices using admin-istrative data provide only a modest improvement overadjustment for age alone [5]. In 2001, they conducted anextensive validation study of four adaptations of the CCIas well as the CDS-1 and CDS-2. They concluded that co-morbidity indices provide only a limited ability to controlfor confounding, acknowledging nonetheless their useful-ness because of their ease of use and time and resourcessavings [7]. In 2005, Needham et al. conducted a review of10 articles on CCI adaptations for administrative data witha specific emphasis on risk adjustment in critical careresearch. They found no difference in mortality predictionwhether using CCI derived from administrative data or chartreview and with the various adaptations of the CCI [6].
A recent systematic review has been publishedcomparing the predictive ability of various diagnosis-based indices using administrative data [4]. The authorsperformed a meta-analysis examining the indices’ perfor-mance in predicting short- and long-term mortality. Theirextensive comparative analysis resulted in findings similarto those in our study; specifically, the EI and RomanoCCI demonstrated significantly superior performance inpredicting mortality outcomes. Although our study and thatby Sharabiani et al. focus on comorbidity indices usingadministrative data, they examine different aspects of thetopic and use distinct approaches. Our study provides anextensive description of the various indices available,including how they were developed and results of their vali-dation studies, and outlines the differences between them.The aforementioned review only includes diagnosis-basedindices and reports only on studies comparing the predic-tive performance of various indices. Therefore, they didnot discuss medication-based indices or include studiescomparing medication- and diagnosis-based indices. Ourstudy offers complementary information valuable to re-searchers trying to understand the differences betweenavailable instruments. It will assist researchers in selectinga comorbidity index that best meets the needs of their spe-cific study, including when administrative data are availableon both diagnosis and medication information.
12 M. Yurkovich et al. / Journal of Clinical Epidemiology 68 (2015) 3e14
There are, however, limitations to our systematic review.First, we chose to focus specifically on comorbidity indicesusing administrative data and did not evaluate comorbidityindices using chart review or self-reported data. Predictivevalidity of CCI adaptations may differ with these data sour-ces. Second, although our search strategy was comprehen-sive, it is possible that studies were missed. However, giventhe consistency of our results, it is unlikely that missedstudies would significantly alter the main findings of ourreview. Finally, the number of published comparisonstudies between medication- and diagnosis-based indiceswas limited, and results were not entirely consistent acrossstudies.
5. Conclusion
Comorbidity indices are used to control for the overallburden of comorbidities in administrative data studies anddemonstrate validity in predicting mortality; however, theirability to fully adjust for confounding due to comorbiditymay be limited. We recommend using a diagnosis-basedindex, such as the Quan EI, van Walraven EI, or RomanoCCI, in studies in which the outcome of interest is mortal-ity. One must consider the ICD version used when selectinga specific index. For studies evaluating health care utiliza-tion, in which medication data are available, we recom-mend using a medication-based index, such as theRxRisk-V. Overall, the appropriate selection of a comorbid-ity index for use with administrative data should take intoaccount the type of data available, the study population,and the specific outcome of interest in the study.
Supplementary data
Supplementary data related to this article can be foundonline at http://dx.doi.org/10.1016/j.jclinepi.2014.09.010.