Using information systems to measure and improve quality

International Journal of Medical Informatics 53 (1999) 115–124

Using information systems to measure and improve quality

David W. Bates a,b,*, Elizabeth Pappius a, Gilad J. Kuperman a, Dean Sittig a,Helen Burstin b, David Fairchild b, Troyen A. Brennan b, Jonathan M. Teich a,b

a Partners Information Systems, Boston, MA 02115, USAb Brigham and Women’s Hospital, Boston, MA, USA

Abstract

Information systems (IS) are increasingly important for measuring and improving quality. In this paper, wedescribe our integrated delivery system’s plan for and experiences with measuring and improving quality using IS. Ourbelief is that for quality measurement to be practical, it must be integrated with the routine provision of care andwhenever possible should be done using IS. Thus, at one hospital, we now perform almost all quality measurementusing IS. We are also building a clinical data warehouse, which will serve as a repository for quality informationacross the network. However, IS are not only useful for measuring care, but also represent powerful tools forimproving care using decision support. Specific areas in which we have already seen significant benefit includereducing the unnecessary use of laboratory testing, reporting important abnormalities to key providers rapidly,prevention and detection of adverse drug events, initiatives to change prescribing patterns to reduce drug costs andmaking critical pathways available to providers. Our next major effort will be introduce computerized guidelines ona more widespread basis, which will be challenging. However, the advent of managed care in the US has producedstrong incentives to provide high quality care at low cost and our perspective is that only with better IS than existtoday will this be possible without compromising quality. Such systems make feasible implementation of qualitymeasurement, care improvement and cost reduction initiatives on a scale which could not previously be considered.© 1999 Elsevier Science Ireland Ltd. All rights reserved.

Keywords: Quality of care; Measurement; Improvement; Decision support; Integrated delivery system

1. Introduction

Health care costs are rising and all partiesinvolved—government, insurers, hospitalsand patients—are concerned. Costs must be

reduced, but without major compromise ofquality. However, quality measurement inhealthcare has been an elusive goal and thecurrent routine practice of quality measure-ment in healthcare is relatively primitive.Measuring quality without automated tools istime-consuming and labor-intensive, yet the* Corresponding author.

1386-5056/99/$ - see front matter © 1999 Elsevier Science Ireland Ltd. All rights reserved.

PII: S 1 3 8 6 -5056 (98 )00152 -X

D.W. Bates et al. / International Journal of Medical Informatics 53 (1999) 115–124116

new focus on lowering costs while maintain-ing or improving quality demands routinemeasurement [1]. Also, interventions to re-duce costs and improve quality may be mostsuccessful if they are focused at the level ofindividual decisions, yet are non-intrusive, adifficult combination to achieve. Fortunately,information technologies can help with bothquality measurement and quality improve-ment.

Regarding quality measurement, informa-tion systems represent an inexpensive way tocollect information on all patients, ratherthan samples as is often required with chartreview. Furthermore, such information canreadily be grouped in different ways and ma-nipulated. Perhaps most important, this in-formation can be used directly to improvequality, for example to contact all eligiblepatients who have not had some preventivemeasure.

For improving quality directly with infor-mation systems, two other domains are par-ticularly amenable to information-relatedapproaches, specifically diagnostic testing anddrug use. Diagnostic testing costs representup to 25% of all hospital costs [2]. Testordering is an area which physicians controland in which performance could be better.Studies of test-ordering [3–6] have found thatas much as 50% of diagnostic tests in teach-ing hospitals may be unnecessary. Despite agrowing information base about what repre-sents unnecessary utilization, physician be-havior with respect to test ordering has beenremarkably resistant to change over the longterm.

A number of interventions have been at-tempted to decrease utilization of tests [2–6].The major types of intervention studied havebeen feedback, education (including provid-ing information about clinical decision-mak-ing and cost issues), rationing and financialincentives [7,8]. Each of these strategies in the

most successful studies have produced tran-sient reductions of about 25% for targetedtests [8]. However, even the best-acceptedinterventions, those involving feedback andeducation, have had variable success [4,9] andimplementation has often been labor-inten-sive and costly [10]. The other major limita-tion for both types of intervention is thattheir effect tends to decay with time [8,11] ifthe intervention program is not continued:the gains have not been held.

Thus, despite growing information abouthow to better use diagnostic tests [12], inap-propriate use continues [13]. Why is this thecase? The reasons can be divided into twoprimary categories: incentives and informa-tion. In the past, there were few direct incen-tives for physicians to modify behavior, butthis is changing rapidly as a high percentageof patient care is now reimbursed under pre-paid plans and hospitals are now focusing onthe use of services. The reasons related toinformation can be further subdivided: (1)studies on the appropriate use of tests havebeen published in a wide array of sources andhave not been widely incorporated into medi-cal curricula [8]; (2) physicians have difficultyestimating risk and might make better deci-sions if they were better at it [14–16]; (3) theavailable interventions, such as review of uti-lization by senior physicians, have been time-consuming or difficult to incorporate in thelong term [10,11]; and (4) feedback is oftenseparated in time from decision-making [10].

Similar to the laboratory, pharmacy isboth a high volume and high expense compo-nent of health care where there is consider-able variability in practice patterns.Guidelines are prevalent regarding when totreat and which drugs are most cost-effective.However, the impact of these guidelines onphysician behavior is limited in part by thereluctance of physicians to utilize thesesources of information. Furthermore, formu-

D.W. Bates et al. / International Journal of Medical Informatics 53 (1999) 115–124 117

laries differ among insurance plans. Thusthe most cost-effective drug for a given clin-ical situation for a patient in one insuranceplan may represent inappropriate utilizationunder another plan. The need for immedi-ate access to overlapping guideline and for-mulary information makes pharmacymanagement a natural arena for informa-tion systems solutions.

The information-related reasons for inap-propriate resource utilization can be ad-dressed by combining a computerizedorder-entry system used by physicians witha computerized data review and ‘reminder’system that provides needed information atthe time decisions are made, makes sugges-tions and challenges orders when a poten-tial problem is found. Specifically, using thecomputer to provide feedback and re-minders to doctors is reliable and inexpen-sive, compared to manual review ofpractices. Also, order-entry is immediatelygeneralizable to all physicians, once in placerequires little maintenance and can be con-tinued indefinitely. But most important,physician order-entry allows immediate feed-back to physicians at the time they ordertests. To be optimally effective, an interven-tion should occur as close in time to theevent as possible and be constructive andnon-judgmental [17]. Computerized feedbackis ideal in both regards. Because physiciansuse their unique identification numbers toaccess the system, it is possible to track in-dividual physician behavior before and afterinterventions designed to affect suchbehavior.

The goals of this paper are to describechanges which have already been made inone hospital in our new integrated deliverysystem and further changes which can beexpected to have an impact for measuringand improving quality as the system isdeveloped.

2. Materials and methods

The Partners network is an integrated de-livery system including two large teachinghospitals, Brigham and Women’s Hospitaland the Massachusetts General Hospital,the Dana Farber Cancer Institute, as wellsmaller community hospitals such as theNorth Shore Medical Center. It also con-tains a physician network, Partners Com-munity HealthCare (PCHI), which includesover 700 physicians throughout the region.

The overall Partners IS plan calls for de-velopment of an information system thatwill be used across the network. At specificsites, such as Brigham and Women’s Hospi-tal and the Massachusetts General Hospital,a number of applications have already beenbuilt. Some of these applications which arefunctioning include an electronic outpatientrecord; for inpatients, inpatient physicianorder entry with outpatient physician orderentry in development; an event monitorwhich scans the database for events of in-terest; and a sophisticated quality and re-source utilization tracking system.

The network is currently developing ap-plications to support the above types offunctionality network-wide, including theLongitudinal Medical Record (LMR), whichwill serve as the medical record across thecontinuum of care for network patients; amaster patient index; and a data warehouse,which will track both quality measures andresource utilization in comparable waysacross the delivery system.

3. Results

3.1. Quality measurement at one site

Historically, Brigham and Women’sHospital measured quality by allowing each

D.W. Bates et al. / International Journal of Medical Informatics 53 (1999) 115–124118

Fig. 1. Sample Hospital Quality Report. This figure shows the hospital mortality rate over time. We routinely presentdata from the previous year and then data by quarter. These data are available on-line for the Maryland HospitalAssociation measures and can be used by managers who have received clearance. The data are also available bydepartment, division and physician.

department to choose whatever measures itelected and then to report periodically to theadministration. This resulted in little stan-dardization among departments and qualityreports were large stacks of paper which theadministration found difficult to evaluate.

More recently, we completely retooled ourquality measurement structure for the hospi-tal. A central precept was to measure asmuch as possible using information systems.We began measuring a small number ofparameters across the institution, includingMaryland Hospital indicators such as mortal-ity and readmission [18], HEDIS measures[19] and overall satisfaction with hospitaliza-tion (Fig. 1). Departments were divided intoclinical and non-clinical departments. Theclinical departments were asked to choosemeasures falling within one of several cate-gories: efficiency, critical variances and sen-tinel events. For example, the Department ofOrthopedic Surgery chose as its efficiencymeasures average length of stay for total hipand total knee replacement; for critical vari-ances, INR levels in patients on warfarin,deep venous thrombosis and wound infectionrates, postoperative hip dislocation rates andsatisfaction with care; and for sentinel events,inpatient deaths.

Using an electronic record has significantadvantages over billing data for many mea-sures. For example, for Pap smears, the

health maintenance section of an electronicrecord can provide much more accurate in-formation about whether appropriate pa-tients have received Pap smears than billingdata. If only claims data are used, the com-puter will search backwards through severalyears and look for a claim for a Pap smear.Whether a woman has had a hysterectomy,has refused a Pap smear, has moved away orswitched care to another primary careprovider, or has another medical condition(such as terminal cancer) that makes Papsmear unnecessary, cannot readily be consid-ered. Our computerized outpatient electronicrecord deals with these issues by looking atthe problem list to see whether hysterectomyis a coded problem; it is important to differ-entiate whether this was for benign or malig-nant disease. It also looks at the healthmaintenance grid which providers maintainand which allows them to designate whetherone of the above conditions (patient refuses,has changed site of care, etc.) is present. Theelectronic record can also help providers im-prove the rates at which patients have thesemeasures by flagging such needs at individualvisits and by compiling lists of patients bydoctor who are due for preventive services.These lists can be used by providers or byquality management to send letters to pa-tients suggesting they come in for neededservices.

D.W. Bates et al. / International Journal of Medical Informatics 53 (1999) 115–124 119

Routinely obtaining such data will be animportant challenge. We currently are pilot-ing an approach in which we give patients aprinted sheet at time of check-in for an out-patient visit including what we believe aretheir most up-to-date health maintenance andmedication information and ask the patientsto verify this and discuss any discrepancieswith their physician. Another promising ap-proach is to place computer terminals in ex-amination rooms and have patients have acomputer interaction before seeing the physi-cian. This may be an effective strategy bothfor increasing the use of health maintenancemeasures and also for specific problems as itwould allow much more flexibility than asingle printed sheet and would allow patientsto get specific information of interest tothem. For obtaining satisfaction data, itpromises to be particularly effective. In addi-tion, we will probably begin to develop mate-rials which patients can view from home overthe Internet.

3.2. Quality measurement across the network

Another challenge will be developing aquality measurement structure across the net-work. Integrated delivery systems require anumber of information systems components,including clinical/administrative data reposi-tories, telecommunications and networkingsystems, Master Person Indices (MPIs) anddata warehouses. We are currently in theprocess of building a clinical data warehouse,which will be different from the clinical datarepository (Fig. 2). The repository will have areal-time, single-patient focus and will beused to answer questions such as, ‘Has Pa-tient X had a flu shot?’ In contrast, the focusof the warehouse will be analytic, retrospec-tive and mostly aggregated and will be usedto answer questions like, ‘What percent ofpatients in Practice Z have had a flu shot andhow much does this vary by physician?’

A key issue in building the warehouse willbe developing tools to allow users to access itrapidly. However, tool choice depends on thedatabase structure, since on-line analysis pro-cessing (OLAP) tools tend to be built aroundspecific data models. While relational data-bases are most commonly used, complexqueries on very large relational databases canbe slow. Dimensional databases may befaster, but drill-down may be harder. Wehave tentatively selected a relational structureand will probably select as a query tool oneof a new generation of analysis tools, referredto as relational OLAPs, or ROLAPs.

As important as the warehouse structure isthe depth, comparability and accuracy of thedata being included. A unifying theme indeveloping the warehouse will be to promotenetwork-wide measurement of data using in-formation systems whenever possible andalso to integrate collection of these data intothe process of routine care. For comparisonsto be meaningful, it will be necessary to getthe members of the network to agree abouthow to compile individual measures. Signifi-cant work has been required even for mea-sures such as readmission and measuringother domains such as satisfaction compara-bly, for example, will require more coordina-tion. Short-term goals are to be able tomeasure the Maryland Hospital Associationmeasures for hospitals and the HEDIS crite-ria for outpatients across the network, as wellas severity-adjusted prospective expendituresin a variety of resource categories by physi-cian for patient subsets. Among the longer-term goals are to be able to measure qualityand resource utilization by episode withinspecific disease categories across the network.

3.3. Quality impro6ement

Information systems allow direct improve-ment in quality and cost reductions in a

D.W. Bates et al. / International Journal of Medical Informatics 53 (1999) 115–124120

Fig. 2. Data Flow to the Clinical Data Repository and Data Warehouse. This figure shows the data flow to these twoentities and contrasts the information contained in them.

variety of areas, but many of our earliestefforts have focused on diagnostic tests anddrugs. For diagnostic tests, we have alreadyimplemented and evaluated a number of deci-sion support measures, though many remain.In one randomized controlled trial evaluatingthe effect of displaying charges for clinicallaboratory tests, we found that interventionpatients had 4.5% fewer tests ordered andtotal charges for these tests were 4.2% lower,although neither difference reached statisticalsignificance [20]. Even though this differencewas not statistically significant, the annual-ized difference in total laboratory chargeswas $1.7 million and physicians like seeing

these charges, so we have continued to dis-playing them. In another randomized con-trolled trial, we displayed reminders forpotentially redundant tests and found thatabout 70% of such tests were canceled (Fig.3) [21], although the overall impact wassmaller than expected because many redun-dant tests did not have an associated comput-erized order, but were simply sent directly tothe laboratory. In another series of trials, weare evaluating the impact of structured order-ing—asking providers for their reasons forordering tests, with appropriate counter-de-tailing for improving the use of such tests asantiepileptic drug levels, digoxin levels, thy-

D.W. Bates et al. / International Journal of Medical Informatics 53 (1999) 115–124 121

Fig. 3. Sample Reminder for a Redundant Laboratory Test. This figure shows the reminder that would appear whena physician orders a test earlier that a test-specific predefined interval. We have found that when physicians receivethese reminders, they cancel the test about 70% of the time and even when they continue, the results are rarely useful[21].

roid tests and abdominal radiographs [22–25]. For abdominal radiographs, we foundthat we were able to identify low-yield indica-tions, but that clinicians were rarely willing tocancel such tests, although they were morewilling to change to examinations which wereless expensive and more clinically appropriate[25].

Systems can also be used to rapidly com-municate markedly abnormal results toproviders. We have developed an approach inwhich the information system is directly in-terfaced with the paging system and anotherapplication identifies the clinician responsiblefor each patient at any given time, making itpossible to rapidly inform the appropriateclinician about important results [26].

For drugs, drug injuries can be preventedand the costs of direct medication costs canbe reduced using decision support [27]. Com-puterized physician order entry can make

drug ordering safer by showing clinicians de-fault dosages, putting in place dose ceilings,eliminating transcription and requiring com-plete orders. But perhaps most important, anumber of checks can be made in the back-ground, to look for drug allergies, drug–druginteractions and drug-laboratory problems[28]. Guided dose algorithms should make itpossible to more appropriately dose agentssuch as aminoglycosides and heparin.

Furthermore, efficiency can be improvedby making suggestions about dose, fre-quency, route and drug changes. For drug–drug substitutions within classes which havetherapeutically equivalent alternatives, dis-playing guidelines about which drug to usewithin a class has been very effective, result-ing in almost exclusive use of suggested alter-natives. Agents without a therapeuticalternative can also be approached, althoughthis is more difficult. For example, we re-

D.W. Bates et al. / International Journal of Medical Informatics 53 (1999) 115–124122

cently found that display of a guideline forvancomycin decreased vancomycin days perprovider by 37% [29]. Frequency suggestionshave been very successful, for example, wehave found that simply changing the defaultdosing frequency from twice a day to once aday for an expensive antibiotic, ceftriaxone,resulted in a switch in use from 85% twice aday to 85% once a day, almost overnight.

Critical paths are another tool which havebeen used to improve quality. Our hospitalhas developed approximately 20 critical path-ways, which specify expected occurrences andcare plans for a specific condition, such ascoronary artery bypass surgery. These havesignificantly decreased costs for the condi-tions involved, while improving patient satis-faction. Information systems are importantfor pathways in several ways. First, manypaths are essentially sequential sets of orders,so that they relate extremely well to physicianorder entry and it is easier to write order setsusing order entry than on paper. Second,paths rely on determining when somethingspecific occurred (such as removal of theFoley catheter) at serial times and much ofthis can be automated instead of having re-search assistants collect these data. Third, asignificant problem in implementing paths ismaking providers aware they are availablefor a given condition. For example, for thestroke pathway at our institution, only abouta third of patients are enrolled. To deal withthis issue, in several months we will beginrequiring providers to enter the admittingdiagnoses in coded form at the time the pa-tient is admitted and we will direct them toany available paths.

4. Discussion

In the US, the rising expense of health carehas prompted unprecedented focus on costs

and at the same time on measuring qualitybecause of fears that quality will decline ascosts are reduced. While it is clear thatputting in place financial incentives forproviders can reduce costs, this represents ablunt sword. In contrast, information systemscan be used to specifically target areas whereadditional care is needed and other areaswhich represent marginal or unnecessary uti-lization. Although we now know much moreabout what care is indicated, study afterstudy demonstrates huge gaps between cur-rent best practices according to guidelinesand actual performance. Thus, we believethat there are enormous challenges not sim-ply in knowing what to do, but in actuallygetting it done and computers represent apowerful but underutilized tool for meetingthese challenges. The interventions on whichwe are focusing are targeted at practices thatwill directly impact quality of care and pa-tient outcomes. Furthermore, we will be ableto use population-based approaches to targetpatients who have not come in and may thusbenefit most from some of these measures.

Information systems will have their mainimpact in three ways. First, they can be usedto directly improve quality, by gettingproviders the information and decision sup-port they need when they directly interactwith the information system in real time.Second, efficiency and quality can be furtherimproved by using event monitors to look forasynchronous events and communicate themto providers. Third, it will be possible toperform quality measurement using informa-tion systems in ways which will be less expen-sive yet more comprehensive and reliablethan previous methods.

In conclusion, the costs of care continue torise and as more technological advances be-come available this trend will continue. Touse technology appropriately, better decisionsupport is essential. This will involve both

D.W. Bates et al. / International Journal of Medical Informatics 53 (1999) 115–124 123

information display and guidelines. Informa-tion systems offer the best opportunity tobring decision support to the point of careand ensure that guidelines are used. All thiswill take place within the context of inte-grated delivery systems, which will ensurethat redundancy is minimized and willprovide large quantities of data for qualitymeasurement and improvement. Further op-timization of care will depend heavily onroutine quality measurement. In the future,almost all quality measurement will be doneusing information systems and will be seam-lessly integrated into the process of routinecare. Not only will health care providers usethese systems, but patients will use computersin waiting rooms and from home.

Acknowledgements

Adapted from: D.W. Bates, E. Pappius,G.J. Kuperman, D. Sittig, H. Burstin, D.G.Fairchild, T.A. Brennan, J.M. Teich. Mea-suring and Improving Quality Using Infor-mation Systems, MEDINFO 1998, vol. 2,A14–A18.

References

[1] D.W. Bates, E. Pappius, G.J. Kuperman, D. Sittig,H. Burstin, D.G. Fairchild, T.A. Brennan, J.M.Teich, Measuring and Improving Quality UsingInformation Systems, MEDINFO 1998, 2, A14–A15.

[2] J.S. Spiegel, M.F. Shapiro, B. Berman, S.Greenfield, Changing physician test ordering in auniversity hospital. An intervention of physicianparticipation, explicit criteria and feedback, Arch.Intern. Med. 149 (1989) 549–553.

[3] J.M. Eisenberg, S.V. Williams, L. Garner, R.Viale, H. Smits, Computer-based audit to detectand correct overutilization of laboratory tests,Med. Care 15 (1977) 915–921.

[4] S.V. Williams, J.M. Eisenberg, A controlled trialto decrease the unnecessary use of diagnostic tests,J. Gen. Intern. Med. 1 (1986) 8–13.

[5] S.A. Schroeder, K. Kenders, J.K. Cooper, T.E.Piemme, Use of laboratory tests and pharmaceuti-cals. Variation among physicians and effect of costaudit on subsequent use, J. Am. Med. Assoc. 225(1973) 969–973.

[6] R.H. Dixon, J. Laszlo, Utilization of clinicalchemistry services by medical house staff. An anal-ysis, Arch. Intern. Med. 134 (1974) 1064–1067.

[7] A.R. Martin, M.A. Wolf, L.A. Thibodeau, V.Dzau, E. Braunwald, A trial of two strategies tomodify the test-ordering behavior of medical resi-dents, New Engl. J. Med. 303 (1980) 1330–1336.

[8] R.M. Grossman, A review of physician cost-con-tainment strategies for laboratory testing, Med.Care 21 (1983) 783–802.

[9] T.A. Parrino, The nonvalue of retrospective peercomparison feedback in containing hospital antibi-otic costs, Am. J. Med. 86 (1989) 442–448.

[10] J.M. Eisenberg, S.V. Williams, Cost containmentand changing physicians’ practice behavior. Canthe fox learn to guard the chicken coop?, J. Am.Med. Assoc. 246 (1981) 2195–2201.

[11] S.B. Soumerai, J. Avorn, Efficacy and cost-con-tainment in hospital pharmacotherapy: state of theart and future directions, Milbank Mem. Fund Q.Health Soc. 62 (1984) 447–474.

[12] H.C. Sox Jr., Probability theory in the use ofdiagnostic tests. An introduction to critical studyof the literature, Ann. Intern. Med. 104 (1986)60–66.

[13] J. Wennberg Gittelsohn, Small area variations inhealth care delivery, Science 182 (1973) 1102–1108.

[14] R.M. Poses, R.D. Cebul, M. Collins, et al., Theaccuracy of experienced physicians’ probability es-timates for patients with sore throats, J. Am. Med.Assoc. 254 (1985) 925–929.

[15] M. Hlatky, E. Botvinick, B. Brundage, Diagnosticaccuracy of cardiologists compared with probabil-ity calculations using Bayes’ rule, Am. J. Cardiol.49 (1982) 1927–1931.

[16] D.G. Fryback, J.R. Thornbury, Evaluation of acomputerized Bayesian model for diagnosis of re-nal cyst vs. normal variant from excretory uro-gram information, Invest. Radiol. 7 (1976)102–111.

[17] J.D. Restuccia, The effect of concurrent feedbackin reducing inappropriate hospital utilization,Med. Care 20 (1982) 46–62.

D.W. Bates et al. / International Journal of Medical Informatics 53 (1999) 115–124124

[18] Maryland Hospital Association, The Associationof Hospitals and Health Care Systems. Qualityindicator project implementation manual, May,1996.

[19] C. Appleby, HEDIS (Health Plan Employer Dataand Information Set): managed care’s emerginggold standard, Manage. Care 4 (1995) 19–24.

[20] D.W. Bates, G.J. Kuperman, A. Jha, J.M. Teich,E.J. Orav, N. Ma’luf, A. Onderdonk, R. Pugatch,D. Wybenga, J. Winkelman, T.A. Brennan, A.L.Komaroff, M.J. Tanasijevic, Does the computer-ized display of charges of charges affect inpatientancillary test utilization?, Arch. Int. Med. 157(1997) 2501–2508.

[21] D.W. Bates, G.J. Kuperman, E. Rittenberg, J.M.Teich, A. Onderdonk, J. Winkelman, A.L. Ko-maroff, M.J. Tanasijevic, Reminders for redundanttests: results of a randomized controlled trial, Proc.Ann. Symp. Comp. Appl. Med. Care 19 (1995)935.

[22] R.A. Schoenenberger, M.J. Tanasijevic, A. Jha,D.W. Bates, Appropriateness of antiepileptic druglevel monitoring, J. Am. Med. Assoc. 274 (1995)1622–1626.

[23] F. Canas, M. Tanasijevic, N. Ma’Luf, D.W. Bates,Evaluating the appropriateness of digoxin level

monitoring, J. Gen. Intern. Med. 12 (1997) 66.[24] C.G. Solomon, P.K. Goel, P.R. Larsen, M.

Tanasijevic, D.W. Bates, Thyroid function testingin an ambulatory setting: identifying suboptimalpatterns of use, J. Gen. Intern. Med. 11 (1996) 88.

[25] L.H. Harpole, R. Khorasani, G. Kuperman, J.Fiskio, D.W. Bates, On-line counterdetailing: doesit change physician behavior when ordering ab-dominal radiographs?, J. Gen. Intern. Med. 11(1996) 74.

[26] G.J. Kuperman, D. Boyle, A. Jha, M. Tanasijevic,J.M. Teich, J. Winkelman, D.W. Bates, Timelinessof treatment of critical laboratory results in inpa-tients, J. Gen. Intern. Med. 10 (1995) 71.

[27] D.W. Bates, G. Kuperman, J.M. Teich, Computer-ized physician order entry and quality of care,Qual. Manage. Health Care 2 (1994) 18–27.

[28] D.W. Bates, A.C. O’Neil, D. Boyle, J. Teich, G.M.Chertow, A.L. Komaroff, T.A. Brennan, Potentialidentifiability and preventability of adverse eventsusing information systems, J. Am. Med. Inform.Assoc. 1 (1994) 404–411.

[29] K.G. Shojania, D. Yokoe, R. Platt, J. Fiskio, N.Ma’Luf, D.W. Bates, Reducing vancomycin uti-lization using computer order entry: results of anRCT, J. Gen. Intern. Med. 12 (1997) 63.

.