Developing a decision support system for tobacco use ...

Refereed papers

Developing a decision support system fortobacco use counselling using primarycare physiciansTheodore W Marcy MD MPHOffice of Health Promotion Research, and Vermont Cancer Center, University of Vermont College ofMedicine, USA

Bonnie Kaplan PhDYale Center for Medical Informatics, Yale University School of Medicine, Department of Biomedical andHealth Information Services, University of Illinois at Chicago, and Kaplan Associates, Hamden, CT, USA

Scott W Connolly EdD MPHOffice of Health Promotion Research, University of Vermont College of Medicine, USA

George Michel MS MBA

Richard N Shiffman MD MCISYale Center for Medical Informatics, Yale University School of Medicine, USA

Brian S Flynn ScDOffice of Health Promotion Research, and Vermont Cancer Center, University of Vermont College ofMedicine, USA

ABSTRACT

Background Clinical decision support systems

(CDSS) have the potential to improve adherence

to guidelines, but only if they are designed to work

in the complex environment of ambulatory clinics

as otherwise physicians may not use them.

Objective To gain input from primary care phys-

icians in designing a CDSS for smoking cessation toensure that the design is appropriate to a clinical

environment before attempts to test this CDSS in a

clinical trial. This approach is of general interest to

those designing similar systems.

Design and approach We employed an iterative

ethnographic process that used multiple evaluation

methods to understand physician preferences and

workflow integration. Using results from our priorsurvey of physicians and clinic managers, we devel-

oped a prototype CDSS, validated content and design

with an expert panel, and then subjected it to

usability testing by physicians, followed by iterative

design changes based on their feedback. We then

performed clinical testing with individual patients,

and conducted field tests of the CDSS in two

primary care clinics during which four physicians

used it for routine patient visits.

Results The CDSS prototype was substantially

modified through these cycles of usability and clin-

ical testing, including removing a potentially fataldesign flaw. During field tests in primary care clinics,

physicians incorporated the final CDSS prototype

into their workflow, and used it to assist in smoking

cessation interventions up to eight times daily.

Conclusions A multi-method evaluation process

utilising primary care physicians proved useful for

developing a CDSS that was acceptable to phys-

icians and patients, and feasible to use in theirclinical environment.

Keywords: medical informatics, qualitative re-

search, smoking cessation

Informatics in Primary Care 2008;16:101–9 # 2008 PHCSG, British Computer Society

TW Marcy, B Kaplan, SW Connolly et al102

Introduction

Computer-mediated clinical decision support systems

(CDSSs) are software systems that match characteristics

of a patient with a knowledge base of information onrecommended care in order to provide patient-specific

recommendations as well as other information man-

agement services.1–7 A number of CDSSs intended to

support the use of clinical guidelines have improved

physician adherence to guidelines, but others have

been unsuccessful.1,2,8,9

To be successful, a CDSS must work within the

complex environment of ambulatory clinics where thistechnology interacts dynamically with clinicians, patients

and existing office systems.10,11 Workflow integration

is one of the grand challenges for health information

technology. If physicians find these tools too difficult

to incorporate into clinical workflow they will be aban-

doned.12–14 To avoid such problems, a CDSS should

first reflect the needs and preferences of the users (e.g.

physicians) and the organisational systems (e.g. am-bulatory clinics) within which it works.10,14–17 Such a

CDSS should then be introduced into clinical practice

only after a ‘rigorous schedule of iterative usability

testing and formative evaluation’10 during which the

CDSS is modified to reflect the needs of the user and

the demands of the clinical environment.

We followed these recommendations during devel-

opment of a CDSS to assist physicians in using theUnited States Public Health Service (USPHS) Guideline

on Tobacco Use and Dependence Treatment. This

guideline recommends physicians perform the five

‘As’:

1 identify patients’ smoking status (‘ask’)

2 advise those who smoke to quit

3 assess readiness to quit

4 assist in quit attempts and

5 arrange for follow up.18

A CDSS for this guideline could guide physicians in

choosing and prescribing pharmacotherapy, facilitate

referral of patients to counselling resources and pro-

vide for patients a tailored handout with this informa-

tion, potentially improving adherence and effectiveness

and saving time.We employed the three-phase development process

of definition, usability testing, and clinical testing recom-

mended by Wyatt and Spiegelhalter (Figure 1).19 This

process used a multi-method ethnographic approach20

that included surveys of key stakeholders, iterative

usability testing with primary care physicians, validity

testing and consultation with an expert panel, initial

clinical testing with patients, and then pilot testing byphysicians.10,21,22

In the definition phase we surveyed 600 Vermont

primary care and subspecialty physicians and 93 clinic

office managers to determine current practice, the

environment within these ambulatory clinics, percep-

tions of barriers to performing smoking cessation

interventions and preferences among potential infor-

mation management services.4,23 This paper describesthe second and third phases of iterative design and

testing of the CDSS, including initial clinical testing

that demonstrated it was feasible for physicians to use

this CDSS in two primary care clinics.

Methods

Iterative usability and validity testing

The initial prototype was developed by the Yale Center

for Medical Informatics based on the responses to the

surveys,23 on the smoking cessation resources in

Vermont and neighbouring states and on the content

of the USPHS Guideline.18 The purpose of the second

phase was formative evaluation of the evolving CDSS.

Figure 1 A depiction of the process used to develop the tobacco use treatment computer-mediated decision

support system (CDSS)

Developing a decision support system for tobacco use counselling using primary care physicians 103

Our evaluation was guided by two theories relevant to

developing information technology that favor adop-

tion by users: the Technology Acceptance Model 2 and

Rogers’ Diffusion of Innovations.24,25 Semi-structured

interview items were constructed to address the fol-

lowing attributes from these two theories of the CDSSas viewed by the physicians:

. perceived usefulness including job relevance and

the quality of output. relative advantage over their current smoking cess-

ation interventions. compatibility with clinic systems. perceived complexity of the CDSS compared to

other information technology and their. intention to use the CDSS if it were available.

Because the small number of participants would make

valid analyses difficult, we chose not to utilise quanti-

tative measures of these perceptions. Qualitative methods

were chosen to complement our prior surveys and to

provide more rich, in-depth and nuanced information

than a quantitative questionnaire could provide.

Physician panels for usability testing

The working prototype was first subjected to usability

testing with three physicians active in the tobacco control

community. In the second round of testing, four

physicians were randomly selected from a list of primary

care physicians in Chittenden County, Vermont. Allseven physicians invited to participate agreed to do so.

Usability testing consisted of each physician using the

CDSS during hypothetical patient encounters presented

by a test monitor (SC) while an observer (TWM)

recorded the interactions. The testing combined three

sources of data:

1 a think-aloud protocol

2 handwritten field notes during observation

3 audio taped ethnographic interviews that included

the items derived from the two theories described

above.26

These interviews and observations were analysed by

two of the investigators (TWM and SC) using codes

based on the two theories. Validity was addressed by

reviewing our conclusions with the participants,27,28

by discussing results with expert panel members and

by having one investigator (BK), who was not present

at these sessions, review data analysis and interpret-

ation. Following these analyses, we developed a list of

design changes that were then incorporated into the

revised CDSS used in the next round of usability

testing.

Expert panel

We formed an expert panel consisting of three experts

on tobacco use treatment, one each in behavioural

counselling, pharmacotherapy and patient education,

and an additional physician with expertise in ambu-

latory clinic processes. These individuals reviewed the

CDSS’s validity as an implementation of the USPHS

Guideline and provided guidance on the content and

redesign of the CDSS throughout this process. We alsoconsulted with additional individuals who had exper-

tise in clinic information systems, Vermont’s tobacco

use cessation services, readability of patient handouts

and Medicare billing documentation and compliance.

Clinical testing

Testing with patients

The initial clinical testing of the modified CDSS was

performed in the ambulatory clinic of one of the

investigators (TWM). Once technical issues of trans-

ferring administrative data to the CDSS and printing

accurate documents were resolved, the investigator usedthe CDSS with consenting patients identified through

the clinic’s standard screening process as current or

recent (within one month) smokers. Attempts were

then made to interview these patients by telephone

within two weeks of this visit to assess the patients’

perceptions of the encounter and the CDSS.

Field tests in primary care clinics

Two physicians and the staff in each of two primary

care clinics agreed to field test the CDSS. These two

clinics were selected because they were not involved in

any other current outpatient research studies and they

used a common patient administrative database pro-

gram (GE Healthcare). At the end of the testing, each

physician was interviewed separately by TWM using asemi-structured interview guide based on the same

items as those in the usability testing. The interview

was audio taped for transcription and review. The

protocols for the usability and clinical testing were

reviewed and approved by the University of Vermont’s

Institutional Review Board.

Results

Phase 2: iterative usability testing

In our 2003 surveys, we found that 94% of the clinics

had a computerised registration system, but only 20%


of Vermont ambulatory clinics had an electronic health

record (EHR) into which a CDSS could be integrated,23

similar to a contemporary national estimate.29 There-

fore, a CDSS intended for wide adoption would need

to function in offices in which physicians might not

otherwise use a computer during patient visits. How-ever, the CDSS could potentially utilise information

from a computerised administrative database. Both

physicians and clinic office managers preferred that

the CDSS be on a handheld computer (PDA) for both

space and cost considerations23 and both groups

wanted the CDSS to:

1 provide patient-specific information

2 generate tailored patient handouts

3 utilise state of residence and type of health insur-

ance in forming recommendations and

4 document the intervention for the medical record.23

The initial prototype of our smoking cessation PDA

decision support system (SC-PDA) and all subsequent

prototypes of this CDSS used a web browser on a PDA

to connect via a wireless local area network to a serverin a ‘client–server’ relationship. Each day, a software

routine on the server pulled a flat file of data on

scheduled patients from the administrative database

that included the primary physician, insurance cover-

age, medical record number, date of birth and resi-

dence. The server also contained the CDSS algorithms

based on the USPHS Guideline18 and information about

local cessation resources. Based on input by the physi-cian on the PDA screens during the patient–physician

interaction, the server compiled and printed documents

at the checkout station with information specific to

the patient.30 Figure 2 provides a schematic of the

prototype system used in the clinical testing and

Figure 3 shows examples of two of the screens.

Prior research demonstrated that an active promptto use a CDSS was more effective than a passive system

relying on a physician to remember to use the sys-

tem.1,2 We, therefore, designed the first prototype so

that the vital signs and smoking status were recorded

on a computer at intake and then communicated via

the network to the physician’s PDA as an electronic

prompt that would alert the physician to consider

using the SC-PDA.All seven physicians who used the SC-PDA in simu-

lated clinical encounters provided numerous sugges-

tions for improvements or alterations. We modified

the SC-PDA in response to this feedback and that of

the expert panel as summarised in Table 1.

For illustrative purposes, we review the process that

led to a major revision in the SC-PDA: changing the

electronic alert to a paper-based prompt to use the SC-PDA. Two of three physicians in Round 1 of usability

testing had negative opinions about the electronic

alert. Without an EHR in the clinic they saw no value

in entering vital signs electronically other than for the

sole purpose of identifying smoking status, and viewed

this process as interfering with current systems for

patient intake. To gauge preferences, in the second round

of usability testing we presented both the electronicalert and an alternative paper-based prompt for using

Figure 2 A schematic of the final prototype that was tested in the clinical testing. Stickers on the clinic vital

sign record would prompt the physician to use the SC-PDA with appropriate patients based on smoking status.

The server acted as a central repository for patient administrative data and guideline information. The

physician communicated with this server over a wireless network via a PDA from which additional information

could be entered about the patient and through which information about patient-specific guideline

recommendations and resources were displayed. The server compiled and printed patient specific documents

based on these data elements


the SC-PDA. The alternative method retained the

transfer of patient administrative data to the physician’s

PDA, but not the vital signs or smoking status. Instead,

the clinic’s existing vital signs record was adapted toindicate the smoking status by a coloured sticker that

could alert the physician to open the SC-PDA pro-

gram, select the specific patient from their roster,

confirm the smoking status and then proceed with

the intervention. Three of four primary care physicians

preferred the paper-based visual prompt because it

required the least change in their clinic intake system.The fourth physician did not like either version because

it was his opinion that asking about smoking status

every visit would alienate patients. Based on the majority

Figure 3 Examples of two of the screens on the PDA with which the physician interacts on the SC-PDA. In the

screen on the left, the physician would determine and then check any of the boxes corresponding to some of

the factors that would be cautions or contraindications to the use of one or more FDA approved medications

for smoking cessation. The screen on the right would then display the recommended medications in green

(light grey in the figure) and any cautioned or contraindicated medications in yellow or red (dark grey). In this

example, Bupropion is contraindicated because of the risk of lowering seizure threshold in patients with a

prior history of seizures or head injury

Table 1 Major design changes in the SC-PDA resulting from usability testing

Problem noted Design alterations

1 Electronic alert to use SC-PDA with appropriatepatients incompatible with clinic workflow and

systems

Alternative paper-based visual prompt on vital signssheet tested with physicians and then substituted

2 Time to obtain health history to review for

medication cautions/contraindications viewed asexcessive

Nicotine nasal spray deleted from medication

choices to reduce number of screened conditionsTwo-stage screening to exclude common

contraindications to initial medication use first

(pregnant, unstable angina, serious arrhythmia)

3 Unable to cycle from ‘not motivated’ to

‘motivated’ if patient changes intention to quit

during counselling

Button on screen recycles user back to ‘Assess’

readiness to consider quit attempt

4 Insufficient counselling guidance Optional screen with prompts allows MD toprovide more counselling if desired

5 Layout of tailored handouts not inviting;

language at too high a reading level

Tailored handout scripts reviewed and edited by

education and readability consultants

6 Need health record documentation that supports

Medicare billing

The documentation note was redesigned to supplement

the standard documentation of the visit and to

comply with requirements for billing for tobacco

cessation counselling


responses, we changed to the paper-based visual prompt

in the subsequent prototype.

Phase 3: clinical testing

Testing with patients

Nine patients (three female; six male) were recruited

to have one investigator, a physician, use the CDSS

with them. Seven of the patients were current smokers

and two were recent quitters. The sequence of screens

appropriate for each patient’s smoking status and

treatment preferences was completed in an average

time of ten minutes (range 8–13 minutes). All but one

of the nine patients actively looked at the PDA screenswith the physician during the process. Five of the nine

completed a structured telephone interview with

another investigator (SC) within two weeks of the

visit; one declined the interview, one did not recall the

SC-PDA, and two were unable to be contacted. Four of

the five interviewed patients rated the SC-PDA posi-

tively on its usefulness in assisting the discussion on

smoking; the other patient was neutral and none hadany negative comments about the SC-PDA or having

the physician use it with them. Some stated that being

able to follow the PDA screens made the questions

easier to understand and facilitated their decision-

making. Several design problems were discovered during

this phase of the testing, and the subsequent design

changes are outlined in Table 2.

Field tests in primary care clinics

Neither of the two participating clinics had a system

for identifying a patient’s smoking status. Working with

the clinic staff, the process of obtaining and recording

vital signs was adapted to include determination of

smoking status by intake personnel and using stickers

to indicate this status on the vital sign sheet. This

process was implemented before any further changes

were made so that adjusting to this system would beless likely to affect SC-PDA testing. After two weeks,

physicians were each given a PDA and trained on the

SC-PDA program in one session of approximately 60

minutes. Clinic personnel were trained on the hand-

ling of the tailored patient handouts, fax referral forms

and chart documentation notes. The participating

physicians used the SC-PDA during a trial period of

three weeks. Each physician’s assigned PDA only dis-played the patients on his or her own schedule. An

additional entry labelled ‘generic’ allowed the program

to be used with unscheduled patients, though without

the benefit of the administrative information.

The field tests in the two clinics were completed as

scheduled except for a single day when the SC-PDA

was inoperative because of a server malfunction. In the

physician interviews following the trial periods, thefrequency of use of the SC-PDA estimated by each of

the four physicians ranged between only four times in

three weeks to as many as eight times in one day out of

18–20 daily patient visits, with the most cited frequencies

as one and four times a day. The physician using the

Table 2 Design alterations adopted after initial clinical testing with patients

Problem noted Design alterations

1 In recent quitters, no ability to increase dosage

to address significant withdrawal symptoms

SC-PDA changed to allow physician to choose

appropriate dose of nicotine products based on

withdrawal symptoms

2 Prescription instructions not available at the end

of the SC-PDA algorithm when most physicians

write prescriptions

Last screen of the algorithm for each patient

provides prescription instructions for

recommended medications if any selected

3 If nicotine inhaler selected, physician needed to

be prompted to write a script

Pop-up screen added to remind physician that the

inhaler is not over the counter and requires a script

4 Printing order of medication instructions did

not follow recommendations if two selected

In some patients, nicotine patch is first medication

initiated, with addition of lozenge or gum only if

continued strong urges. Order of medication

instructions changed to correspond with this usual

sequence

5 Names of several of the MAO inhibitors

unfamiliar to physicians as they checked for

this contraindication to Bupropion use

Information button allows physicians to review

generic and brand names of medications with this

property while in screen to review medical history

cautions and contraindications to specific

medication use


SC-PDA the least stated that he had few opportunities

as there were few active smokers in his panel of patients.

The clinic support staff corroborated the frequency of

use of the SC-PDA based on their handling of the

patient tailored handouts distributed at checkout.

Two of the physicians reported that the ability toshow the screens to the patient and have them ‘rally

around’ the computer and follow along with the steps

of making recommendations was a relative advantage

of the SC-PDA over standard counselling.

‘... so it was sort of a different approach. Was not just a

provider saying ‘‘look, I want you to quit’’. It was a

provider saying, ‘‘I want you to quit and we have this

program that we’re using that could really maximize your

chances at success’’.’ (Physician 1)

All four physicians commented favourably on the

ability to personalise both the spoken and written

counselling, check for medication cautions and contra-

indications, and document their services to support

billing as well as insurance coverage of medications.

The two physicians in the first trial clinic found thetime required to use the SC-PDA was a relative dis-

advantage. With probing, we learned to reduce user

time by emphasising during training that the SC-PDA

could print a tailored handout even if physicians chose

not to do the intervention and that physicians could

skip the quit date screen if they preferred. Prior to the

trial in the second clinic, we adapted the training to

highlight these options and eliminated a counsellingselection screen. The two physicians in the second trial

clinic both reported that they could use the SC-PDA

efficiently. The physician in this clinic who used it the

most stated;

‘It actually is pretty quick and easy. I suspect that most of

my interventions were in the lower end of three to ten

minutes.’ (Physician 4)

This physician also commented that she would tend tobill for the service when using the SC-PDA, whereas

before she had not, partially compensating for the

additional time spent. Notably this physician had not

previously used a PDA.

Discussion

We developed a CDSS for smoking cessation inter-

ventions hand-in-hand with physicians – the intended

end users. We used an ethnographic approach in order

to design a CDSS that provided advantages over how

physicians usually advise patients about smoking, thatwas compatible with clinic workflow, and that was inte-

grated with how physicians conduct patient visits.14,21

Through usability testing, we were better able to

understand the workflow of the clinical practice, trans-

late these preferences into the design of our SC-PDA

and then implement the SC-PDA in two ambulatory

clinics. This experience illustrates the value of a staged

multi-method ethnographic process for obtaining

detailed end user feedback during the developmentof health information technology.

For example, we first incorporated an electronic

alert into the prototype SC-PDA because this charac-

teristic had been associated in the literature with im-

provements in physician adherence. Usability testing

demonstrated, however, that physicians were resistant

to a different method of identifying smoking status

because of its perceived impact on existing office sys-tems. This opinion was not apparent in our surveys of

physicians and clinic managers. Had we retained the

electronic alert, we could have experienced multiple

failed attempts during implementation. Additional

problems with the SC-PDA became apparent only

when using it with patients in the clinical setting.

Our pilot tests in two clinics demonstrated that

physicians did use the final SC-PDA prototype in actualclinical settings, and did see a relative advantage of the

SC-PDA over usual practice. The time required to use

the SC-PDA is a recognised barrier to preventive care

in general and tobacco counselling in particular.31,32

We took several measures to address this in our design.

First, additional information about medications and

counselling was provided in screens that were available if

desired, but it was not necessary to work throughthese. Second, we acted on physician recommendations

to streamline information and reduce the content and

number of screens. Third, the SC-PDA could produce a

tailored handout for a patient even if the physician

opted not to use the intervention during a patient visit.

An unanticipated observation was that both patients

and physicians saw benefits in having the patient view

the screens along with the physician. This sharedviewing of screen contents engaged the patient and

appeared to improve patient understanding. A CDSS

designed for patients to use by themselves would not

capitalise on this interaction. The future development

of decision support systems should exploit a CDSS’s

potential for interactive, collaborative decision making.

There are limitations to this approach to CDSS

development. The time-intensive nature of usabilitytesting reduces the number of end users who can

provide feedback. Those who accept an invitation to

do usability testing may not be representative of other

physicians. In addition, a CDSS developed through

this process still may not increase guideline adherence

or improve patient outcomes even if physicians use

the CDSS. Clinical trials are necessary to address these

questions. However, this type of CDSS developmentshould precede any large-scale testing to avoid ex-

pensive null results from a CDSS that physicians will

not use during these clinical trials.


A particular limitation to our methods was that one

of the developers (TWM) recruited the physicians and

clinics selected for usability testing and was present

during the usability testing. However, other evaluators

(SC and BK) were not involved in the SC-PDA devel-

opment. Wears et al recommend that the developersof a health information technology system should not

be the evaluators of the system as there is the potential

for bias.11 The developer/evaluator may selectively

record criticisms and the subjects may be less open

with the developer about problems they perceive.

Conclusions

Successful integration of health information tech-

nology into clinical practice will require collaborative

development of these systems with physicians, patients

and support staff. The combination of multiple and

iterative ethnographic methods incorporating surveys,usability testing and expert panels is feasible and

useful. Through this process we avoided costly and

potentially fatal errors in the design of our CDSS.

ACKNOWLEDGEMENTS

We gratefully acknowledge the participation of our

physician and patient subjects in this study and the

assistance from members of our expert panel. Finan-cial support of this research was provided by grants

from the National Cancer Institute: NCI R03 CA111640

and Dr Marcy’s career development award NCI K07

CA102585–01.

REFERENCES

1 Garg A, Adhikari N, McDonald H et al. Effects of

computerized clinical decision support systems on prac-

titioner performance and patient outcomes: a systematic

review. Journal of the American Medical Association 2005;

293:1223–38.

2 Kawamoto K, Houlihan C, Balas E and Lobach D.

Improving clinical practice using clinical decision sup-

port systems: a systematic review of trials to identify

features critical to success. British Medical Journal 2005;

330:765–72.

3 Blumenthal D and Glaser J. Information technology

comes to medicine. New England Journal of Medicine

2007;356:2527–34.

4 Shiffman R, Brandt C, Liaw Y and Corb G. A design

model for computer-based guideline implementation

based on information management services. Journal of

the American Medical Informatics Association 1999;6:99–

103.

5 Committee on Quality of Health Care in America of the

Institute of Medicine. Crossing the Quality Chasm: a new

health system for the 21st century. Washington, DC:

National Academy Press, 2001.

6 Frist W. Health care in the 21st century. New England

Journal of Medicine 2005;352:267–72.

7 James B. Making it easy to do it right. New England

Journal of Medicine 2001;345:991–3.

8 Hunt D, Haynes R, Hanna S and Smith K. Effects of

computer-based clinical decision support systems on

physician performance and patient outcomes. Journal of

the American Medical Association 1998;280:1339–46.

9 Shiffman R, Liaw Y, Brandt C and Corb G. Computer-

based guideline implementation systems: a systematic

review of functionality and effectiveness. Journal of

the American Medical Informatics Association 1999;6:

104–14.

10 Hesse B and Shneiderman B. eHealth research from the

user’s perspective. American Journal of Preventive Medi-

cine 2007;32:S97–S103.

11 Wears R and Berg M. Computer technology and clinical

work: still waiting for Godot. Journal of the American

Medical Association 2005;293:1261–3.

12 Lee J, Cain C, Young S, Chockley N and Burstin H. The

adoption gap: health information technology in small

physician practices. Health Affairs 2005;24:1364–6.

13 Wachter R. Expected and unanticipated consequences

of the quality and information technology revolutions.

Journal of the American Medical Association 2006;295:

2780–3.

14 Elson R. Clinical practice guidelines: the role of tech-

nology in perspective. Disease Management Health Out-

comes 1997;1:63–74.

15 Kaplan B, Brennan P, Dowling A, Friedman C and Peel

V. Towards an informatics research agenda: key people

and organizational issues. Journal of the American Medi-

cal Informatics Association 2001;8:235–41.

16 Friedman C. Information technology leadership in aca-

demic medical centers: a tale of four cultures. Academic

Medicine 1999;74:795–9.

17 Littlejohns P, Wyatt J and Garvican L. Evaluating

computerised health information systems: hard lessons

still to be learnt. British Medical Journal 2003;326:860–3.

18 Fiore M, Jaen C, Baker T et al. Treating Tobacco Use and

Dependence: 2008 update. Clinical Practice Guideline

Rockville, MD: US Department of Health and Human

Services Public Health Service, May 2008.

19 Wyatt J and Spiegelhalter D. Evaluating medical expert

systems: what to test and how? Medical Informatics 1990;

15:205–17.

20 Kaplan B. Evaluating informatics applications: clinical

decision support systems literature review. International

Journal of Medical Informatics 2001;64:15–37.

21 Kaplan B and Shaw N. Future directions in evaluation

research: people, organizational, and social issues.

Methods of Information in Medicine 2004;43:215–31.

22 Kaplan B. Addressing organizational issues into the

evaluation of medical systems. Journal of the American

Medical Informatics Association 1997;4:94–101.

23 Marcy T, Skelly J, Shiffman R and Flynn B. Attitudes and

opinions of physicians and clinic managers towards

clinical decision support systems to improve adherence

to the tobacco use treatment guidelines. Preventive

Medicine 2005;41:479–87.


24 Venkatesh V and Davis F. A theoretical extension of the

technology acceptance model: four longitudinal field

studies. Management Science 2000;46:186–204.

25 Rogers E. Diffusion of Innovations (5e). New York, NY:

Free Press, 2003.

26 Kaplan B, Drickamer M and Marottoli R. Deriving

design recommendations through discount usability

engineering: ethnographic observation and thinking-

aloud protocol in usability testing for computer-based

teaching cases. In: Musen M (ed) Proceedings of the 2003

AMIA Annual Symposium. Bethesda, MD: AMIA, 2003,

pp. 346–50.

27 Kaplan B and Maxwell J. Qualitative research methods

for evaluating computer information systems. In:

Anderson J and Aydin C (eds) Evaluating the Organ-

izational Impact of Healthcare Information Systems. New

York: Springer, 2005, pp. 30–55.

28 Smith III A. Design and conduct of subjectivist studies.

In: Friedman C and Wyatt J (eds) Evaluation Methods in

Medical Informatics. New York: Springer, 1997, pp. 223–

53.

29 Hing E and Burt C. Office-based Medical Practices:

methods and estimates from the National Ambulatory

Medical Care Survey. Advance Data from Vital and

Health Statistics. Hyattsville, MD: National Center for

Health Statistics, 2007.

30 Michel G, Marcy T and Shiffman R. A wireless, handheld

decision support system to promote smoking cessation

in primary care. In: Friedman C (ed) Proceedings of the

American Medical Informatics Association Symposium,

2005. Bethesda, MD: AMIA, 2005, pp. 530–4.

31 Jaen C, Stange K, Tumiel L and Nutting P. Missed

opportunities for prevention: smoking cessation coun-

seling and the competing demands of practice. Journal of

Family Practice 1997;45:348–54.

32 Yarnall K, Pollak K, Ostbye T, Krause K and Michener J.

Primary care: is there enough time for prevention?

American Journal of Public Health 2003;93:635–41.

CONFLICTS OF INTEREST

None.

ADDRESS FOR CORRESPONDENCE

Theodore W Marcy MD MPH

Office of Health Promotion Research

University of Vermont College of Medicine

1 South Prospect Street, 4th Floor Arnold

Burlington

Vermont 05401–3444, USATel: +1 802 656 3650

Fax: +1 802 656 8826

Email: [email protected]

Accepted May 2008

Preliminary findings reported in this paper were presented in an abstract at the 2006 American Society of

Preventive Oncology and in an abstract at the 2006 Translating Research into Practice and Policy (TRIPP)

conference.

Developing a decision support system for tobacco use ...

Documents