ISSN 1713-1383

ISSN 1713-1383

Copyright © 2004. HEC Montréal.Tous droits réservés pour tous pays. Toute traduction et toute reproduction sous quelque forme que ce soit est interdite. HEC Montréal, 3000, chemin de la Côte-Sainte-Catherine, Montréal, Québec, H3T 2A7 Canada. Les textes publiés dans la série des Cahiers de la Chaire de recherche du Canada en technologie de l’information dans le secteur de la santé n'engagent que la responsabilité de leurs auteurs.

Toward a Multidimensional Assessment of PACS Success

Par : Guy ParéLuigi LepantoDavid AubryClaude Sicotte

Cahier de la Chaire de recherche du Canada en technologie de l’information dans le secteur de la santéNo 04-01 - Juillet 2004

Toward a Multidimensional Assessment of PACS SuccessGuy Paré, Luigi Lepanto, David Aubry, and Claude Sicotte


Guy Paré, Ph.D.Canada Research Chair in Information Technology in Health Care

HEC Montréal3000, chemin de la Côte-Ste-Catherine

Montréal (Québec)Canada H3T 2A7

Phone : (514) 340-6812Fax : (514) 340-6132

Email: [email protected]

Luigi Lepanto, M.D.Centre hospitalier de l’Université de Montréal

Department of Radiology1058, rue St-DenisMontréal (Québec)Canada H2X 3J4

Phone: (514) 890-8350 ext. 35606Fax: (514) 412-7359


David Aubry, M.Sc.HEC Montréal

3000, chemin de la Côte-Ste-CatherineMontréal (Québec)Canada H3T 2A7

Phone : (514) 340-6476Fax : (514) [email protected]

Claude Sicotte, Ph.D.Health Administration Department

Faculty of MedicineUniversity of Montreal

C.P. 6128 Downtown StationMontréal (Québec)Canada H3C 3J7

Phone: (514) 343-5611Fax: (514) 343-2448


Copyright © 2004. HEC Montréal. 2

mailto:[email protected]






Abstract

A picture archiving and communications system (PACS) is an integrated workflow system for managing images and related data, which is designed to streamline operations throughout the whole patient care delivery process. PACS has become a mature technology over the past few years, and has been widely implemented in several developed countries. Evaluation of PACS success is a major challenge to healthcare organizations. A review of previous PACS research suggests a fragmented and focused evaluation approach, thus offering limited discussion of comprehensive views of PACS success or systematic and practical guidance to its evaluations. Based on two prevalent information systems success models, this paper proposes and describes an integrated framework for evaluating PACS success in hospital settings. It details the validation process of the proposed model and its related measurement instrument at a large tertiary-care teaching hospital in Canada. Future research directions that extend the proposed

model are highlighted.

Keywords: PACS, system success, evaluation study, hospital.


HEC, 07/19/04,

check

HEC, 07/19/04,

to avoid research twice in the same sentence.


1. Introduction

PACS has become an important component of many radiology departments and hospitals around the world [1]. A large number of studies have attempted to identify those factors that contribute to PACS success [e.g., 2, 3]. Results from these studies [e.g., 4, 5] clearly reveal that, ultimately, in order for a PACS to succeed, healthcare organizations and managers must adequately address various types of challenges: technological (e.g., integration with other information systems), managerial (e.g., project management), organizational (e.g., availability of resources), behavioural (e.g., change management) and political (e.g., alignment among key participants).

However, the dependent variable in these studies – PACS success – remains undefined. Different researchers have addressed different aspects of success, making comparisons difficult, and the prospect of building a cumulative tradition for research similarly elusive. Most investigations have considered a single or, at best, a small number of factors, contributing to a fragmented view of PACS success. Broadly, these studies may be classified into those that consider the impact of PACS on radiologists’ workload and productivity [6], those that consider its clinical implications [7], and those associated with performance of the radiology department [8]. In short, past empirical and evaluative studies have provided limited discussion of conceptual frameworks for holistic or comprehensive understanding of PACS success or systematic and practical guidance to its operationalization.

To organize this diverse research, as well as to present a more integrated view of the construct of PACS success, a comprehensive success model is introduced. Our aim is to synthesize previous research into a more coherent body of knowledge, and to provide guidance to managers, clinicians and researchers.

Evaluation of system success or effectiveness has been a fundamental issue and dominant focus in information systems (IS) research over the past 30 years. Since PACS is a particular or specialized form of system, a logical and reasonable departure point for evaluating its success is the relevant IS literature. From this perspective, the proposed multidimensional model of PACS success is based on DeLone and McLean’s IS success framework [9, 10] which has emerged as a dominant model for system evaluation research. Our integrated model also comprises key constructs from a complementary framework, namely, Battacherjee’s IS continuance model [11].

The remainder of this paper is organized as follows. Section 2 reviews relevant previous PACS evaluation studies and highlights the research motivation. Section 3 provides an overview of Delone and McLean’s success framework, as well as that of Battacherjee and concludes with the presentation of the resulting model for evaluating PACS success. Section 4 describes the operationalization of the success model in terms of methods and measures. In section 5, we detail the adopted research design and methods in order to validate the proposed model and the related measures, as well as test the strength of the relationships between variables. Data analyses are presented in section 6, followed by a discussion of the key findings in section 7. The paper concludes with a summary and discussion of implications for future research in section 8.



2. Literature review and research motivation

Evaluation represents a critical issue in PACS research and practice. Unfortunately, in searching for a PACS success measure, rather than finding none, one discovers that there are nearly as many measures as there are studies. The reason for this is understandable when one considers that “information,” as the output of an information system such as a PACS, can be measured at different levels, including the technical level, the semantic level and the effectiveness level. Shannon and Weaver [12] defined the technical level as the accuracy and efficiency of the system which produces the information, the semantic level as the success of the information in conveying the intended meaning, and the effectiveness level as the effect of the information on the receiver.

As reported in DeLone and McLean [9], the three levels of information of Shannon and Weaver are shown to yield six distinct categories or aspects of PACS success. As shown in Table 1, these categories are system quality, information quality, use, user satisfaction, individual impact, and organizational impact.

Table 1. Categories of PACS success variables

Shannon & Weaver

Categories of IS success variables

Technical level System qualitySemantic level Information qualityInfluence level Use

|User satisfaction

|Individual impact

|Organizational impact

Looking at the first of these categories, some researchers have chosen to focus on the desired attributes of the PACS itself. For instance, Cox and Dawe [13] examined the speed of image availability, the ease of use of the system, and the frequency of system breakdown. Gale et al. [6] evaluated several aspects of the quality of a PACS interface design. For their part, Tucker et al. [14] considered several image issues, integration of PACS with radiology information system (RIS) and hospital information systems (HIS) to be an important component of PACS success or failure.

Rather than measure the quality of the PACS performance, other researchers have preferred to focus on the quality of the information that the system produces, primarily in the form of images and reports. For instance, Lou et al. [15] considered the data integrity and completeness of acquired images. High quality images in terms of timeliness, accuracy, completeness, etc. were also considered to be a key success factor in several evaluative studies [e.g., 13, 16, 17, and 18].

Another dimension of quality, not presented in Table 1, has also been studied in prior PACS research, namely, service or support quality. Indeed, a few investigators found that



the technical support is a key aspect of PACS success; regardless of whether this service is delivered by an internal IS department or outsourced to an external service provider. Technically competent staff and a realistic time commitment were found to be associated with PACS success [4, 13, and 14].

At the influence level, some researchers have analyzed the interaction of the information project with its recipients, namely, the radiologists, technologists and clinicians, by measuring use and user satisfaction. Fundamentally, the use of a PACS is central to its success. Some studies have computed actual use (as opposed to reported use) by radiologists, technologists and clinicians through hardware monitoring which recorded the connected time per day and the number of study retrievals [19], the number of system functions utilized and the amount of time spent interpreting/reviewing images [20].

Perceived usage was also considered in several evaluative studies. For instance, Tamm et al. [21] asked oncology clinicians the number of studies and radiology reports they viewed per week and the amount of time they spent reviewing each study’s image. With regard to user satisfaction, Bryan et al. [22] studied the major causes of radiologists’ satisfaction and dissatisfaction (frustration) with the PACS. Users’ expectations have also been studied widely. For example, Baumann and Gell [4] conducted a longitudinal survey of 1,000 facilities around the world. Clinicians reported that their expectations of the PACS had been met in 81% of cases and 97% of the users would recommend PACS to others. More recently, Pilling [17] assessed the acceptability to radiologists of a PACS. Respondents judged that PACS had made a positive change in their working practices and had met their expectations.

Undeniably, impacts, whether at the individual or the organizational level, represent the most widely used construct of PACS success. A vast majority of researchers have been interested in the influence which the PACS has on users. For instance, Bryan et al. [22], Kato et al. [23], and Reiner et al. [3] investigated the impact of PACS on radiologists’ productivity and report/ interpretation time. Hertzberg et al. [7] examined the relative accuracy of interpretation of sonography when viewed on a PACS workstation or on film. Other researchers have studied the influence of PACS on technologists’ productivity. For instance, Reiner and Siegel [24] assessed the impact of filmless operation and computed radiography on technologists’ examination times compared with conventional film-screen radiography.

Finally, researchers have been concerned with workflow and other performance issues. For instance, Hayt et al. [8] studied the impact of a PACS on radiology operations and service at a large urban hospital. Precisely, they found that, with the aid of a PACS, the hospital gained complete control of a runaway film problem and report turnaround time changed from being completely unacceptable to acceptable. In the same vein, Mattern et al. [25], Pavlicek et al. [16] and Weatherburn et al. [26] examined the impact of electronic imaging on several outcome measures including image reject rates, time to final diagnosis, time to final treatment and need for follow-up. Blado et al. [27], collected data on rejected images and images from repeated examinations. Dackiewicz et al. (2000) examined the influence of digital radiography on clinical workflow and patient satisfaction. Redfern et al. [29] evaluated the relationship between patient volume and



workflow for radiologists who began to interpret images from multiple clinical sites after the introduction of a PACS. Reiner et al. [30] studied the impact of filmless operation on the relative frequency of in-person consultations in the radiology department between radiologists and clinicians. As a final example, Weatherburn and Bryan [31] examined whether the doses for the radiographic examination of the lateral lumbar spine changed as a result of the introduction of a hospital-wide PACS.

In conclusion, once this expanded, but rather fragmented view of PACS success is recognized, it is not surprising to find that there are so many different measures of this success in the literature depending on the aspect of PACS on which the researcher focused his or her attention. As mentioned earlier, a review of relevant prior IS success frameworks may shed light on the needed comprehensive view of PACS success as well as systematic and practical guidance for its evaluations. In particular, the IS success framework by DeLone and McLean [10] as well as Chattaberjee’s IS continuance model [11] have emerged as important and prevalent frameworks for evaluating IS success. The following section describes both models upon which the proposed integrative model of PACS success is developed.

3. An integrated model of PACS success

Motivated by the need for a comprehensive framework for advancing and integrating IS research findings, DeLone and McLean [9] postulated an IS success framework. Based on the communication theory of Shannon and Weaver [12], the information influence theory of Mason [32], and a fairly comprehensive synthesis of the important system evaluation research conducted between 1981 and 1987, the original model, which was published in 1992, offers a multidimensional lens to IS success and, at the same time, singles out a set of common measurements for each success dimension.

Since the publication of the original framework, about 300 articles in refereed journals have referred to, and made use of, this IS success model. Several empirical studies explicitly tested the relationships among the variables identified in the original model [e.g., 33, 34, and 35]. Yet, other studies have implicitly tested the model by investigating multiple success dimensions and their interrelationships [e.g., 36, 37]. Taken as a whole, these studies gave strong support for the proposed associations among the IS dimensions and helped to confirm the causal structure in the model. Judged by its frequent citations in articles published in leading IS journals, this framework has become a dominant evaluation model in IS research.

Based on research contributions since the publication of the model, DeLone and McLean updated their original success framework in 2003. The model in Figure 1 indicates that success of an information system is multi-dimensional and can be represented by the quality characteristics of the system itself (SYSTEM QUALITY); the quality of the output (INFORMATION QUALITY); the quality of the technical support or service (SERVICE QUALITY); the consumption of the output of the system (USAGE); the user’s response to the system (USER SATISFACTION); and, ultimately, the positive impacts the system has (NET BENEFITS).



Figure 1. DeLone and McLean’s revised success model

DeLone and McLean’s revised model makes two important contributions to the understanding of IS success. First, it provides a scheme or a framework for categorizing the multitude of IS success measures that have been used in the literature. Second, it suggests a model of temporal and causal interdependencies between the categories.

Two constructs were added to DeLone and McLean’s model to recognize complementary research findings in the IS field. DeLone and McLean are primarily concerned with acceptance behaviours, namely, use (or intention to use). While usage represents an important indicator of system success, long-term viability and its eventual success depend on its continued use. As explained by Bhattacherjee [11], IS continuance is not an alien concept in IS research. Indeed, many studies have acknowledged the existence of a post-acceptance stage when IS use transcends conscious behaviour and becomes part of normal routine activity. Innovation diffusion theory suggests that adopters eventually re-evaluate their earlier acceptance decision and decide whether to continue or discontinue using an innovation [38]. In line with such reasoning, like Bhattacherjee, we think it is important to differentiate between acceptance and continuance behaviours and, hence, we include system continuance intention as the ultimate dependent variable in our own success model (see Figure 2).

The model tested by Battacherjee [11] is based on expectation-confirmation theory (ECT) [39]. Per ECT, users’ IS continuance intention is determined primarily by their satisfaction with prior IS use and their perceived usefulness of IS use (perceived net benefits). Therefore, as shown in Figure 2, both user satisfaction and net benefits are associated with system continuance. Lastly, we posit that PACS continuance intention is influenced (both directly and indirectly) by another construct, namely, confirmation of expectations following actual use of the system. Confirmation is positively related with system continuance (and user satisfaction) because it implies the realization of the expected benefits of IS use, while disconfirmation denotes failure to achieve expectation [11].



Figure 2. An integrated model of PACS success

As depicted in Figure 2, the resulting model comprises eight interrelated dimensions of PACS success: perceived system quality, perceived information/image quality, perceived service quality, system use, user satisfaction, perceived net benefits, confirmed expectations, and system continuance intention.

“System quality,” in a PACS environment, measures the desired characteristics of a PACS such as its reliability, ease of use, availability, security, and response time. Information/image quality captures the content issue of a PACS. Patient information and images produced by a PACS must be precise, understandable, complete, and available on time, to name a few, if we expect radiologists and other groups of adopters to use it. “Service quality,” the overall support delivered by the service provider, applies regardless of whether this support is delivered by the internal IT department or outsourced to a PACS service provider. Consistent with that commonly defined clinically, service quality can be examined in terms of service consistency, reliability, timeliness, empathy, assurance, and accuracy or adequacy.

Next, we concur with DeLone and McLean that “system usage” is an appropriate indicator of success in most IT implementation projects, and PACS is no exception. “User satisfaction” remains an important means of measuring users’ opinions about PACS. “Net benefits” are the most important success measures, as they capture the balance of positive and negative impacts of PACS on radiologists, technologists, physicians and hospitals in general. “Net benefits” success measures are most important, but they cannot be analyzed and understood without “system quality,” “information/image quality,” and “service quality.” Next, as explained above, “confirmed expectations” are positively related to “user satisfaction” with PACS use because it implies realization of the expected benefits of the system. Lastly, while initial acceptance (usage) of PACS is an important first step toward realizing success, long-term viability of a PACS and its eventual success depend on its continued use “system continuance” rather than first-time use. It is then hypothesized that system continuance will be positively associated with user satisfaction, net benefits and confirmed expectations. In short, the


PERCEIVED

SYSTEM QUALITY

PERCEIVED INFORMATION

QUALITY

PERCEIVED SERVICE QUALITY

USER

SATISFACTION

PERCEIVED

NET BENEFITS

CONFIRMED

EXPECTATIONS

SYSTEM USAGE

SYSTEM CONTINUANCE

INTENTION

H14 (NOT TESTED)

H1

H2

H3

H4

H5

H6

H7

H8

H10

H9

H12

H11

H13


hypotheses that follow directly from the proposed model are summarized in Table 2. As mentioned earlier, three groups of users are most affected by the introduction of PACS in a hospital environment, namely, radiologists, clinicians, and radiology technologists. The interaction of each group with PACS, as well as the impact of PACS on each group differs. The radiologists represent the group whose entire work environment and work practices are changed by PACS. The same can be said of clinicians, when analyzed from the perspective of their interaction with the radiology department and their use of medical images. Of course, clinicians perform many other tasks that are not related to medical imaging, but in their use of images for diagnosis and treatment, the tools at their disposal have been replaced. On the other hand, technologists interact with PACS in a more limited way. Specifically, only that part of their work which involves the production and handling of film is replaced by PACS. Technologists are not involved in image interpretation, except at the level of ensuring minimal quality assurance standards. This explains why certain research hypotheses listed in Table 2 do not apply to them.

Table 2. Research hypothesesR T C

H1 Use of PACS is positively associated with perceived quality of the system.

● ●

H2 Users are more satisfied with PACS of higher perceived system quality.

● ● ●

H3 Use of PACS is positively associated with perceived information quality.

● ●

H4 Users are more satisfied with PACS of higher information quality. ● ●H5 Use of PACS is positively associated with perceived service quality. ● ●H6 User satisfaction is positively associated with PACS service quality. ● ● ●H7 Levels of user satisfaction and levels of PACS use are mutually and

positively associated.● ●

H8 Users’ extent of confirmation is positively associated with their satisfaction with PACS use.

● ● ●

H9 Perceived net benefits are positively associated with PACS use. ● ●H10 Perceived net benefits are positively associated with user satisfaction. ● ● ●H11 Users’ level of satisfaction with PACS usage is positively associated

with their PACS continuance intention.● ● ●

H12 Perceived net benefits are positively associated with users’ PACS continuance intention.

● ● ●

H13 Users’ extent of confirmation is positively associated with their PACS continuance intention.

● ● ●

Legend: R=radiologists; T=technologists; C=clinicians

4. Operationalization of the proposed model

The next step consisted in reviewing the IT and digital imaging literature in search of specific perceptual measures for the various constructs included in the conceptual model.

In and of itself, perceived system quality is a multi-dimensional construct. For one thing, it comprises ease of use, that is, the extent to which learning and using a system is free from effort. Perceived system sophistication represents another key aspect of system quality. Put simply, it represents the perceived diversity and quality of functionalities



offered by the PACS. Ease of access to PACS, both onsite and offsite, represents another dimension of perceived system quality. Next, reliability of the hardware and software components of a PACS is also key to its success. Reliability mainly refers to perceived frequency of system failures and breakdowns (hardware component) as well as perceived number of “bugs” contained in the system (software component). Response time (in terms of image downloading and visualizing) represents another dimension of system quality. Extent of PACS integration with RIS and other hospital information systems has also been identified as a key dimension of PACS quality [14]. Lastly, security also represents a fundamental aspect of PACS quality when one considers that the overall damages and costs associated with a destroyed PACS archive storage and server are comparable to losing the entire onsite film archive of the hospital department. As a consequence, adequate security procedures should include both data redundancy as well as PACS data recovery [40]. In short, all measures developed for the system quality construct were original scales, except for the ease of use dimension which was adapted from Seddon and Kiew [34].

Perceived information quality refers to the quality of the images and patient information produced by the PACS. Several aspects are essential with regard to information quality, namely, timeliness, accuracy, completeness, ease of understanding or interpretability, and relevance [27, 41]. The scale used to measure the quality of the images and information generated by a PACS was adapted from Doll and Torkzadeh [42]. The overall quality of the images produced by the system was also assessed using Pilling’s original measure [17].

Service quality, in the context of PACS implementation, refers to the perceived quality of the support and service provided by the provider of the system and/or the internal staff responsible for PACS support and maintenance. This construct was measured using a 13-item scale developed in the marketing area [43] and then adapted to the IT context [44].

PACS usage, from a perceptual standpoint, can be measured using a variety of dimensions and measures common to technology acceptance or adoption studies. First, intensity of use is frequently used as a measure of system success. By reflecting the amount of time engaged with the technology, intensity clearly relates to the technology’s degree of embeddedness. We propose to measure it as the amount of time spent using the system (as a self-reported value). Precisely, we ask respondents to indicate the average number of hours they spend using the PACS per week and which percentage this time represents of their work. The said measure was adapted from Seddon and Kiew [34]. Second, frequency of use is also often used as a success criterion. Frequency of use was adapted from Raymond [45]. This measure provides a perspective of use slightly different than time. It is measured on a seven-point Likert-type scale ranging from “less than once a day” to “several times a day.”

A third dimension refers to the various PACS functionalities or features used by the users (e.g., access to images from home via Internet; split screen functionality; personalized configuration of toolbar). Scope is therefore defined as the degree to which the PACS is used for a variety of purposes. It is based on the theory that as an individual appropriates a technology for more purposes, it becomes a greater part of that



individual’s work system [46]. A new scale was developed in order to capture this dimension of PACS usage.

Next, user satisfaction refers to the degree to which a user is satisfied with his or her overall use of the PACS. Collective findings from prior IS research have suggested that user satisfaction is a strong and critical manifestation of systems success. A four-item scale developed by Battacherjee [11] was adapted to the PACS context to measure radiologists’, technologists’ as well as physicians’ satisfaction with the system. This measure has demonstrated high psychometric qualities in prior IS studies.

Perceived net benefits represent another multidimensional success construct. To evaluate the users’ perception of the impacts of the PACS, their views were sought on whether there was a comparative improvement, pre- and post-PACS deployment. Table 3 synthesizes the various benefits radiologists, technologists, and clinicians were asked to rate. The questions that emerged from the instrument development phase use various formats, including:

Seven-point Likert scales, in asking for judgements on factors such as perceived speed of clinical decision making;

Simple quantitative responses to questions like how much time does the PACS save you;

Open-ended questions, to obtain a broader perspective on the respondents’ views about the benefits of a PACS.

Next, confirmed expectations basically refer to the users’ perception of the congruence between expectation of PACS use and its actual performance. This construct was measured using a three-item scale adapted from Bhattacherjee’s [11]. Lastly, system continuance intention, that is, users’ intention to continue using the PACS, was measured using a two-item scale also developed by Batthacherjee [11].

Table 3. Measures of perceived net benefitsR T C

Speed of image availability ● ●Number of lost images ● ●Number of unread studies ● ●Speed of clinical decision making ● ●Overall report turnaround time ● ●Number of repeated examinations ● ● ●Number of rejected images ● ●Time devoted to image searching ● ●Time devoted to quality control ●Accuracy of diagnoses ● ●Number of patients who move through the procedure room per hour ●Overall personal productivity ● ● ●Radiologists/clinicians relations ● ●Clinicians/patients relations ●Overall quality of patient care ● ●Quality of work life ● ● ●Legend: R= radiologist survey; T= technologist survey; C= clinician survey.

5. Methodology



In order to test the content validity of the proposed research model, a series of in-depth interviews were first conducted with representative respondents at the Centre hospitalier de l’Université de Montréal (CHUM) where a PACS was implemented throughout the year 2002. CHUM is a multi-site, tertiary teaching facility composed of Hôtel-Dieu, Hôpital Notre-Dame and Hôpital St-Luc. The facilities are currently located on multiple sites, with plans to move to a single new facility scheduled to open in 2010. The academic medical center has over 1,400 licensed beds housed in its three campuses. CHUM counts over 900 physicians, 47 radiologists, and over 150 radiology technologists. Lastly, over 365,000 radiology exams are produced each year.

Given that different stakeholders, having different needs and interests, may attribute different outcomes to the PACS, may ignore outcomes they don’t want to think about, and may evaluate the “same” outcomes differently, interviews were then conducted with twelve representatives from the three groups of users as well as PACS managers. Conclusively, the overall success model shown in Figure 2 appeared to characterize well the reality of a PACS’ success in a hospital context. Indeed, all variables included in the model were identified by at least one respondent although the most referenced or cited dimensions included system quality, information/image quality, user satisfaction and perceived net benefits.

Next, in order to refine further our questionnaire instruments, a pre-test was administered to a relatively small number of potential respondents. The primary objective was to have additional feedback on the content of each measure before distribution to the potential respondents. Interviews were then conducted with three residents. All three reviewers were very thorough in their comments and several suggestions were offered to improve the wording of the scales. In fact, most of the changes made specifically affected the format of the instruments without affecting their substance.

A full-scale survey was recently conducted at the CHUM in order to assess the reliability and validity of our success measures as well as the strength of the relationships between the various constructs. As explained earlier, a distinct questionnaire was then built for radiologists (n=47), technologists (n=160), and clinicians (n=649). All measures included in the three versions of the questionnaire were identical except for usage (scope) and perceived net benefits whose items were tailored to each group of respondents.

Subjects received, through internal mail, a packet that contained a cover letter, a questionnaire, and a return envelope. Participation was voluntary, and respondents were assured that their individual responses would be treated as confidential. Four weeks later, a reminder letter was sent to all participants, requesting that those who had not yet participated complete the questionnaire. A total of 232 questionnaires were returned to the researchers (27% return rate). While this is lower than desired, it is not unusual for large scale surveys. Among the returned questionnaires, 24 (51% response rate) were completed by radiologists, 77 (48% response rate) by technologists, and 131 (20% response rate) by physicians. Note that fourteen questionnaires returned by physicians were removed from our database due to missing data, leaving us with a final sample of 218 responses.



Of these usable questionnaires, 36.4% were received from Saint-Luc Hospital, 36.9% from Notre-Dame Hospital, and 26.6% from Hotel-Dieu Hospital. The age of the respondents follows a normal distribution and the sample was equally constituted of men and women. As expected, only 10.6% of the sample had prior experience with a PACS. Demographic data about the respondents in the final sample are shown in Table 4.

In order to assess non-response bias, questionnaires returned after the given deadline (four weeks after the mailing) were treated as non-responses. T-tests on demographics and key constructs of the study showed non significant differences between respondents and non respondents for all three groups (radiologists, technologists, and clinicians) of respondents. Even though this is a commonly used method to assess non-response bias, the possibility of bias is not entirely eliminated and results should be interpreted accordingly.

Table 4. Profile of the respondentsRadiologists

(n=24)Technologists

(n=77)Clinicians(n=117)

Overall(n=218)

Age20-30 years 0.0% 20.0% 2.6% 8.4%31-40 years 17.4% 22.7% 30.2% 26.2%41-50 years 56.5% 37.3% 36.2% 38.8%51-60 years 17.4% 20.0% 23.3% 21.5%Over 61 years 8.7% 0.0% 7.8% 5.1%

SexMale 52.2% 12.0% 73.0% 49.3%Female 47.8% 88.0% 27.0% 50.7%

Site (campus)St-Luc 47.8% 40.0% 31.9% 36.4%Notre-Dame 39.1% 34.7% 37.9% 36.9%Hotel-Dieu 13.0% 25.3% 30.2% 26.6%

Prior experience with PACSYes 30.4% 3.9% 11.0% 10.6%No 69.6% 96.1% 88.9% 89.4%

6. Data analysis

Data analysis began with an examination of the measurement model in terms of its reliability and discriminant validity. Table 5 presents the results associated with the assessment of the internal consistency of each scale. The composite reliability coefficients of all the measurement scales but two, satisfied Nunally’s [47] guidelines. Only integration and scope of use (radiologists only) showed a weak reliability coefficient of 0.66. For integration (in the radiologists’ and technologists’ questionnaires), the low alpha is a result of the scale not having enough variability. Based on the results of the reliability analysis and the inter-item correlation coefficients matrix (not shown here), no item was removed from the measurement instruments.



Table 5. Internal consistency results.

Construct Variable# of

itemsAlpha

System Quality Ease of use 8 .91Usefulness 3 .83Integration 2 .66Reliability 5 .86Ease of access 3 .79Interface quality 6 .78Rapidity 6 .78

Information / Image Quality Information quality 6 .91Image quality 8 .91

Service Quality 13 .97Usage Intensity 2 .83

Frequency 1 -Scope – Radiologists 8 .66Scope – Physicians 6 .72

User Satisfaction 4 .94Net Benefits Radiologists and physicians1 19 .80

Technologists 6 .92Confirmed Expectations 2 .77System Continuance Intention 3 .821 A combined measure was developed in order to satisfy the usual requirement of at least five

times as many respondents as items [47: p.262). We intend to validate the success model in Figure 2 in other hospital settings and collect sufficient data as to test the reliability of our original measures (radiologists: 23 items; physicians: 27 items).

Following the assessment of the measurement model, descriptive statistics were computed. Table 6 presents the means and standard deviation of the main constructs in the study for the three groups of respondents. This table also provides the results of the Mann-Whitney U-test, which tested differences between the populations of respondents on these constructs.

Table 6. Descriptive statistics

VariableRadiologists Technologists Clinicians

Mann Whitney or Kruskal Wallis test

Mean Std. Dev.

Mean Std. Dev.

Mean Std. Dev.

Chi-Square

Asymp.Sig.

Ease of use 5.5 0.9 - - 5.1 1.0 2.266 .132Usefulness 4.8 1.2 - - 4.7 1.2 .166 .684Integration 2.8 1.7 5.1 1.2 3.3 1.6 66.976 .000Reliability 5.6 1.0 4.4 1.1 4.9 1.1 23.685 .000Ease of access 4.5 1.4 - - 4.2 1.4 .404 .525Interface quality 4.5 1.0 - - 3.9 1.1 5.185 .023Rapidity 4.4 1.2 - - 3.9 1.1 2.430 .119Information quality 5.3 1.1 - - 5.2 1.0 .180 .671Image quality 5.8 0.5 - - 5.0 1.0 13.360 .000Service quality 5.0 1.3 5.1 1.2 4.9 1.3 1.781 .410Intensity of use1 25.5 12.2 - - 5.6 8.3 45.696 .000

Table 6. Descriptive statistics (cont’d)



VariableRadiologists Technologists Clinicians

Mann Whitney or Kruskal Wallis test

Mean Std. Dev.

Mean Std. Dev.

Mean Std. Dev.

Chi-Square

Asymp.Sig.

Frequency of use 6.7 0.8 - - 5.3 1.9 13.313 .000Scope of use2 4.9 1.0 - - 2.2 0.8 - -User satisfaction 5.6 0.9 5.4 1.0 5.0 1.2 7.241 .027Net benefits3 5.1 1.0 5.1 0.8 5.0 0.9 0.24 .877Confirmed expectations 5.3 1.0 5.3 1.0 4.9 1.2 8.601 .014Continuance intention 6.7 0.6 6.2 1.0 6.2 0.9 12.385 .0021 Scales for all constructs, but Intensity of use, are 1 (low value) to 7 (high value). Scale for “Intensity of use” refers

to the average number of hours per week spent working with the PACS.2 Scales for Scope of use differed between groups and, hence, comparisons of means were not computed for these

constructs.3 As explained earlier, a combined measure was used for radiologists and clinicians. The comparison of means test

concerns these two groups of respondents only.

With regard to system quality, radiologists, technologists, and clinicians differ significantly on their scores on the integration and reliability dimensions. Precisely, technologists perceive the PACS to be more integrated to other patient-related systems than radiologists and clinicians. Radiologists are however those who find the PACS to be most reliable. To a lesser extent, radiologists have more positive behavioral beliefs than clinicians with regard to the quality of the PACS’ interface. Further, radiologists and clinicians did not differ significantly on their scores on the ease of use, usefulness, ease of access and rapidity dimensions. Both groups perceive the PACS to be relatively easy to use, useful, easy to access and rapid.

Furthermore, it is interesting to note that while radiologists and clinicians have similar (positive) beliefs with regard to the quality of the patient-based information produced by the PACS, their perceptions differ with regard to the quality of the digital images. Indeed, radiologists have significantly more positive beliefs than clinicians on this dimension. In addition, although radiologists use the PACS more intensively and frequently than clinicians, both groups have similar perceptions with regard to the net benefits obtained from the use of the technology.

Lastly, all three groups of respondents differ significantly on their scores on the remaining constructs, that is, user satisfaction, confirmed expectations, and system continuance intention. Even though, overall, all three groups view the adoption of a PACS positively, the mean scores indicate that radiologists and technologists seem to be more satisfied and their expectations to be met at a higher level than clinicians. Scores on behavioral intention show that overall users will continue using PACS (mean = 6.3). The clinicians and technologists scores for behavioral intention are significantly lower than radiologists scores, but are quite above the neutral point on the scale.

Next, hypothesis testing was evaluated using linear (stepwise) regression coefficients. More advanced statistical approaches such as Partial Least Squares (PLS) and Structural Equation Modeling (SEM) could not be used because of the small number of radiologists and technologists in our sample. The regression results are shown in Tables 7 through 9 for radiologists, technologists, and clinicians, respectively.



Table 7 presents results pertaining to the relationships of the predictors of system continuance, perceived net benefits, and user satisfaction for radiologists. First, findings reveal that user satisfaction and, to a lesser extent, confirmed expectations, have significant positive effects on radiologists’ intention to continue PACS usage. On the other hand, the data show that perceived net benefits were not found to have a significant and direct effect on system continuance intention for this group of professionals. The study variables explained forty-one percent of the variance in system continuance intention. Second, twenty-three percent of the variance in perceived net benefits was explained by the only hypothesized predictor, namely, user satisfaction. Third, a linear regression of predictors on radiologist’s satisfaction with the usage of PACS was conducted. The model explains seventy-nine percent of the variance in the criterion variable, although only two of the twelve predictors were statistically different from zero. The standardized regression coefficients show that system reliability and ease of use had significant and positive effects on radiologists’ satisfaction with the PACS. On the other hand, all of the other system quality variables (rapidity, usefulness, integration and ease of access) as well as the quality of the information, the quality of the technical service and the extent to which expectations were met had no significant direct effect on user satisfaction.

Table 7. Linear regression of independent variables on system continuance intention, perceived net benefits, and user satisfaction (radiologists)

Criterion variable

Independent variable Parameter estimate

SE Standardized coefficients

t-value p<

System continuance intention

Intercept 5.071 .687 7.377 .000User satisfaction ** .498 .143 .668 3.474 .003Confirmed expectations *

.308 .125 .525 2.467 .025

Net benefits .149 .622 .543Overall model. F=12.068. p<.005; R2=.446; Adjusted R2=.409

Net benefits Intercept 2.951 .862 3.423 .003User satisfaction * .367 .148 .528 2.484 .024Overall model. F=6.172. p<.05; R2=.278; Adjusted R2=.233

User satisfaction

Intercept .303 .932 .326 .750Reliability *** .801 .128 .851 6.245 .000Ease of use ** .609 .172 .591 3.545 .004Information quality .115 .569 .580Interface quality .086 .329 .748Rapidity .114 .543 .597Confirmed expectations .252 1.503 .159Image quality .230 1.733 .121Usefulness .230 1.678 .132Integration .258 1.857 .100Ease of access .206 1.194 .267Rapidity .114 .702 .503Service quality .229 -2.104 .069Overall model. F=38.996. p<.001; R2=.812; Adjusted R2=.792

*** p<.001; ** p<.005; * p<.05



Table 8 reports the results pertaining to the determinants of system continuance intention, perceived net benefits, and user satisfaction for technologists. First, as expected, user satisfaction and perceived net benefits were found to have significant and positive effects on system continuance intention among technologists (R2=.47). On the other hand, the relationship between confirmed expectations and system continuance intention was not supported for this group of professionals. Second, forty percent of the variance in perceived net benefits was explained by the only hypothesized predictor, namely, user satisfaction. Third, the standardized regression coefficients show that, as expected, confirmed expectations, system reliability, and service quality had a significant and positive influence on technologists’ satisfaction with the PACS (R2=.59). However, results indicate that system integration had no significant direct effect on satisfaction for this group of professionals.

Table 8. Linear regression of independent variables on system continuance intention, perceived net benefits, and user satisfaction (technologists)

Criterion variable



t-value p<


Intercept 2.719 .483 5.634 .000User satisfaction ** .380 .114 .390 3.333 .001Net benefits ** .325 .100 .380 3.243 .002Confirmed expectations .151 1.160 .251Overall model. F=29.671. p<.001; R2=.485; Adjusted R2=.469

Net benefits Intercept 1.232 .604 2.039 .046User satisfaction *** .724 .110 .636 6.587 .000Overall model. F=43.387. p<.001; R2=.404; Adjusted R2=.395

User satisfaction

Intercept -.009 .602 -.155 .877Confirmed expectations ***

.594 .108 .540 5.483 .000

Reliability ** .271 .089 .266 3.028 .004Service quality * .193 .092 .199 2.098 .040Integration .020 .194 .847Overall model. F=28.940. p<.001; R2=.608; Adjusted R2=.587

*** p<.001; ** p<.005; * p<.05

Lastly, Table 9 summarizes the results pertaining to the relationships of the predictors of system continuance intention, perceived net benefits, and user satisfaction for clinicians. First, the model explains forty-three percent of the variance in system continuance intention for this group of professionals. Contrary to radiologists, clinicians’ intention to pursue PACS usage was exclusively influenced by perceived net benefits while user satisfaction and confirmed expectations had no significant positive effects on the criterion variable. Second, as expected user satisfaction and, to a lesser extent, intensity of use, were found to have a significant positive effect on perceived net benefits (R2=.33). On the other hand, the other two dimensions of usage, namely, frequency of use and scope of use, had no influence on the extent of perceived benefits. Third, like technologists, clinicians’ satisfaction with usage of the PACS was explained first and foremost by the extent to which their expectations regarding the impacts of the PACS were initially met. Clinicians’ satisfaction was also influenced by one of the system quality dimensions, namely, usefulness. Both variables explained sixty-four percent of



the variance in the criterion variable. On the other hand, the data show that, contrary to our expectations, all other six dimensions of system quality as well as information/image quality, service quality and all three dimensions of PACS usage had no significant and direct effect on clinicians’ satisfaction with the technology.

Table 9. Linear regression of independent variables on system continuance intention, perceived net benefits, and user satisfaction (clinicians)

Criterion variable



t-value p<


Intercept 3.178 .082 7.632 .000Net benefits *** .617 .088 .662 7.013 .000Confirmed expectations .039 .349 .728User satisfaction -.088 -.778 .439Overall model. F=49.184. p<.001; R2=.438; Adjusted R2=.430

Net benefits Intercept 2.395 .444 6.288 .000User satisfaction *** .473 .083 .595 5.713 .000Intensity of use ** .002 .008 .240 2.303 .025Frequency of use .144 1.186 .241Scope of use .101 .812 .421Overall model. F=17.175. p<.001; R2=.360; Adjusted R2=.339

User satisfaction

Intercept .885 .618 1.433 .162Confirmed expectations ***

.503 .142 .535 3.534 .000

Usefulness * .305 .149 .310 2.047 .049Service quality .219 .1.537 .135Information quality .099 .671 .508Image quality .027 .184 .856Ease of use .037 .155 .878Integration .104 .670 .508Reliability .117 .752 .458Ease of access .006 .049 .961Interface quality .142 1.131 .267Rapidity .058 .451 .656Intensity of use .174 1.491 .148Frequency of use .072 .555 .584Scope of use .011 .097 .924Overall model. F=22.075. p<.001; R2=.654; Adjusted R2=.641

*** p<.001; ** p<.005; * p<.05

7. Discussion

Rapidly, PACS technology is becoming a reality in many North American, European, and Asian hospitals. Amid the growing interest in, and implementation of, PACS around the world, it is essential to address the challenge of evaluating PACS success. A review of the digital imaging literature revealed a fragmented evaluation approach, most studies focusing on a single or a small group of success measures. As a consequence, discussion on comprehensive views of system success or systematic guidance to its evaluations is limited. Based on the IS success model by DeLone and McLean as well as the IS



continuance model of Battacherjee, this paper proposed a multidimensional, integrated model for evaluating PACS success from multiple stakeholders’ perspectives.

Several research hypotheses were fully or partially supported in the present study. For one thing, user satisfaction strongly influenced the perceived net benefits of the PACS on all three categories of users, namely, radiologists, technologists, and clinicians. In this study, the more satisfied the users were with the PACS, the more strongly they agreed the system helped them perform well in the context of their job. Again, this result is consistent with the results of prior studies in the IT domain [e.g., 33].

Next, DeLone and McLean [10] suggested that user satisfaction might be interpreted as a response to three types of user aspirations for a system: system quality, information quality, and service quality. Battacherjee [11] further suggested that confirmed expectations were directly related to user satisfaction. Perceptions of system quality, information quality and service quality as well as confirmed expectations should then explain a large proportion of variance in user satisfaction. As explained in detail below, the results from the PACS implementation that was the object of our study only provide partial support for this proposition and factors influencing user satisfaction strongly varied across user types. First, only three dimensions of system quality directly influenced users’ satisfaction. PACS reliability was found to influence both radiologists’ and technologists’ satisfaction with the system. Radiologists’ satisfaction was also influenced by the ease of use of the system while perceived system usefulness, which captures the instrumentality of PACS use, was the only system quality variable that strongly influenced clinicians’ satisfaction with the PACS. With the exception of system reliability, these results are consistent with prior IT research which found that perceived usefulness and perceived ease of use are salient beliefs influence technology acceptance attitudes and behaviours across a broad range of computing technologies and user populations [e.g., 48, 49, and 50]. All four other system quality variables, namely, system integration, ease of access, interface quality and rapidity were not found to influence radiologists’, technologists’ nor clinicians’ satisfaction. The fact that only perceived system usefulness significantly influenced clinicians’ satisfaction with PACS is a reflection of the primary impact of PACS on their work. The ability to have instant access to images from any point in the hospital is the greatest perceived advantage, avoiding time wasting trips to the radiology department or the inconvenience of lost films. The results for radiologists show a concern with efficiency and productivity, since this is best guaranteed by a system that is reliable and easy to use. As mentioned earlier, technicians are not, in actual fact, users; nevertheless, they benefit from the implementation of PACS because the system eliminates the tasks of physically producing and manually handling films. From this perspective the quality of the system is less important than its reliability. It is surprising that reliability was not significantly linked clinician satisfaction since in the implementation studied, all film-based images were eliminated and the PACS assumed the role of a mission critical application. The fact that since the early implementation period there has not been a prolonged and generalized failure of the system may, at least in part, explain this result.

There a number of reasons why system integration, ease of access, interface quality, and rapidity of access did not influence the users’ satisfaction. System integration refers to the linking of different PC based clinical applications at the user level. This requires



synchronising the different applications and ensuring communication between them. CHUM is in the early stages of implementation of some of these clinical information systems and integration is not yet adequate. Although end users recognise the added value of integration, lack of it at this time does not detract from the satisfaction they may feel toward the PACS. For their part, ease of use and rapidity of access were probably appreciated early in after implementation but are now probably taken for granted. And the fact that interface quality was not associated with satisfaction is probably due to the lack of a point of comparison, since for the vast majority of users this was their first experience with PACS.

Second, contrary to our expectations, neither perceived information quality nor perceived image quality generated or produced by the PACS influenced radiologists’ and clinicians’ satisfaction with the system. Image quality was a much more important issue in the early years of PACS deployment. Today the technology has matured and images of excellent quality are the norm.

Third, it was hypothesized that user satisfaction was positively associated with PACS technical service quality. As reported in the preceding section, this result was supported but for technologists only. In the servicing agreement concluded with the PACS provider, technical service at the end user level, be it PC related or application related was under the responsibility of the hospital technical staff. The PACS provider was responsible for the enterprise servers, a layer often transparent to the users. For clinicians and radiologists the system has, in general, performed well possibly explaining why service has not been an issue in the satisfaction expressed by these two groups. In a particular aspect of the technologists’ work which required integration between the PACS and another application used routinely by them some problems have persisted. The ability of the service team to fix this recurrent issue has made service quality an important parameter influencing their degree of satisfaction.

Lastly, satisfaction with PACS was also predicted by users’ confirmation of the realization of expectations in relation to PACS usage. However, this result was significant for only two out of three groups of respondents, namely, technologists and clinicians. The larger effect size of confirmation, relative to perceived usefulness (for clinicians) and reliability (for technologists), suggests that technologists and clinicians view the realization of their expectations as more salient than instrumentality and reliability of PACS in forming affect and, ultimately, intention about system continuance intention. For the primary users of PACS, the radiologists, satisfaction, - which, at the onset of implementation, might have been influenced by whether or not expectations had been met - two years after deployment, appears to be solely dependent on the system’s reliability and efficiency.

The next series of research hypotheses concern the antecedents of PACS continuance intention. The model in Figure 2 posits that continuance intention is influenced by three variables, namely, user satisfaction, perceived net benefits, and confirmed expectations. All three variables were found to be positively associated with continuance intention, but none of the variables was significant for all three groups of PACS users. On one hand, results of the study provide additional support for the expectation-confirmation theory’s contention that satisfaction with PACS use is the strongest predictor of users’



continuance intention. Indeed, a strong association was found between satisfaction and actual continuance behaviours for both radiologists and technologists. To a much lesser extent, radiologists’ continuance intention was also influenced by the confirmation of expectations while that of technologists was also related to the perceived net benefits. On the other hand, clinicians’ intention to continue using PACS in the future was significantly and positively influenced by a single variable, namely, the perceived net benefits from PACS usage. Contrary to our expectations, continuance intention for the latter group was neither influenced by their satisfaction with prior usage of the technology nor by the extent to which their initial expectations had been met. Clinicians plan to continue using PACS regardless of the level of satisfaction because of the advantages that PACS affords them in their practice. As mentioned earlier, the possibility of instantaneous access to images anywhere in the hospital has a significant positive impact on productivity and the quality of their practice. Satisfaction with PACS may be more related to the particular commercial product chosen, resource management at the hospital level, or other issues that are not related to the concept of electronic management and distribution of medical images.

Lastly, contrary to our expectations, most hypotheses concerning the antecedents and consequences of PACS usage (actual use) were not supported. Indeed, no significant relationship was found between use (in terms of frequency, intensity and scope) and system quality, information quality, and service quality. Similarly, with only one exception (see Table 9), actual PACS usage was not related to user satisfaction and perceived net benefits. Contrary to Seddon [51], who posits that use is not an indicator of IS success but that user satisfaction is because it is related to perceived impacts, we are not rejecting usage as an appropriate measure of PACS success. In our view, the problem in the present study is more methodological than conceptual. On one hand, simply measuring the frequency and amount of time a PACS is used by radiologists and clinicians may not properly capture the relationship between usage and other constructs such as user satisfaction and perceived net benefits. On the other hand, as suggested by DeLone and McLean [10], it can be argued that declining usage may be an important indication that the anticipated benefits are not being materialized and that users are not satisfied. Consequently, enriching our understanding of use may position us to better understand individual and organizational outcomes of PACS usage. We also posit that PACS usage must not be rejected as a success variable when usage is mandatory (as in the present study). Even when usage is required, variability in the quality and intensity of this use is likely to have a significant impact on the satisfaction of users and the realization of system benefits. Researchers must therefore not only rely on perceptual measures of PACS use but also consider objective ones obtained through computer monitoring.

8. Conclusion

The research reported here signifies an important first step toward a comprehensive and holistic understanding of PACS technology success in the hospital setting. The research model presented in Figure 2 was partially supported by the data collected at CHUM. Of the thirteen hypothesized relationships tested by regression analysis, eight were found to be significant (for one or several groups of PACS users) and the reminder not significant (for all three groups of PACS users). The analysis provided strong support



for relationships between user satisfaction and perceived net benefits, user satisfaction and continuance intention, perceived net benefits and continuance intention, confirmed expectations and user satisfaction, and some dimensions of system quality and user satisfaction.

Continued research will be needed in several areas. For one thing, further model validation will be important. One possible direction is to validate and thus enhance the model by conducting other case studies of PACS success in various hospital and clinical settings. Eventually, empirical tests of the model will be essential. Towards this, survey studies that target various users and user groups in hospitals, clinics, and other clinical settings are desirable. In addition, while users’ perceptions play a significant role in evaluating PACS success, further research is required to consider both subjective and objective measures of PACS success, and to provide a comprehensive model of PACS success appropriate to various groups of users. Objective measures can be easily obtained for several variables presented in Figure 2 including dimensions of system quality (e.g., system integration, system reliability, ease of access), PACS usage, and net benefits (both at the individual and departmental levels).



References

[1] H.K. Huang, PACS – Basic Principles and Applications, John Wiley & Sons, New York: NY, 1999.

[2] M.C. Trudel and G. Paré, "Barrières à l’innovation technologique dans les hôpitaux : étude du premier cas d’adoption du PACS au Québec", Gestion- Revue internationale de gestion, 29(1), 2004, pp.36-45.

[3] B.I. Reiner, E.L. Siegel and K. Sidiqqui, "Evolution of the Digital Revolution: A Radiologist Perspective," Journal of Digital Imaging, 16(4), December 2003, pp.324-330.

[4] R.A. Bauman and G. Gell, "The Reality of PACS: A Survey," Journal of Digital Imaging, 13(4), November 2000, pp.157-169.

[5] M.D. Ralston and R. Coleman, "Sharing of a Single Picture Archiving and Communications System among Disparate Institutions: Barriers to Success," Journal of Digital Imaging, 15 (supplement 1), 2000, pp.3-6.

[6] D.R. Gale, M.E. Gale, R.K. Schwartz, V.V. Muse and S.T. Walker, "An Automated PACS Workstation Interface: A Timesaving Enhancement," American Journal of Radiology, 174, January 2000, pp.33-36.

[7] B.S. Hertzberg, et al. "PACS in Sonography: Accuracy of interpretation using film compared with monitor display," American Journal of Radiology, 173, November 1999, pp.1175-1179.

[8] D.B. Hayt, "Filmless in 60 days: The Impact of PACS within a large urban hospital," Journal of Digital Imaging, 14(2), June 2001, pp.62-71.

[9] W.H. DeLone and E.R. McLean, "Information Systems Success: The Quest for the Dependent Variable," Information Systems Research, 3(1) March 1992, pp.60-95.

[10] W.H. DeLone and E.R. McLean, "The DeLone and McLean Model of Information Systems Success: A Ten-year Update," Journal of MIS, 19(4), Spring 2003, pp.9-30.

[11] A. Battacherjee, "Understanding information systems continuance: an expectation-confirmation model," MIS Quarterly, 25(3), September 2001, pp.351-370.

[12] Shannon, C.E. and Weaver, W. The Mathematical Theory of Communication, University of Illinois Press, Urbana: IL, 1949.

[13] B. Cox and N. Dawe, "Evaluation of a PACS System on an Intensive Care Unit ," Journal of Management in Medicine, 16(2&3), pp.199-205.



[14] D.M. Tucker, G.T. Barnes and R.E. Koehler, "PACS in the ICU," Radiology, 196, 1995, pp.297-304.

[15] S.L. Lou, "An automated PACS image acquisition and recovery scheme for image integrity based on the DICOM standard," Computerized Medical Imaging and Graphics, 21(4), 1997, pp.209-218.

[16] W. Pavlicek, et al. "Quality of Service Improvement from Coupling a Digital Chest Unit with Integrated Speech Recognition, Information, and PACS," Journal of Digital Imaging, 12(4), 1999, pp.191-197.

[17] J.R. Pilling, "Picture Archiving and Communication Systems: The Users' View," British Journal of Radiology, 76(908), 2003, pp.519-524.

[18] M.E. Blado and A.B. Tomlinson, "Monitoring the Accuracy of a PACS Image Database," Journal of Digital Imaging, 15 (supplement 1), 2002, pp.87-95.

[19] K.W. McEnery, et al., "Evaluation of Soft Copy Distribution of Diagnostic Studies before Soft Copy Interpretation," Journal of Digital Imaging, 15(1), 2002, pp.76-80.

[20] H.K. Huang, S.L. Lou and W.P. Dillon, "Neuroradiology Workstation Reading in an Inter-hospital Environment: a Nineteen Month Study," Computerized Medical Imaging and Graphics, 21(5), 1997, pp.309-317.

[21] C.K. Tamm, et al., "Measure of the Utility of a Clinical PACS: Comparison of Self-Reported Measures and Direct Measures of PACS Usage by Clinicians," Journal of Digital Imaging 16(1), 2003, pp.58-60.

[22] S. Bryan, et al., "The Benefits of Hospital-wide PACS: a Survey of Clinical Users or Radiology Services," The British Journal of Radiology, 72, 1999, pp.469-478.

[23] H. Kato, et al., "Preliminary time-flow study: comparison of interpretation times between PACS workstations and films," Computerized Medical Imaging and Graphics, 19(3), 1995, pp.261-265.

[24] B.I. Reiner and E.L. Siegel, "Technologists’ productivity when using PACS: Comparison of film-based versus filmless radiography," American Journal of Radiology, 179, July 2002, pp.33-37.

[25] C.W.T. Mattern, et al. "Electronic Imaging Impact on Image and Report Turnaround Times," Journal of Digital Imaging, 12(2), May 1999, pp.155-159.

[26] G.C. Weatherburn, et al., "A comparison of image reject rates when using film, hard copy computed radiography and soft copy images on PACS workstations," The British Journal of Radiology, 72, 1999, pp.653-660.



[27] M.E. Blado, Y. Ma, R.a. Corwin and S.G. Carr, "Impact of Repeat/Reject Analysis in PACS," Journal of Digital Imaging, 16 (Supp. 1), 2002, pp.22-26.

[28] D. Dackiewics, et al., "Impact of digital radiography on clinical workflow and patient satisfaction," Journal of Digital Imaging, 13(2), supplement 1, May 2000, pp.200-201.

[29] R.O. Redfern, et al., "Radiology workflow and patient volume: Effect of PACS on technologists and radiologists," Journal of Digital Imaging, 13(2), May 2000, pp.97-100.

[30] B.I. Reiner, et al., "Impact of filmless radiology on frequency of clinician consultations with radiologists," American Journal of Radiology, 173, November 1999, pp.1169-1172.

[31] G.C. Weatherburn and S. Bryan, "The effect of a PACS on patient radiation doses for examination of the lateral lumbar spine," The British Journal of Radiology, 72, 1999, pp.534-545.

[32] R.O. Mason, "Measuring Information Output: A Communication Systems Approach," Information & Management, 1(5), October 1978, pp.219-234.

[33] T. McGill, V. Hobbs and J. Klobas, "User-Developed Applications and Information Systems Success: A Test of DeLone and McLean’s Model," Information Resources Management Journal, 16(1), 2003, pp.24-45.

[34] P.B. Seddon and M.Y. Kiew, “A partial test and development of the DeLone and McLean model of IS success”, Proceedings of the ICIS, 1994, pp.99-110.

[35] A. Rai, et al. "Assessing the validity of IS success models: an empirical test and theoretical analysis," Information Systems Research, 13(1), 2002, pp.50-69.

[36] M. Igbaria and M. Tan, "The consequences of the IT acceptance on subsequent individual performance," Information & Management, 32(3), 1997, pp.113-121.

[37] G. Torkzadeh and W.J. Doll, "The development of a tool for measuring the perceived impact of IT on work," Omega, 27(3), 1999, pp.327-339.

[38] Rogers, E.M. Diffusion of Innovations (5th edition), Free Press, New York: NY, 2003.

[39] R.L. Oliver, "A Cognitive Model for the Antecedents and Consequences of Satisfaction," Journal of Marketing Research, 17, 1980, pp.460-469.

[40] B.J. Liu, et al. "Trends in PACS image storage and archive," Computerized Medical Imaging and Graphics, 27, 2003, pp.165-174.



[41] F. Cao, et al. "Medical Imaging Security in a HIPAA mandated PACS environment," Computerized Medical Imaging and Graphics, 27, 2003, pp.185-196.

[42] W.J. Doll and G. Torkzadeh, "The Measure of End-Ùser Computing Satisfaction," MIS Quarterly, 12(2), June 1988, pp.259-274.

[43] T.P. Van Dyke, et al. "Measuring IS service quality: concerns on the use of the SERVQUAL questionnaire," MIS Quarterly, 21(2), 1997, pp.195-208.

[44] W.J. Kettinger and C.C. Lee, "Perceived service quality and user satisfaction with the IS function," Decision Sciences, 25(5-6), 1995, pp.737-765.

[45] L. Raymond, "Organizational characteristics and MIS success in the context of the small business," MIS Quarterly, 8(1), March 1985, pp.37-52.

[46] M.J. Tyre and W.J. Orlikowski, "Windows of opportunity: Temporal patterns of technological adaptation in organizations," Organization Science, 5(1), 1994, pp.98-118.

[47] J. Nunally, Psychometric Methods, 2nd edition, McGraw-Hill, New York: NY, 1978.

[48] F.D. Davis, R.P. Bagozzi, and P.R., Warshaw, “User Acceptance of Computer Technology: A Comparison of Two Theoretical Models,” Management Science, 35(8), 1989, pp.982-1003.

[49] K. Mathieson, “Predicting User Intentions: Comparing the Technology Acceptance Model with the Theory of Planned Behavior,” Information Systems Research, 2(3), 1991, pp.173-191.

[50] S. Taylor and P.A. Todd, “Understanding Information Technology Usage: A Test of Competing Models,” Information Systems Research, 6(1), June 1995, pp.144-176.

[51] P. Seddon, “A Respecification and Extension of the Delone and McLean Model of IS Success,” Information Systems Research, 8(3), September 1997, pp.240-253.


ISSN 1713-1383

Health & Medicine