A comparison of case retrieval times: Film versus picture archiving and communications systems

A Comparison of Case Retrieval Times: Film Versus Picture Archiving and Communicat ions Systems

Steven C. Horii, Betty A. Levine, Gregory Goger, Seong Ki Mun, Rob Fielding, Brian S. Garra, S.-C. Benedict Lo, Brian Krasner, and Harold Benson

One of the advantages that a picture archiving and communications system (PACS) is supposed to pro- vide over a film-based operation is improved perfor- mance in retrieving images. Although it seems self- evident that this should be so, this experiment was intended to verify this and to provide some time comparisons for the two methods. The experiment consisted of randomly selecting ultrasound and com- puted tomography cases and determining how Iong it took to retrieve files at a PACS workstation or in person from the fUe room. To simulate actual retrieval volumes, a total of 40 cases from current to 6 months old, 20 cases from the past year, and 10 cases more than 1 year old was selected. Results indicate that PACS retrieval can indeed be faster than file room retrieval. However, the difference is less for recent cases than for older cases. For cases 6 or fewer months old, the workstation retrieval was approximately 2.5 minutes faster per case than the film file room. This time difference increased markedly when extended to the 1-year and older-than-l-year groups. This report details the results of this study and provides informa- t ion about the reliability of the two archives.

KEY WORDS: PACS, evaluation, performance study, design.

I N AN EVALUATION of the Georgetown picture archiving and communications sys-

tem (PACS) that was performed for the Digital Imaging Network System project, 1 most users of the system thought that using the PACS was a faster method for retrieving studies than going to the file room. Although this was the common belief, we only performed a small number of timing tests to verify this, and these tests were not performed systematically.

Films are retrieved fo ra number of reasons, usually by both referring physicians and radiolo- gists. A previous study 2 showed that films were often retrieved by the physician rather than by a

From the Department of Radiology, Georgetown University Hospital, Washington, D.C.

Address reprint requests to Steven C. Horii, MD, Department of Radiology, Hospital of the University of Pennsylvania, 3400 Spruce St, Philadelphia, PA 19104.

Reprinted with permission from Medical Imaging VI: Picture Archiving and Communications Systems, Society of Photo- Optical lnstrumentation Engineers, 1992.

0897-1889/92/0503-0002503.00/0

clerk or secretary. Any time saved in this pro- cess then translates into time saved for the physician. For the radiologist, the use of prior studies is mostly for comparison with a new study of the same type or for correlation with a study of a different type. This is particularly true of the sectional imaging examinations (such as computed tomography [CT], magnetic reso- nance [MR], and ultrasonography [US]). For either type of retrieval, a delay in obtaining the prior examination can result in a delay in performing the current procedure or in report- ing the results of it. The necessity of rapid access to prior examinations is one of the important factors that motivate the use of PACS.

The need for old studies, however, follows a well-known pattern. As Dwyer et aP have shown for individual studies and Greinacher et al 4 for the film archive as a whole, requests for exami- nations decline exponentially with time. This is the reason for the design of both film and PACS archives as layered or sequential systems. Typi- cally, film files have a short-term storage area for the most current cases, a long-term area for older examinations, a n d a very long-term ar- chive, which is often placed off site to reduce costs. Because the number of accesses will be small, the penalty of increased time for retrieval is traded off against the lower cost oŸ warehouse- type storage. PACS storage usually follows a similar model. The current work is kept in local or central high-speed storage with movement to high capacity but slower storage for long-term archiving. Very long-term storage is done by removing the long-term media from the archive device (eg, optical disk jukebox) and storing it separately.

EXPERIMENTAL BACKGROUND AND METHODS

The Georgetown film file room is organized asa series of three storage areas in the hospital and a long-term archive located off campus. Figure 1 illustrates the film filing hierarchy.

Aftcr examinations are performed and inter- preted, they are moved to the Active File. They remain in this area, which is within the Radiol-

138 JournalofDigitallmaging, Vol 5, No 3 (August), 1992: pp 138-143

FILM AND PACS RETRIEVAL COMPARISON 139

1 Active Master Archlve Flle 3o D ~ Flle 3 M -"

,n Campus

] Long term fl!e L

/ Off Campus

>IY

Fig 1. Film file organization.

ogy Department, for approximately 1 month. Based on the date of the most recently per- formed examination, after the stay in the Active File, the films are moved to the Master File, which is located in a less active area of the department. Films are stored in the Master File for approximately 3 months, after which they are moved to the Archive. The Archive is still in the department but is furthest from the Active File. Finally, after 1 year, films are moved off campus to the long-term file. In this location, they are retained for the time legally required.

Figure 2 shows the design of the PACS image archive. Images from current studies are stored on magnetic disks in the database management system (DMS). Some studies are automatically routed to disk storage at workstations for re- view. This workstation storage corresponds to film images in an automated light-box viewing system (alternator). Because the images are also stored in the DMS, they can be purged from the workstation when its storage is filled. After approximately 5 days, also based on the examination date, the examinations are moved to optical disk storage. The optical disk jukebox holds 79 disks, each of which stores 2 Gbyte. With the image acquisition rates during this

Work.. on, Oat-- ~�91 Opc. I Magnetic ~ Magnetlc Disk

Disk I Dlsk Jukebox

1,o Purge > IY

Shelf Storage I~ i

_

Fig 2. PACS archive design.

study, a little more than i year's studies can be accommodated in the jukebox. After this time, the disks are unloaded and stored on shelving in the machine room.

A film file is retrieved by visiting the window at the Active File area. File clerks then check the hospital information system (HIS) termi- nals, which support a film jacket tracking sys- tem. The HIS tells the clerks in which file (Active, Master, Archive, or warehouse [off campus]) the jacket should be located, or ir the jacket has been signed out. Ir signed out, the location is given. The file clerk will then retrieve the case ir it is in one of the on-campus files, direct the person seeking the file to the location noted ir the jacket is signed out, or ask if the person wants to put the request on the ware- house retrieval list ir the file is off campus.

A PACS retrieval is initiated by entering the patient's medical record number or name at the workstation. Then, ir the patient is known to the system, the PACS displays a list of all of the examinations it has in its files for that patient. The person seeking the study then selects the examination to be retrieved. Within 3 minutes, ir the file is on a disk not in the jukebox, the system delivers such a notice when queried. In this event, the system manager is contacted and loads the appropriate disk into the jukebox manually. With the current system software, the notification that the study has been restored from the optical to magnetic disk is not auto- matic. The notification requires that the worksta- tion text screen be updated, so the user has to perform some action to receive such an update. In practice, this is often done by having the user toggle back and forth between the main menu text screen and the patient data screen, because this can be done with one keystroke for each action. As a study is restored from the optical disk, it is automatically routed to the requesting workstation and will be stored on the worksta- tion magnetic disk ir there is enough space.

US and CT cases were randomly selected from the log books normally kept for these examinations. The log books list the patient, medical record number, brief history, and find- ings. Originally, we had intended to include MR studies as well, but during the time covered by the study a major change was made in the way MR images were acquired by PACS. We

140 HORII ET AL

changed from a digitized vŸ acquisition to an all-digital capture. This changed the compliance of the technologists in acquiring images because the digital acquisition required less effort on their part. Because of this change in method, MR examinations were excluded from the re- trieval pool. US and CT images had been acquired by vŸ digitization throughout the study period. Table 1 lists the numbers and types of cases selected. We used a linearly decreasing total number of cases with time, because ah exponentially decreasing number would have meant a much larger number of recent cases of a much smaller number of older cases. CT examinations were drawn from both neuroradiologic and body case types.

The case sample included more US studies because the log book for the recent cases was more accessible. The same case was retrieved both from the film file and ffom the PACS.

This selection meant that most cases would be in the on-campus files for film and in the jukebox for the PACS. For the cases older than 1 year, all would be off campus for film and on disks not in the jukebox for the PACS. Timing rules were established for both scenarios. For film, the clock was started when the person retrieving films got in line at the file room and stopped when the jacket was retrieved and the target study pulled out to verify that the correct examination was found. For PACS, the clock was started when the person doing the retrieval entered the patient's medical record number or name at the workstation (if there had been time waiting at the workstation for someone else to finish using it, it too would have been included) and stopped when the first image of the study appeared on the display and the user verified that it was the correct examination.

The file room was notified that such a study was being performed but that it posed no threat to file room personnel because there would be no connection of retrieval times with specific individuals. Although the possibility of introduc-

Table 1. Cases Used in the Study

Study Type

Study Age US CT

_<6 mo 30 10 >6 mo and _<1 yr 8 12 >1 yr 5 5

ing a bias was considered, some of the study personnel were unknown to the file room clerks, and the clerks would have been unwilling to let them sign films out. Also, file clerks were instructed not to prioritize requests from the study personnel but to handle such requests in a routine manner. To avoid the bias of particu- larly busy times at the file room, film retrievals were performed at random times during the work day. A similar policy was followed for PACS workstation retrievals.

Results were tabulated and analyzed using Microsoft Excel (Microsoft Corp, Redmond, WA) and Cricket Software StatWorks (Cricket Software, Malvern, PA) for the Apple Macin- tosh (Apple Computer, Cupertino, CA).

RESULTS

The results of the data collection are summa- rized in Tabte 2. Over all cases, the average retrieve time for PACS was 5 minutes 17 sec- onds, and that for film was 6 hours 48 minutes 40 seconds. This difference was significant (P < .01) by Student's t test.

In addition to the data for retrieval time, notes were also made of which cases could not be retrieved either on film or from the PACS. These "failure rates" are summarized in Table 3.

In an effort to minimize the number of cases that could not be recovered for film, the file clerks and the person searching followed all the usual procedures in the event that a jacket could not be found in any of the normal locations. The exception was that a repeat request was not tried after a 1- or 2-day wait. For the PACS retrieval, if the patient could not be found by medical record number, a search using the patient's last name was made. This would yield a list of patients and occasionally located a patient record that had been saved under an incorrect number.

DISCUSSION

From Table 2 it can be seen that for recent cases, the performances at retrieving the images from film and PACS were comparable. Though the PACS is faster, the difference is not statisti- cally significant. The film retricval time had a large variance, which was expected. The vari- ance found in the PACS retrieval times (for

FILM AND PACS RETRIEVAL COMPARISON

Table 2. Summary of Retrieval Times

141

Retrieval Time

US CT Total

Study Age PACS Film PACS Film PACS Film

~ 6 m o 3:21 3:58 3:40 13:42 3:26 6:07 (2:13) (3:08) (0:54) (20:01) (1:58) (10:38)

> 6 m o and ~1 yr [ 2:41 1:12:56 3:19 1:22:13 3:01 1:18:14 ] (0:43) (1:23:11) (0:201 (1:14:29) (0:39) (1:21:18)

> 1 yr [ 8:51 25:47:47 29:42 66:44:00 20:46 59:54:42 ] (1:05) - - * (34:39) (1:25) (28:10) (6:44:56)

NOTE: The retrieval times are given as minutes and seconds or hours, minutes, and seconds, depending on the magnitude of the time. Values are means with SD in parentheses. The boxed values for study ages of between 6 months and 1 year and greater than year indicate that the differences between PACS and film retrieval times are significant. Signfficance was tested usin9 Student's t test, and the null hypothesis was accepted for P > .05. The same values were found to remain significantly different when a paired-value t test for performed. For this latter test, only cases with both film and PACS time values present were used, excluding those cases for which either the film or PACS version of the study could not be retrieved. In addition to the differences between PACS and film retrieval times, significant differences were also found between US and CT retrieval times for film for studies less than 6 months old and for PACS for studies greater than 1 year old. Possible reasons for these differences are described in the discussion.

*Only one case found; no SD calculated.

cases up to 1 year old) was primarily the result of the number of images in the different studies and how busy the DMS was with other tasks. In both instances, increasing the numbers of im- ages a n d a DMS being busy with other tasks resulted in increased retrieval times. 5

The significant difference found between film CT and US retrievals was more difficult to explain. One factor we have noted is that requests for CT films are more common than for US films (given equal numbers of both examina- tion types). This would result in CT films being signed out more often, in a location other than the file room, or lost. A review of the findings for films not found showed that for the cases 6 months old or newer, 20% of CT and 6% of US films could not be found. For cases between 6 months and 1 year old, the proportions were 33% of CT and 25% of US films. For cases more than 1 year old, the proportions inverted, with 20% of film CT examinations not found and 80% of film US studies not retrieved. Because only five examinations of each study type were

Table 3. Percentage of Cases Not Found

Percentage of Cases Not Found

Study Age PACS Film

_<6mo 5 10 >6moand < l y r 15 30 > 1 yr 30 40

selected in the over-l-year category, some of this difference could be from sampling error. Actual clinical experience suggests that lar bet- ter than 80% of prior US studies can be found. A baseline study performed for the Georgetown DINS project showed that in general the film file room did much better than these figures would indicate. For 826 randomly selected pa- tients, 90% of the film jackets were found. 6 The present study showed the bias of particular studies (eg, CT, US) which, because they tend to be ordered for more specific diagnostic prob- lems than plain films, may also be requested and signed out more frequently, increasing the like- lihood of misfiling or loss.

There is an important difference in the rea- sons for the cases not being retrieved for PACS and film. For film cases, the log books showed unequivocally that the examination was per- formed and that films were made. A failure to retrieve a case in these circumstances was a failure of the film filing system. For the PACS, a study not being retrievable was almost certainly the result of a failure of the technologist to capture the study on the system and not the result of data loss from a hardware system failure. In general, hardware failures in the Georgetown PACS will prolong retrieve time but not make studies irretrievable. An example of this accounts for the difference in retrieval times for the PACS US and CT cases more than

142 HORII ET AL

1 year old. In all but one instance, the system manager loaded the appropriate optical disk, and the case was then restored to the worksta- tion. In one case (a CT study), the optical disk jukebox hada temporary hardware failure dur- ing this process. The result was a much longer retrieval time while the problem was corrected. There was no true data-loss PACS failure dur- ing this study. It is possible to capture a case in the PACS but "lose" it electronically. Supplying the wrong medical record number will make it difficult to retrieve the study later, but this type of error can be circumvented by searching on the patient's name. Using a different patient's medical record number and failing to notice that the study is being saved under the incorrect name is one circumstance that is equivalent to misfiling a film jacket. These types of errors were reduced in our system when the HIS interface was implemented. 7 This allowed the HIS to supply patient demographics to the PACS and obviated a redundant entry of these data when creating a PACS examination.

The decreasing percentage of nonretrieved cases on the PACS for the more recent studies is due to improved technologist compliance in acquiring cases. For US, one factor was connect- ing all four procedure rooms to the acquisition system. Before this, studies performed in one of the rooms (not the most frequently used) were not acquired on the PACS. For CT, PACS acquisition tended to lapse on weekends and during evening shifts. However, improving train- ing and emphasizing to the technologists the importance of finding the studies resulted in marked improvements in PACS acquisition rates.

With the exception of two instances, the reasons for failure to locate the film cases are not fully known. In the two instances noted, records showed that the patients had signed out the films and never returned them. Otherwise, films being held without being properly signed out and films misfiled are the two main pre- sumed causes of failure to retrieve the studies.

It could be argued that because such a small

proportion of films are requested for older cases, the expenses of PACS archiving and increasing staffing of file rooms ate not cost- effective. This study made no attempt to corre- late use of the retrieved case with the length of time needed to find it. The cases were randomly selected and so had no clinical reason for being requested. However, in our experience, approx- imately three to five patients per day in US have some delay either in reporting the results of their examinations or in having their studies performed because of time delays in locating the prior films. This translates into delays get- ting results to the referring physicians and increased waiting time for the patients. We have also had to repeat portions of CT scans, though less frequently, because film retrieved after the study was begun showed abnormalities in unsus- pected areas.

A film-based solution to this problem is to produce duplicate sets of films. Although this can work because one set of films never leaves the radiology department, it not only entails the use of double the amount of materials, but also increased personnel or personnel time to pro- cess, sort, and file these films. A PACS-based solution has costs of its own and cannot yet fully address the problem of providing images to physicians outside the hospital. Hybrid PACS, such as those that use electronic archiving but are capable of printing results of examinations or subsets of examinations on demand, and those that can similarly print on low-cost media, may be the best approach for the short term.

SUMMARY

This study has shown that for a randomly selected set of US and CT examinations, PACS and film retrieval times were similar if the cases were less than 6 months old. For examinations older than this, however, the PACS was signifi- cantly faster at retrieving the cases. This differ- ence increased with the age of the examina- tions. The PACS was also more reliable than film in terms of the percentage of cases success- fully retrieved.

REFERENCES

1. Elliott LP, Mun SK, Horii SC: Digital Imaging Net- 2. Mun SK, Benson HR, Welsh C, et al: Baseline study of work System (DINS) Evaluation Report. McLean, VA, radiology services for the purpose of PACS evaluation. Proc MITRE Corporation, contract no. N-55201, March 1990 SPIE 914:978-987, 1988

FILM AND PACS RETRIEVAL COMPARISON 143

3. Dwyer III S J, Templeton AW, Martin NL, et al: Cost of managing the digital diagnostic images for a 614 bed hospital. Proc SPIE 318:3-8, 1982

4. Greinacher CFC, Bach F, Mª K, et al: A realistic approach to evaluating digital imaging systems. Proc SPIE 454:91-98, 1984

5. Cerva JR, Kerlin BD, Pocinki LS: Filmless radiology: The design, integration, implementation, and evaluation of

a digital imaging network. McLean, VA, MITRE Corpora- tion, MITRE report MTR-90W00040, pp 3-1-3-4, 1990

6. Mun SK, Benson HR, Welsh C, et al: Baseline study of radiology services. Proc SPIE 914:985, 1988

7. Levine BA, Mun SK, Benson HR, et al: Assessment of the integration of a HIS/RIS with a PACS. Proc SPIE 1234:391-397, 1990