Evaluation of Teleradiology Image Quality Using a SMPTE Test Pattern

Abstract

It is important to evaluate the quality of teleradiology systems that provide images upon which clinical decisions are made. This study suggests that evaluation can be done using standard test images, such as that produced by the Society of Motion Picture and Television Engineers (SMPTE). The article further demonstrates that imaging systems that have discernible loss of resolution on test patterns also suffer detectable loss of image quality on clinical images.

INTRODUCTION

The recent proliferation of teleradiology systems makes it increasingly important to evaluate the quality of the digital images they produce. One approach might be to compare transparencies from plain film radiography, computed tomography (CT), magnetic resonance (MR), etc. with the corresponding digital images. This method of evaluation has the advantage of being based on images routinely encountered in clinical practice. The problem with comparisons based on clinical images is that they tend to be subjective, non-quantitative, and non-reproducible by others who do not have access to the same images.

The solution proposed in this study, and advocated by the American College of Radiology (ACR) [Ref. 1], is the use of a standard image containing lines and grey scales produced by the Society of Motion Picture and Television Engineers (SMPTE) (Figure 1) [Ref. 2]. The SMPTE image has been described in detail in the literature [Ref. 3,4]. Other test patterns have also been proposed [Ref. 5,6].

Use of a standard pattern also allows evaluation of images based on traditional 35 mm. and digital photography as well as those generated from semi-professional video capture devices and flatbed scanners.

The Clinical Setting

When a CT study needs to be sent to a physician's home workstation for emergency interpretation, the data can be sent directly from the CT console via modem without the need for "hard copy" output onto transparency film. Images are compressed 10:1 using a wavelet based algorithm prior to transmission.

For non-digital studies such as radiographs, transparencies are digitized at 2k resolution using a Lumisys75 scanner. Images were compressed 30:1 prior to transmission, via modem, to the physician's home workstation.

METHODOLOGY

A standard SMPTE image file is included in the software package supplied with the GE spiral CT workstation. Images based on this SMPTE file were generated in a variety of ways and subsequently evaluated by the following methods:

• Method 1 -- (The "SMPTE file"): After wavelet compression, the SMPTE digital image file was sent via modem directly to the physician's home workstation.

• Method 2 -- (The "SMPTE transparency"): The SMPTE image was transferred onto a sheet of film from the workstation console using the departmental laser printer just as would be done for a routine CT study. This filmed image became the standard against which all other images were compared. In addition to being viewed directly, the SMPTE transparency was processed in a variety of ways:
  • Method 2a The SMPTE transparency was digitized at the hospital using a Lumisys75 flatbed scanner at 2k resolution. The resulting digital file was stored on a network server and then sent by modem to the physician's home workstation. Images were compressed 30:1 and 10:1 prior to transmission.

  • Method 2b Photographic slides of the SMPTE transparency were obtained using a 35 mm Olympus OM-1 camera equipped with a Zuiko 50 mm. macro lens using Kodak Gold ASA 100 slide film. Slides were stored digitally on a commercially produced Kodak PhotoCD.

  • Method 2c Digital images of the SMPTE transparency were obtained by video capture using:
    • A Panasonic WS 1410 black and white video camera equipped with a manually adjustable Fujinon-TV Zoom lens and a Snappy (Play, Inc.) digitizer attached to the parallel port of a Dell 166 MHz portable computer.

    • A Sony CCD-V101 Hi8 video camera with an F1.4 lens.

  • Method 2d Method 2d Digital images were generated by scanning the SMPTE transparency at 300 dpi using an Epson 636 flatbed scanner equipped with a transparency unit attached to the parallel port of an Amiga 4000 computer.

  • Method 2d Digital images of the SMPTE transparency were obtained using an Olympus D-600L digital camera. Images were digitized at 1260x1024 resolution and stored as JPEG files, both at high quality (~300 KB file) and super high quality (~1MB file), with larger files reflecting less compression.

Images sent to the home workstation were displayed on the monitor and also screen-captured at 300 dpi. Computer monitor images were also photographed using the Olympus OM-1 35 mm camera described above and stored on a Kodak PhotoCD.

RESULTS

• Method 1 -- (The "SMPTE File") The SMPTE image sent directly from the CT console ("SMPTE file") was evaluated on the home workstation monitor.

  Because the SMPTE image thus generated was distorted and not representative of CT images routinely sent, it was excluded from further analysis (Figure 2). Poor image quality was hypothesized to be due to the image compression methodology which could not be bypassed on our commercially supplied system.

• Method 2 -- The SMPTE Transparency The primary standard for comparison was the original image recorded on the same transparency film used for CT scans.

  Although considered the "gold standard," transparency film is not without its limitations. Films are costly to produce and store and easily lost or misplaced. Images are capable of no further enhancement and, if printed at non-optimal window and level values, cannot be subsequently adjusted. Filmed studies must be reviewed on-site if they are not sent elsewhere by mail or messenger.

  • Method 2a -- The Hospital-scanned SMPTE Film Images generated from the SMPTE transparency film by digitization were compressed and transmitted via the hospital-based teleradiology system and reviewed on the home workstation.

    The teleradiology images reproduced high-contrast elements but failed to resolve low-contrast detail even at 10:1 compression. High contrast elements had artifactual "companion shadows" (Figure 3). Photographs of the monitor screen and those obtained digitally via screen capture were visually identical to the images on the monitor itself.

  • Method 2b -- 35 mm Photography 35 mm slides or photographs of the transparency can serve as a conveniently transportable, secondary standard for purposes of documentation, publication, and education. They are not suitable for primary diagnosis.

    Photographs of the SMPTE transparency were judged capable of excellent fidelity provided sufficient care was taken with the photographic technique (Figure 1).

    Slides and photographs suffer from the same limitations as transparency film. In addition, because they are a derivative of the original, there must, at least in theory, be some loss of information, although this may not be visible under ordinary conditions. Further, because slides and photographs must be first developed, the information they contain is not immediately available.

  • Method 2c -- Video Image Capture Digital images generated by video capture of the SMPTE transparency were judged to be adequate for teaching and web applications. Images were not as sharp as those from flatbed scanning. Limiting factors may have included the camera, the camera lens, and limitations inherent in NTSC (National Television Standards Committee) devices (Figure 4).

    Video image capture is not as convenient as flatbed scanning and requires careful attention to alignment and focusing. The set-up is not as stable physically and requires more horizontal and vertical space than a flatbed scanner.

    Video cameras are suitable for digitizing 3-dimensional objects, such as pathology specimens, and for capturing motion.

  • Method 2d -- Non-compressed Flatbed Scanning Digital images generated by scanning the SMPTE transparency with an Epson scanner reproduced all detail with excellent fidelity Figure 5). Images were superior to those generated by the hospital-based teleradiology system (compare Figures 3 and 5).

    The off-site scanning procedure differs from the hospital- based system in two ways. First, on the Epson, the non-film area can be masked prior to digitization so that the gray scale is produced from the transparency alone and not from the surrounding area. Second, the teleradiology system is configured so that images must be compressed prior to transmission while those generated on the Epson are not compressed.

    Flatbed scanning is fast, efficient, and convenient. Unlike photographic techniques, no focusing is involved. The scanner footprint is small and space-conserving and the instrument itself is sturdy and not readily subject to misalignment.

  • Method 2e -- Digital Photography Digital photography has, until recently, suffered from high cost, low resolution, and poor camera characteristics. Currently available megapixel cameras have overcome these and other limitations. Digital photographs of the SMPTE image test were found to have adequate resolution even with JPEG compression and, compared with film based technology, images are almost instantaneously available.

    However, on the SMPTE test image, some mild curvature was observed at the edges, demonstrating that digital photography is still subject to issues such as lens quality and camera technique (Figure 6). Nonetheless, digital photographs of actual CT images are impressive (Figure 7).

DISCUSSION

The obvious question is how the degradation of image quality seen on a SMPTE test pattern relates to clinical imaging.

A literature review of studies evaluating teleradiology systems using clinical images is interesting. In most studies, teleradiology images were derived from digitizing transparencies, usually plain films (radiographs). When CT scans were sent via modem, these too were digitized, usually with a flatbed scanner but sometimes with video capture. Direct transmission of the CT data (image file) was not performed. Minimal or no data compression was used. Conclusions differed regarding the efficacy of teleradiology.

Some studies, such as that from the Veterans Affairs Medical Center in Boston, enthusiastically endorsed teleradiology, particularly when high-resolution monitors were used [Ref. 7].

Others, such as that from the Johns Hopkins Medical Institutions, concluded that "Results with the teleradiology system were found unacceptable for primary interpretation of the spectrum of radiographs seen in an emergency department" [Ref. 8]. Yet other studies evaluated and endorsed teleradiology for specific applications, such as intravenous urography, while noting limitations, which in one study, consisted of a higher false positive rate compared with interpretation of the original films [Ref. 9].

The commercial teleradiology system demonstrated in this paper to have difficulty reproducing SMPTE images also has limitations when used clinically. For example, spiral CT scans performed to exclude aortic dissection are difficult to interpret because of loss of vascular edge detail. In addition, non- digital studies, in which film must be digitized prior to transmission, are less sharp than digital studies sent directly from the imaging console (Figures 8-10).

The latter observation is important because most studies in the literature are based on digitizing transparencies, whereas most of the emergency studies in our clinical practice are CT scans, in which digital data are sent directly to the home workstation without first creating film. CT images derived from digital data can be "windowed" and "leveled" while digitized transparencies can only have their brightness and contrast adjusted.

CONCLUSION

Although hospital-based teleradiology systems do produce CT images suitable for preliminary interpretation, the SMPTE test pattern reveals that compression algorithms can result in a visibly apparent loss of resolution. Current "lossy" compression methodology represents a trade-off between image quality and the need for rapid transfer of large volumes of data.

The data in this brief study suggest that, at least for some radiological studies, use of high speed data transfer lines, such as that provided by ISDN, ADSL, and other technologies, may be a better solution for transferring large data sets rather than compressing data and using inexpensive modems. In addition, rigid, proprietary systems that do not allow compression to be bypassed are probably best avoided because of the possibility of image degradation.

Digital imaging holds the promise of image enhancement, convenient storage and cost effectiveness but only if image quality can be maintained.

REFERENCES

1. American College of Radiology. ACR Standard for Teleradiology. Res. 35 - 1998 (http://www.acr.org)

2. Society of Motion Pictures and Television Engineers (SMPTE). Specifications for Medical Diagnostic Imaging Test Pattern for Television Monitors and Hard-copy Recording Cameras. Recommended Practice RP 133-1986. SMPTE Journal 1986; 95: 693-695.

3. Gray, J.E., Lisk, K.G., Haddick, D.H., et al. Test Pattern for Video Displays and Hard-copy Cameras. Radiology 1985; 154: 519-527.

4. Gray, J.E. Use of the SMPTE Test Pattern in Picture Archiving and Communication Systems. Journal of Digital Imaging 1992; 5(1): 54-58.

5. Halpern, E.J., Esser, P.D. An Improved Phantom for Quality Control of Laser Scanner Digitizers in Picture Archival and Communications Systems. Journal of Digital Imaging 1991; 4(4): 241-247.

6. Esser, P.D., Halpern, E.J.,, Amis, E.S. Quality Assurance of Picture Archiving Communication Systems with Laser Film Digitizers. Journal of Digital Imaging 1991; 4(4): 248-250.

7. Gale, M.E., Vincent, M.E., Robbins, A.H. Teleradiology for Remote Diagnosis : A Prospective Multi-year Evaluation. Journal of Digital Imaging 1997; 10(2): 47-50.

8. Scott, Jr., W.W., Bluemke, D.A., Mysko, W.K., et al. Interpretation of Emergency Department Radiographs by Radiologists and Emergency Medicine Physicians: Teleradiology Workstation versus Radiograph Readings. Radiology 1995; 195: 223-225.

9. Halpern, E.J., Newhouse, J.H., Amis, E.S., et al. Evaluation of Teleradiology for Interpretation of Intravenous Urograms. Journal of Digital Imaging 1992; 5(2): 101-106.

March/April, 1999

List of Publications -->

Return To Home -->