Chapter 3: CS689 1 Computational Medical Imaging Analysis Chapter 3: Image Representations, Displays, Communications, and Databases Jun Zhang Laboratory.

Chapter 3: CS689 1

Computational Medical Imaging Analysis Chapter 3: Image Representations, Displays, Communications, and Databases

Jun Zhang

Laboratory for Computational Medical Imaging & Data AnalysisDepartment of Computer Science

University of KentuckyLexington, KY 40506

Chapter 3: CS689 2

3.1a: Introduction

Biomedical image is a discrete representation in ordered graphical form of a biological object

It can be functions of one or two dimensions, generated by a variety of means, sampled within different coordinate systems and dimensions, and having variable quantized magnitude associated with each sample

Individual numerical elements in a digital image are pixels (picture elements) for 2D images and voxels (volume picture elements) for 3D images

Chapter 3: CS689 3

3.1b: Biomedical Images

Biomedical images are generated by a variety of energy transmissions through tissues or cells, (e.g., light, X-ray, magnetic fields, waves) and recorded by a sensor or multiple sensors

The mapping relationships between the transmitter, the object, and the sensor(s) are defined physically and mathematically

The correspondence between the object and the resulting image can be analytically described and subsequently analysis of the image can be carried out by direct inference, as if the analysis were performed on the object itself

Chapter 3: CS689 4

3.2a: Volume Image Representation 3D biomedical images are generally modeled

as a “stack of slices” and effectively represented in the computer as a 3D array

Important factors in 3D images: organization of the tomographic data, reference coordinate system used to uniquely address its values in three-space, transformations to obtain desired isotropicity and orientation, and useful display of the data, either in “raw” or processed format

Chapter 3: CS689 5

3.2b: Volume Image Data Organization Multidimensional biomedical images

reconstructed from tomographic imaging systems are characteristically represented as sets of 2D images

A pixel contains a value that represents and is related to some local property of the object

A pixel has a defined spatial position in two dimensions relative to some coordinate system imposed on the imaged region of interest

3.2 Pixels and an Example

Chapter 3: CS689 6

Chapter 3: CS689 7

3.2c: 3D Voxels

3D images represent the object as adjacent planes through the structure

3D images often contains voxels that are not cubic (the volume is anisotropic), as the thickness of the 2D images is greater than the size of the pixels

Many 3D biomedical image processings prior to visualization and analysis involve the creation of isotropic volume images from anisotropic data, using various spatial interpolation methods

The third dimension can be time in some cases with 2D images acquired repetitively over some time

Chapter 3: CS689 8

3.2d: CT/MRI Image Voxels

Chapter 3: CS689 9

3.2e: A Volume Date from 3D Voxels

Chapter 3: CS689 10

3.2f: 3D and 4D Images

Some 3D representation may consist of spatially correlated 2D images from different scanning modalities, or different scan types within a modality (a multispectral image set)

4D biomedical images are organized as collections of 3D images of varying time

4D organization can also consist of 3D multispectral volume image sets, each represents a different modality or type of acquisition

5D data sets consist of 4D time varying of different modalities

Chapter 3: CS689 11

3.3a: Coordinate Systems and Object Orientation The orientation of the structure being imaged is

specified in order to standardize the location and orientation of the object in a regular grid, permitting spatial correlation and comparison between volume images

One is origin-order system – mathematically rigorous, but not always intuitively obvious

The other is view-order consistency – a more natural and intuitive way of conceptualizing 3D volume images, but not mathematically consistent

Chapter 3: CS689 12

3.3b: Origin-Order System

In origin-order system, the coordinate system maintains an unambiguous, single 3D origin in the volume image with the order of all voxels in a single line of a plane

Consistency applied to a left-handed coordinate system, the projection of the origin is always from the origin outward. Volume images can be correctly reformatted into orthogonal sections

Flipping images about one or three of the axes corrupts the integrity of the image and results in a volume image that its mirror-reversed

Chapter 3: CS689 13

3.3c: View-order Consistency

This system is not concerned with the maintenance of a single unambiguous 3D origin. It simply posits a relationship that should hold between the orientation of the sectional images and their order in the data file

It accepts sectional images in any of six orientations, and assigns the axes to the row, column, and image directions of the image data regardless of the physical section orientation

The main advantage is that it is highly intuitive and completely insensitive to orientation

Chapter 3: CS689 14

3.3d: Transverse (Axial) Slices in the XY Plane

Chapter 3: CS689 15

3.3e: Sagittal Slices are in the ZY Plane

Chapter 3: CS689 16

3.3f: Coronal Slices are in the ZX Plane

Chapter 3: CS689 17

3.3g: What is the View of this Slice?

Chapter 3: CS689 18

3.3e: What is the View of this Slice?

Chapter 3: CS689 19

3.4a: Value Representation

At each pixel or voxel, a value is measured and/or computed by the imaging system. The range of the representation must encompass the entire range of possible measured and/or computed values and must be of sufficient accuracy to preserve precision in the measured values

This range of representation is the dynamic range of the data, determined by both the imaging system and the numeric scale

Discrete representation of value in the computer system cannot capture the entire dynamic range of the acquired signal from the imaging system

Chapter 3: CS689 20

3.5a: Interpolation and Transformation Most 3D biomedical volume images are sampled

anisotropically, with the slice thickness often significantly greater than the in-plane pixel size

Visualization and measurement usually require volume data to be isotropic, and the data must be post-processed before it can be used properly

Possible problems for anisotropic data: incorrect aspect ratio along each dimension for visualization, and aliasing artifacts due to difference in sampling frequency

Aliasing and Anti-Aliasing

Chapter 3: CS689 21

Chapter 3: CS689 22

3.5b: Interpolation

First-order linear interpolation of the gray level values is commonly used for 3D volume image

Values of “new pixels” between existing pixels are simply interpolated (averaged) values of the existing pixels. It is a tri-linear interpolation in 3D. The interpolation in each of the dimensions is completely separable

Tri-linear interpolation works well for most biomedical images

Chapter 3: CS689 23

3.5b: Linear Interpolation

Discrete data Linear interpolation

Chapter 3: CS689 24

3.5b: Polynomial & Spline Interpolation

Polynomial interpolation Spline interpolation

Chapter 3: CS689 25

3.5c: High Order Interpolation When the ratio in size between any of the voxel

dimensions in 3D becomes greater than approximately 5 to 1, tri-linear interpolation does not work well, which will provide poor approximations

High order interpolation, such as cubic interpolation, may be used, with increased computational cost

Cubic interpolation uses more than the immediate adjacent voxels and uses a cubic polynomial to estimate the intermediate values

Shape-based interpolation methods may be used for certain particular (known) structure of interest

Chapter 3: CS689 26

3.5c: Nearest Neighbor Interpolation It is fast.

Most useful if you need to see directly the voxels of the data you are displaying

Thickness of the corornal slices

Chapter 3: CS689 27

3.5c: Linear Interpolation Linear between

slice interpolation.

The data is interpolated with nearest neighbor with the slice plane

Chapter 3: CS689 28

3.5c: Tri-Linear Interpolation 3D linear

interpolation based on the neighboring voxels to each pixel on the displaced screen slice

Blurring of data problem

Chapter 3: CS689 29

3.5c: Cubic Interpolation Best quality

images Cubic

interpolation may take a long time to update on slower machines

Chapter 3: CS689 30

3.6a: Computers and Computer Storage In computers, each switch value is referred to as a

binary unit, or “bit” 8 bits form a “byte”. A computer word may be of size

8 bits, 16 bits, 32 bits, or 64 bits For evaluation of images by eye, comparative visual

sensitivity is limited to approximately 9 bits of gray scale, but the eye can adapt to lighting conditions over about a 20-bit range

Radiologists can modify the lighting conditions of the film to take advantage of the entire dynamic range expressed in the film

Chapter 3: CS689 31

3.6b: Numeric Values

Most computers have “natural” word sizes of 32 or 64 bits, and have special capabilities for handling numeric values of 8 or 16 bits

The resolving power of the physical sensors used to capture biomedical images typically does not exceed more than 12 or 14 bits of information

Storing 12 bits of data in 32 bits of storage is very inefficient

Chapter 3: CS689 32

3.6c: Computer Storage

Portable storage media: floppy disk (2MB), digital versatile disk (5,000 MB), flash disk (1GB)

Magnetic tapes are ideal for archiving and backing-up computer data

Hard disk: high speed and low cost with good reliability, primary devices for operating systems, computer software, and interim data

Memory (2GB) and cache (2MB) are faster and temporary storage. They are critical for practical interactive image processing and visualization

Chapter 3: CS689 33

3.6d: Storage of Biomedical Images Biomedical images can usually be stored in slow

storage media as they are not viewed frequently Design decisions for storage are based on the

needs of the practitioners who use images, and balanced against the cost of data storage and software systems needed to make them accessible, reliable, and durable

New standards, such as DICOM (Digital Imaging and Communications in Medicine), are evolving that will combine the universal access of film with the flexibility of digital imagery

Chapter 3: CS689 34

3.7a: Display Types and Devices The need for 3D display is emphasized by gaining

an appreciation of the fundamental dilemma of studying 3D irregularly shaped objects like the organs of the body by using tomographic images

3D displays avoid the necessity of mentally reconstructing a representation of the structure of interest

Multidimensional display techniques for 3D tomographic image analysis can be classified into two basic display types based on what is shown to the viewers: direct display and surface display

Chapter 3: CS689 35

3.7b: Direct Display

Direct display extends the concept of 2D buffer display of tomographic images into three dimensions by presenting the volume image as a 3D distribution of brightness

It enables direct visualization of unprocessed image data

Examples of 3D displays are holograms and space-filling vibrating mirrors

Most direct 3D displays use augmenting equipment, including stereo holograms, stereo viewers, head-mounted display, etc.

Chapter 3: CS689 36

3.7b*: Hologram Display

Hologram displays withoverhead and underneathillumination

Chapter 3: CS689 37

3.7c: Stereo Holograms

Left Eye ImageRed Channel Only

Right Eye ImageRed Channel Removed

AnaglyphLeft & Right Images Overlaid

Chapter 3: CS689 38

3.7d: Surface Display

Surface display is to visualize the surfaces that exist within the image volume (cognition & information)

The surface is first identified either manually or in an automated fashion by computer algorithm

The identified surface is represented either as a set of contours or as a surface locus within a volume that reflects light from an imaginary light source

Surface display shows light reflectors, while direct display show light emitters

Direct display facilitates editing and searching

Chapter 3: CS689 39

3.7e: Surface Display

Shaded Surface Display

Chapter 3: CS689 40

Chapter 3: CS689 41

3.8a: Color Sensitivity

The human visual system is highly sensitive to color. In favorable light, humans can detect more than 1 million colors, and non-color-blind individuals can differentiate 20,000 colors

Too many colors in an image usually cause confusion in encoding abstract information

X-ray images are usually displayed in black and white. Colors may be added to make the image more vivid (in CT images)

Colors may be used to represent maps of functions (temperature, pressure, electrical activity), and to distinguish one anatomical object from its surroundings

Chapter 3: CS689 42

3.8b: Colored Brain Images

Chapter 3: CS689 43

3.9a: Two-dimensional Displays Modern computer workstations generally include a

color “bitmapped” display for visual interaction with the system

They generally use cathode ray tubes or large liquid crystal devices (LCD)

Bitmapping is a technique of breaking the surface of a 2D display screen into individually addressable pixels

The value of each pixel is stored in a memory address in the graphics cards, and is scanned and painted onto the screen by the display system many times per second

Chapter 3: CS689 44

3.9b: Presentation Formats

For display of biomedical images, the most important characteristics of the display system are the number of bits per pixel, resolution, and color properties

8-bit gray scale or color mapped displays may not be sufficient for some images containing color gradients over large areas

For maximum color performance, a graphics display subsystem is implemented with three separate raster channels for each of the red, blue, and green guns in the cathode ray tube. Each color is represented with 8-bit value of pixel

Chapter 3: CS689 45

3.9c: Cathode Ray Tube

Chapter 3: CS689 46

3.9d: Color Displays on CRT

Shadow mask CRT close-up Aperture grille CRT close-up

Chapter 3: CS689 47

3.10a: Ergonomics (Work Environment) Our ability to statistically distinguish different

intensities of light is limited to less than two decades

To maximize the use of light-emitting displays, the iris should be fully open

The ambient light in the room should be near the dark end of the gray scale of the display system

A darkened room is optimum for highly detailed analysis and diagnosis

Chapter 3: CS689 48

3.10a: Darkened Room Picture

Chapter 3: CS689 49

3.10b: Printing Biomedical Images Printing of digital biomedical images has

characteristics similar to image display (pixels per inch and color resolution)

Color printing uses color primaries orthogonal to that of display system, since printed materials absorb light whereas displays generate light

The colors for printing are cyan, magenta, and yellow. (Red, green, and blue for display)

Ink jet printers, laser printers, and dye sublimation printers

Chapter 3: CS689 50

3.11a: Three-dimensional Displays Most high performance graphics computers have 3D

graphics accelerators – a special hardware to assist in the display of 3D surface images on computer screens

These accelerators have clock (processing) speed faster than most CPUs. They construct texture maps and inject them into the rendered scene to paint the surfaces of the graphical objects

It is possible to utilize the graphics accelerators to do some numerical computing work, to speed up the computation of image data

Chapter 3: CS689 51

3.11b: Stereoscopic Visualization The simplest and most common method of

generating 3D image displays is to present a pair of appropriately offset 2D images, one for each eye

Cross-fusing is to cross one’s eyes while viewing a pair of images. (Need some practice to adjust the eyes quite rapidly)

The picture will fuse into a single perceived image in the center of the two separate images

Cross-fused images are popular and effective, but they are not natural vision and can be strenuous

Chapter 3: CS689 52

3.11c: 3D Cross-Fusing Stereo View

Chapter 3: CS689 53

3.11d: Other 3D Stereo Devices Rapidly alternating the presentation of the two

stereoscopic images to the eyes and utilizing the persistence effect in the brain to blend the separate images into a single 3D image

Can be produced with computer monitors equipped with double-buffered graphic systems

Special glasses are needed to view these alternating images, with an LCD eyepiece that can be switched on and off alternately

Need synchronization between the switch and screen

Chapter 3: CS689 54

3.11e: 3D Stereo Images

Chapter 3: CS689 55

3.12a: DICOM

DICOM (Digital Image Communications in Medicine) provides a protocol for transmission of image based on their modality, and incorporates metadata for each image within the message

Each DICOM image message provides the basic information required to attach it to a patient or an imaging procedure, encoded information may be redundant

DICOM requires the sending and receiving computers to agree on a common basic method of communication, and a set of well-defined services (such as image storage) specified before the message is sent

Chapter 3: CS689 56

3.12a: DICOM Demonstration

Chapter 3: CS689 57

3.12b: Metadata: The Image Header The first part of many image files provides a

description of the image – file header The information in the header is called the

metadata (the information about the image) Some file formats use formatted header fields to

describe the image, e.g., information can be image type, height and width of the image

Flexible fields such as TIFF (tagged image file format) defines a set of tags, or field definitions, that may be present or absent in an image file

Chapter 3: CS689 58

3.12c: DICOM File

A saved DICOM transmission message file has a type of tagged file format. Everything has a tag, a size, and a value

Image pixels are described as the value of a pixel tag

There are a minimal set of standard tags required, followed by a minimal set of tags for each modality

Many optional tagged values may be included, and the specification includes tags for proprietary data, allowing vendors or developers to encode data specific to their machines or process

Chapter 3: CS689 59

3.12c: First Part of a DICOM File

Chapter 3: CS689 60

3.12c: A DICOM File, composed with data header and the image

Chapter 3: CS689 61

3.12c: Captured DICOM Images

Chapter 3: CS689 62

3.12d: Pixel/Voxel Packing

Most pixel values are stored as binary numeric values with a fixed number of bits

Monochrome data needs a single bit; gray scale data is stored as 8 bits, 12 bits, or 16 bits

For images with multiple channels in each pixel, such as color encoded as RGB, image file formats may incorporate packed or planar schemes

In packed scheme, pixels are grouped by pixel, such as RGB, RGB, RGB, etc

In planar scheme, all of the pixels for one color are placed together as a 2D image, such as RRR, followed by GGG followed by BBB

Chapter 3: CS689 63

3.12d: Pixel Packing

Chapter 3: CS689 64

3.12e: Image Compression

Image compression means to express in compact form the information contained in an image

The resulting image should retain the salient (or exact) features necessary for the purpose for which it was captured

For legal purpose, it is often necessary to insure that decompression restores the image to the same values by which any diagnosis was based

Two main types of image compression algorithms: lossless and lossy

Chapter 3: CS689 65

3.12f: Lossless Image Compression Lossless compression perfectly recovers the

original image after decompression, and works by simply taking advantage of any redundancy in the image, e.g., Huffman encoding, run-length encoding, etc.

Most image data has pixel values or groups of pixels that repeat in a predictable pattern

It typically achieves ratios of 2:1 or 3:1 on medical images

Chapter 3: CS689 66

3.12f: Lossless Compression

Chapter 3: CS689 67

3.12g: Run-length Encoding

The image pixels are scanned by rows, and a count of successive pixel values along with one copy of the pixel are sent to the output

Very effective for large area of constant color Not good for images containing many smooth

gradations of gray or color values An example:Original 2211333334333222 (16 characters)Compressed 22 21 53 14 33 32 (12 characters)

Chapter 3: CS689 68

3.12g: Run-Length Encoding

Chapter 3: CS689 69

3.12g: Run-Length Encoding

Chapter 3: CS689 70

3.12g: Run-Length Encoding

Chapter 3: CS689 71

3.12h: Huffman and Limpel-Ziv Coding These schemes search for patterns in the

data that can be represented by a smaller number of bits

These patterns are mapped to a number of representative bytes and stored by probabilities

Original 12334123343212334 (17 characters) Map 12334 = A, 32 = B Compressed AABA (13 characters)

Chapter 3: CS689 72

3.12i: Lossy Image Compression Lossy compression changes the image

values, but attempts to do so in a way that is difficult to detect or has negligible effect on the purpose for which the image will be used

It can accomplish 10:1 to 80:1 compression ratios before change in the image is detectable or deleterious

Typical techniques are JPEG and wavelets Most often used to reduce bandwidth

required to transmit image data over internet

Chapter 3: CS689 73

3.12i: Lossy Compression

Chapter 3: CS689 74

3.12j: JPEG

The JPEG standard was developed for still images, and performs best on photographs of real-world scenes

The first pass transforms the color space of the image to a luminance-dominated color space, and then downsamples the image and partitions it into selected blocks of pixels

A discrete cosine transform (DCT) is applied to the blocks

The resulting DCT values for each block are divided by a quantization coefficients and rounded to integers

The reduced data is encoded using Huffman etc

Chapter 3: CS689 75

3.12j*: JPEG Lossy Compression

Original: 43K Medium compress: 13K

Too much: 3.5k 256 colors in Netscape

Chapter 3: CS689 76

3.12l: Wavelets

Wavelet transform coefficients are partially localized in both space and frequency, and form a multiscale representation of the image with a constant scale factor, leading to localized frequency subbands with equal widths on a logarithmic scale

Wavelet compression exploits the fact that real-world images tend to have internal morphological consistency, locally luminance values, oriented edge continuations, and higher order correlations, like textures

For CT images, the compression ratio can be 80:1 For chest X-rays, the ratio may be around 10:1

Chapter 3: CS689 77

3.12l*: Example of Wavelet Compression

Original image589824 bytes

JPEG image45853 bytes

Wavelet image45621 bytes

Chapter 3: CS689 78

3.12l: Wavelet Compression

Chapter 3: CS689 79

3.13a: Image Database

Basic requirements for an image database are a means to efficiently store and retrieve the images using an indexing method

Hierarchical file systems, combine hard disk drives with optical or magnetic tape media, and through a file system database provide what appears to be a monolithic set of files to the users

Frequent requests for a large number of files may swamp such a system

Chapter 3: CS689 80

3.13b: Where to Store the Images The images can be stored “inside” or

“outside” of the databases A much smaller metadata about the image

can be stored in the database A small-scale reference image, called

“thumbnail”, extracted from the image can be used in a metadata. It is several orders of magnitude smaller than the original image

An index is needed to link the metadata with the original image

Chapter 3: CS689 81

3.13b: Thumbnail Example

Chapter 3: CS689 82

3.13c: DICOM Database

DICOM database provides a hierarchical tree structure wherein the patient is at the top of the tree

For each patient, there can be several studies, including an image examination by a given modality

Within each study, there can be a series of images, where each series can represent different viewpoints of the patient within the study

Each series may be a single or a set of images DICOM inherently organizes images in a most

suitable way for use in a treatment setting

Chapter 3: CS689 83

3.13d: DICOM Data Hierarchy

Chapter 3: CS689 84

3.13e: DICOM Database (I)

Chapter 3: CS689 85

3.13f: DICOM Database (II)

Chapter 3: CS689 86

3.13f: DICOM Database (III)