Biomedical Engineering in the Metroplex: Hyperspectral Imaging

Biomedical Engineering in the Metroplex: Hyperspectral Imaging

for Medical Use

3/13/2012 SURGERY@utsouthwestern.edu 1

Edward H. Livingston, MD, FACS, AGAF

Professor and Chairman-Gastrointestinal and Endocrine Surgery Professor and Chairman-Graduate Program in Biomedical Engineering

University of Texas Southwestern School of Medicine Contributing Editor-JAMA

The Problem

• Surgeons cannot see what they are doing when they operate

• Does this worry you ? (it should)

3/13/2012 2 SURGERY@utsouthwestern.edu

The Problem

• Gallbladder Disease is one of the most frequent causes for hospitalizations and surgery

• 750,000 hospital admissions annually

• >500,000 Cholecystectomies performed annually

• 0.25-0.5% Bile Duct Injury Result in up to 2,500 injuries per year.

What Does The Gallbladder Do?

•

3/13/2012 4 SURGERY@utsouthwestern.edu

What Things Should Look Like

3/13/2012 6 SURGERY@utsouthwestern.edu

What They Really look Like

3/13/2012 7 SURGERY@utsouthwestern.edu

What Keeps Us Out Of Trouble?

• 5+ Years of intense training after medical school

• Inherent skill, experience

• Intuition, touch, being luckier than good.

• Does these characteristics meet an engineers standards for reliability, reproducibility, QC?

3/13/2012 8 SURGERY@utsouthwestern.edu

Available Technology

3/13/2012 9 SURGERY@utsouthwestern.edu

We Need Engineering Solutions

• Develop an imaging system that penetrates opaque tissues

• Identify important structures by their unique chemical composition

3/13/2012 10 SURGERY@utsouthwestern.edu

Solutions

• Infrared imaging-penetrates tissue

• Spectral Imaging-Identify the chemical spectrum for bile

3/13/2012 11 SURGERY@utsouthwestern.edu

Bile is Green

3/13/2012 12 SURGERY@utsouthwestern.edu

Hyperspectral Imaging

3/13/2012 13 SURGERY@utsouthwestern.edu

LCTF HSI

3/13/2012 SURGERY@utsouthwestern.edu 14

Figure 1. In vivo visible-reflectance hyperspectral imaging system. a, Broadband light

source produces radiation, which is reflected off a subject and spectrally discriminated by

a liquid crystal tunable filter (LCTF), transduced by a CCD detector, and digi...

Zuzak K J et al. Circulation 2001;104:2905-2910 Copyright © American Heart Association

Near Infrared Macroscopic Hyperspectral Imaging

Gallbladder, Cystic Duct and Liver Tissue

650 750 850 950 1050 0.0

0.2

0.4

0.6

0.8

1.0

Wavelength nm A

pp

aren

t Ab

sorb

ance

650 750 850 950 1050 0.0

0.2

0.4

0.6

0.8

1.0

Wavelength nm

Ap

par

ent A

bso

rban

ce

Gallbladder

Liver

650 750 850 950 1050 0.0

0.2

0.4

0.6

0.8

1.0

Wavelength nm

Ap

par

ent A

bso

rban

ce

Cystic Duct

Principle Component Analysis 60 pound swine, post mortem

Zuzak & Livingston

Problems

• The system was too slow

• 1 minute acquisition time

• Liquid crystal filter was limiting

• Too slow for clinical use

• Spectral artifacts from motion

3/13/2012 17 SURGERY@utsouthwestern.edu

Solution

• Collaboration between UTSW and UTA

• Partner with TI and Elcan

• Develop a DLP based system to generate light of specific wavelengths at discrete time intervals

• Vast system performance improvement

3/13/2012 18 SURGERY@utsouthwestern.edu

DLP

3/13/2012 SURGERY@utsouthwestern.edu 19

Initial HSI

3/13/2012 20 SURGERY@utsouthwestern.edu

10-04-2011 Case 07 (Schwarz)

Image Frame # 26 of 101 (Wavelength = 480.00)

CBD Arteriole

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

380 480 580 680 780

Ap

par

en

t A

bso

rban

ce (

a.u

.)

Wavelength (nm)

10-4-2011 CBD

Arteriole

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

380 480 580 680 780

Ap

par

en

t A

bso

rban

ce (

a.u

.)

Wavelength (nm)

10-4-2011 CBD

Arteriole

Image Frame # 26 of 101 (Wavelength = 480.00)

10-18-2011 Case 08 (Schwarz)

Procedure notes state middle structure is CBD

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

380 430 480 530 580 630 680 730 780

Ap

par

en

t re

fle

ctan

ce i

nte

nsi

ty (

a.u

.)

Wavelength (nm)

Unfiltered Spectra 07-19-2011 CBD

05-26-2011 CBD

08-09-2011 CBD

08-30-2011 CBD-6

08-30-2011 (2) CBD

09-20-2011 CBD

10-04-2011 CBD

10-18-2011 CBD

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

380 430 480 530 580 630 680 730 780

No

rmal

ize

d a

pp

are

nt

refl

ect

ance

(a.

u.)

Wavelength (nm)

Filtered and Normalized Spectra

07-19-2011 CBD

05-26-2011 CBD

Avg norm spectrum

08-09-2011 CBD

08-30-2011 (1) CBD-6

08-30-2011 (2) CBD

09-20-2011 CBD

10-04-2011 CBD

10-18-2011 CBD

Hemoglobin

• Least Square Solution

• Obtain pure substance spectral standards (saturated and destaurated Hgb)-Vector P

• Projection Matrix

• Project the observed spectra onto the saturated and desaturated Hgb planes to get the amount of Hgb

3/13/2012 SURGERY@utsouthwestern.edu 25

Oxyhemoglobin

Deoxyhemoblobin

520 540 560 580 600 620 640 Wavelength (nm)

520 540 560 580 600 620 640 0.0 0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

1.0

0.9 0.9 0.9

Ap

par

ent A

bso

rban

ce

Axy(

λ i)=

-lo

g 10(R

xy(

λ i)

/ R

xy(

λ i) ∞

)

Reference Spectral Components:

Oxyhemoglobin and Deoxyhemoglobin

520 0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1 Measured

Superposition

HbO2 Reference

Hb Reference

540 560 580 600 620 640

Wavelength (nm)

Deconvolution Example

520 0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1 Measured

Superposition

HbO2 Reference

Hb Reference

540 560 580 600 620 640

Wavelength (nm)

Deconvolution Example

520 0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1 Measured

Superposition

HbO2 Reference

Hb Reference

540 560 580 600 620 640

Wavelength (nm)

Deconvolution Example

DLP Hyperspectral Imaging System

Focal Plane

Array

Programmable Spectral

Light Source DLP® Inside

Tissue

Sample

COMPUTER

Visualization

Synchronized

Hardware

Control

Image

Processing

Acquire Data Cube Digitized

Hyperspectral Image

Cube

Graphical User Interface (GUI)

DLP Hyperspectral Imaging System

GUI developed by:

Timing Diagram for DLP HSI with 126shot Method

Open Shutter HQ2

Expose CCD

Close Shutter

Save as temp00n.dat

A/D conversion

Command OL 490 to

Illuminate Spectrum n

n = n+1

Start Acquisition

(n = 1)

Is n = 126? no yes

Process with Oxyz

Algorithm in MATLAB

Output bitmap image

tshot = 116.3 ms

tacquisition = 116.3 * 126 = 14 648 ms

tprocessing = 8 432 ms

40 ms

Ttotal = 23 080 ms

Timing Diagram for DLP HSI with 3shot Method

Open Shutter HQ2

Expose CCD

Close Shutter

Save as temp00n.dat

A/D conversion

Command OL 490 to

Illuminate Spectrum n

n = n+1

Start Acquisition

(n = 1)

Is n = 3? no yes

Process with 3shot

Algorithm in MATLAB

Output bitmap image

tshot = 93.3 ms

tacquisition = 93.3 * 3 = 280 ms

tprocessing = 110 ms

1.25 ms

Ttotal = 390 ms

R² = 0.3774

0.6

0.7

0.8

0.9

1

1.1

1.2

0 10 20 30 40 50 60 70

Series1

Poly. (Series1)

Rel

ativ

e %

Hb

O2

No

rmal

ized

to

Co

ntr

ol

Time after Clamping (minutes)

Clamped

Control

Cut

Clamp Off

Iced

Images from Human Partial Case 15 (12-18-09) and Plot of all Cases

Control 8 min after clamping 1 min after clamp off 10

30

50

70

90

Relativ

e Co

ntrib

utio

n o

f Hb

O2

Video Rate Hyperspectral Imaging

Biochemical Visualization of Human Kidney Over Time

Animal Study

DLP Hyperspectral Imaging

• Picture of pig kidney immediately after arterial clamping

• Assuming arterial blood is shunted away from kidney by arterial clamp

Invasive Lycox probe inserted into kidney

Kidney

DLP Hyperspectral Imaging

Kidney

10

20

30

40

50

60

70

80

90

Rel

ativ

e C

ontr

ibuti

on o

f H

bO

2

-Discovery:

The Kidney had two renal arteries.

Only the lower pole artery was clamped,

Hyperspectral Imaging immediately indicated

continued perfusion of the upper Kidney pole.

Hyperspectral Image discovers Pig Kidney with 2 Renal Arteries

DLP Hyperspectral Imaging

Kidney

10

20

30

40

50

60

70

80

90

Rel

ativ

e C

ontr

ibuti

on o

f H

bO

2

Hyperspectral Image of Pig Kidney with Both Renal Arteries Clamped

Re-situating the clamp over both

arteries led to ischemia of both

upper lower poles.

Note:- 3 shot Images showing more oxygenation in intestines as kidney becomes deoxygenated

5 min 32 sec before clamping

4 min 44 seconds before clamping

Clamp on 1 minute after clamping

2 minutes after clamping

3 minutes after

clamping

0

10

20

30

40

50

60

70

80

90

100

-6 -5 -4 -3 -2 -1 0 1 2 3 4

Kidney

Intestine

Time before and after Clamping in Minutes

Clamp On

% H

bO

2

Pig Partial Nephrectomy Trial 2

0

20

40

60

80

100

120

-40 -20 0 20 40 60 80 100

Kidney

Intestines

Clamp On (100%) Clamp Off

Re

lati

ve P

erc

ent

Hb

O2

Time before and after Clamp (Minutes)

Control 27 minutes before clamping

5 minutes after clamping

15 minutes after clamping

34 minutes after clamping

57 minutes after clamping

1 minute after clamp off

14 minutes after clamp off

Pig Partial Nephrectomy Case 1 (Kidney and Intestines)

Control (0 min)

Clamped (15 min)

Clamped & After Icing (11 min)

After Clamping (3 min)

Clamped (21 min)

Clamped (31 min)

After Clamp Off (33 min)

Clamp Off (1 hr 5 min)

Human Partial Nephrectomy Case 3

Partial Nephrectomy

0

10

20

30

40

50

60

70

80

90

100

0 10 20 30 40 50 60 70

Time in Minutes

Rela

tive P

erc

en

t H

bO

2

% HbO2

Clamp On Clamp Off Clamp On & After Icing

In Vivo, Visible Reflectance, Hyperspectral Imager

Coupled to a Zeiss Neurosurgical Microscope

Source

Illumination

Reflection

Of Subject

10

20

30

40

50

60

70

80

90

% H

bO

2

Control After Induced Ischemia

Damaged Tissue Normal Tissue

Brain Surgery Monitoring for

Preventing Post Surgery Stroke

DLP Hyperspectral Microscope

Monitoring Diabetic Retinopathy

Double Knock Out (Apoe-/-, db/db) Mouse Model

B

10

30

50

70

90

10

30

50

70

90

Rel

ati

ve O

xyh

emo

glo

bin

Parental 2: Wildtype Apoe /Hom db

Day: 81 Day: 151

Parental 1: Hom Apoe/ Wildtype db

Day: 79 Day: 142

Double Knock Out Diabetic Mouse:

Hom Apoe/ Hom db

Day: 78 Day: 136

Data Collected Using LCTF

LCTF Hyperspectral Imaging System

For Imaging Human Retina

0

Visible LCTF

Illumination Mirror CCD

Beam Splitter

Magnification

Knob

Light Source

Joystick

Table Top

Common Center

of Rotation

Subject’s

Head rest

Intensity Control

Knob

Eye Piece

10

20

30

40

50

60

70

80

90

Rela

tive

Oxyh

em

og

lob

in

Con

trib

uti

on

Gray-scale Encoded

Hyperspectral Image

530 550 570 590 0

0.2

0.4

0.6

0.8

1.0

Wavelength (nm)

Ap

pare

nt

Ab

sorb

an

ce

530 550 570 590 0

0.2

0.4

0.6

0.8

1.0

Wavelength (nm)

Ap

pare

nt

Ab

sorb

an

ce

Microscopic Hyperspectral Imager:

Human Retinal Imaging of Oxyhemoglobin Contribution %

Hb

O2

Area 1 Area 2 0

10

20

30

40

50

60

70 P < 0.0001

Hyperspectral Fundus Camera

Spectral Light Source

LLG

Fundus Camera

Laptop CCD Camera

10

20

30

40

50

60

Relative HbO2

Need to Improve

Visualization Algorithms

artifacts

In Vivo Hyperspectral Imaging of Human Tissue: Spatial Variation of Percentage of HbO2 and Surface Temperature in Response to Burn

Bright Field

1 inch

Multivariate Least Squares

with respect to Oxyhemoglobin

Ap

pare

nt A

bso

rban

ce

Ax

y(λ i

) =

-lo

g10

(Rx

y(λ i

) /

Rx

y(λ i

) ∞)

500 540 580 620 660

Spectra Under

points and

Wavelength (nm)

Thermal Image

1 inch 1 inch

BURNS

10

30

50

70

90

Relative Contribution of HbO2

10

30

50

70

90

Relative Contribution of H2O

Chemically Encoded

HbO2

Chemically Encoded

H2O Color Photo

BURNS

Post Fasciectomy

10

20

30

40

50

60

70

80

90

Relative Contribution of HbO2

10

20

30

40

50

60

70

80

90

Relative Contribution of H2O

Color Photo Chemically Encoded

Oxyhemoglobin

Chemically Encoded

Water

Patient 002,

Monitoring Post Amputation

Left Lower Limb, Foot, left BKA

Session 1

051707 Session 2

052107

Digital

Image

A B1 A B2

Vis

HbO2

A 62 3.9

B 62±1.2

A 70 4.2

B 70±3.2

10

20

30

40

50

60

70

80

90

A

B1 A

B2

NIR

HbO2

A 76.5 6.5

B 84.5±2.1

A 68.9 1.7

B 67.6±1.1 10

20

30

40

50

60

70

80

90

Per

centa

ge o

f H

bO

2

0

10

20

30

40

50

60

70

80

90

100

Per

centa

ge o

f H

bO

2

0

10

20

30

40

50

60

70

80

90

100

B1

Zuzak & Livingston

Suture Tension

Monitoring Post Amputation Recovery

A 75.83 2.59

B 76.3±0.86

Patient 002,

Amputation, Left AKA

10

20

30

40

50

60

70

80

90

B A

A 56.91 2.91

B 56.2±0.63

10

20

30

40

50

60

70

80

90

Patient 005,

Amputation, Left AKA A

B

A 85.61 3.53

B 81.58±4.38

Patient 006,

Amputation, Left BKA

10

20

30

40

50

60

70

80

90

A B

0

10

20

30

40

50

60

70

80

90

100 P

erce

nta

ge o

f H

2O

0

10

20

30

40

50

60

70

80

90

100

Per

cen

tage

of

H2O

0

10

20

30

40

50

60

70

80

90

100

Per

centa

ge o

f H

2O

Patient 002, 005, 006

Session I

Per

centa

ge o

f H

2O

Per

centa

ge o

f H

2O

Per

centa

ge o

f H

2O

85.83 % HbO2 81.05 % HbO2

Per

centa

ge o

f O

xyhem

ogl

obin

10

20

30

40

50

60

70

80

90

Hyperspectral Imaging

Visible

10

20

30

40

50

60

70

80

90

Per

centa

ge o

f oxy

hem

ogl

obin

NIR

Data collected 10th minute

after removing the tight shoe

71.7893 +/- 3.7673

79.1163 +/- 2.208

57.3525 +/- 5.7005

62.0387 +/- 3.2884

63.8259 +/- 5.3345

69.8058 +/- 2.8421

Diabetic Neuropathy

Other Potential Applications

• Neurological Surgery - Preventing Stroke After Brain Tumors.

• Plastic Surgery - Skin flaps, Wound Management and Burns.

• Surgery - Non-Invasive Laparoscopic Optical Biopsy During.

• Urology – Tumor Removal during Kidney Surgery.

• Ophthalmology – Monitoring Diabetic Retinopathy.

• Clinical Monitoring – Managing Lower Limb Amputations.

The speed and versatility of DLP® technology is making

Hyperspectral Imaging practical for a wide variety of

surgical and clinical applications.

Versatility

3/13/2012 SURGERY@utsouthwestern.edu 57

Placenta Control

Prior to ICG infusion

Vascular infusion

Of ICG via cannual

Fluorescence of ICG Vascular Infusion

and

DLP Spectral Excitation

Picture of Placenta

Prior to ICG infusion

The DLP Illuminates tissue with predetermined

spectral illumination exciting the ICG to fluoresce.

Major seed funding provided by

• Texas Instruments

Supplemental funding provided in part by

• Hudson-Pen Endowment

• Smith Endowment

• Department of Energy.

BME in the Metroplex

3/13/2012 SURGERY@utsouthwestern.edu 60

1972

3/13/2012 SURGERY@utsouthwestern.edu 61

The Future

3/13/2012 SURGERY@utsouthwestern.edu 62