-
IMAGE MOSAICING AND REAL-TIME IMAGING WITH MEMS BASED HANDHELD
CONFOCAL MICROSCOPE
Wibool Piyawattanametha
Faculty of Engineering, King Mongkuts Institute of Technology
Ladkrabang, Thailand
Advanced Imaging Research Center, Faculty of Medicine,
Chulalongkorn University Thailand
ABSTRACT In this paper, we demonstrated a handheld confocal
fluorescence microscope using dual-axis confocal architecture with
a microelectromechanical systems (MEMS) scanner. The laser sources
for the microscope are both in visible (660 nm) and near-infrared
(778 nm) wavelengths. The microscope will be used for in vivo
cervical cancer screening in real-time with human patients. The
maximum imaging depth is over 300 m into the tissue for epithelial
cancer screening. The maximum field of view (FOV) is 550 m 500 m
with 14 frames/second.
1. INTRODUCTION
Over the past several decades, medical imaging technologies such
as PET, MRI, and CT have played a critical role in the early
detection and treatment of cancer. These imaging modalities have
the ability to detect cancer at the anatomic scale (e.g. tumors)
but lack the resolution needed to see cancer at the cellular scale
(e.g. colon or skin cancers). As a result, the diagnosis of cancer
at the cellular scale is usually performed through tissue biopsy
and subsequent pathology. Recently, a new class of miniature
confocal microscopes has emerged that can detect cancerous and
pre-cancerous tissues at the cellular scale [1-2]. The endoscope
has the potential to revolutionize health care by replacing tissue
biopsies with in vivo pathology for fast, cheap, and non or
minimally-invasive screening. Cervical cancer is the first and
second most common cause of cancer death in women in Asia and
developing countries, respectively [3]. At present, popular
diagnostic techniques are Papanicolaou (Pap) and Human Papilloma
Virus (HPV) Deoxyribonucleic acid (DNA) tests [4]. However, the
aforementioned techniques require sample preparations and medical
expertise to correctly diagnose results [5]. The purpose of our
work is to develop a miniature cervical cancer screening tool based
on confocal microscopy technique for in vivo real time diagnosis.
Confocal microscopy technique offers subcellular to cellular
resolution with optical section property enabling three-dimensional
(3D) imaging [6].
2. CONFOCAL SYSTEM SETUP
Fig. 1a shows a photograph of handheld dual-axis confocal
microscope. It is a rigid type microscope with 10-mm diameter at
its imaging tip. Furthermore, the length of the microscope is
designed to be 200 mm compatible with the actual length of human
cervix (60-140 mm) [5]. Fig. 1b shows optical setup of the
microscope. Excitation wavelength is 778 nm. The laser light is
coupled into an input single-mode optical fiber (numerical aperture
0.12) achromat collimator (2.8 mm in diameter) as shown in Fig. 2.
The light exiting collimator has its full width at half maximum
(FWHM) diameter of 950 m. The collimated beam is focused by a
parabolic mirror (focal length of 4.7 mm) before reflecting of from
one side of the MEMS mirror [3]. MEMS scanner die size is 3.23.1
mm2 with the mirror shape of a barbell to accommodate input
illumination and output collection.
Figure 1: (a) Photograph of handheld dual-axis confocal
microscope (b) Schematic drawing of the microscope showing internal
components. Scale bar is 5 cm.
The focused beam exits the microscope housing via a solid
immersion lens (SIL) and illuminating on either tissues soaking in
indocyanine green (ICG) dye or
978-1-4799-4963-2/14/$31.00 2014 IEEE
2014 International Computer Science and Engineering Conference
(ICSEC)
978-1-4799-4963-2/14/$31.00 2014 IEEE 140
-
specimens. A MEMS scanner will perform a raster scanning with
point by point illumination over the entire FOV. Fluorescent signal
from the sample is defocused into the output path of the microscope
via the collection side of the microscope consisting of a MEMS
mirror, a parabolic mirror, and an output collimator. The maximum
imaging frame rate is 14 frames/second. Then, the fluorescence
signal is travelling through an emission long pass filter (790 nm)
to block the excitation wavelength before entering a
photomultiplier tube (PMT). A low noise amplifier is used to
convert light into an electrical signal, which then can be acquired
by a data acquisition (DAQ) system via LabView. The acquired image
is displayed on personal computer in real-time. Imaging depth, FOV,
contrast, gain adjustment can be executed through the custom
software in LabView. Image mosaicing has been implanted to help
enlarge FOV into a millimeter-scale for ease of tissue
interpretation during real-time diagnosis.
Figure 2: Schematic drawing of a 2.8 mm diameter achromat
collimator assembly. The FWHM beam size is approximately 1.8
mm.
3. IMAGE MOSAICING Image mosaicing is a technique to enlarge the
FOV for image display based on multi-picture alignment method and
commonly used in the field of vision-based robot navigation systems
and virtual reality [7]. Moreover, this method can be applied to
microscopes or endoscopes in displaying the whole interested areas,
which cannot be shown within only one video frame shot. Individual
small frame shots used in image mosaicing technique requires the
overlapping display areas for the utility of image alignment to be
as smooth as possible. Image mosaicing application in biomedical
imaging solves the problem in FOV limitation. Microscope and
microendoscope integrated with image mosaicing technique are able
to perform the high-resolution panorama image in subcellular level
for ex vivo and in vivo abnormal tissue detection and diagnostics
in real time [8]. Therefore, rapid image processing for feature
frame alignment used in such equipment is necessary to instantly
display the interested field. Image mosaicing method is
generally
divided into two main processes consisting of image registration
and image blending process. Image registration is a method to
identify the coordinate relationship between two aligned frames,
where the overlapping areas in each picture are similar to each
other as much as possible, and place the feature frames in proper
positions [8]. Image blending process provides the method for
adjusting light intensity of the images at the edge of frames
alignment to improve the image display concord. In this work, we
have presented the image blending process based on computer vision
technique. The limited FOV of tissue section sequence frames taken
from optical microscope were enlarged and displayed by image
mosaicing technique. Computer vision technique allows us to
estimate the moving length between the present and the previous
feature frame for image alignment process. This motion tracking
technique provides the rapid moving length calculation method to
determine the frame positions in the area of interest [9].
4. RESULTS
4.1 Microscope resolution We use a multi-wavelength generator
(Qioptic, Inc.) to couple more than one wavelength into the
handheld microscope. Each wavelength has the maximum output power
of around 40 mW. However, after coupling the laser power into the
handheld imaging system, typical power output measured at the tip
of the microscope is in the range of 2-4 mW. The first imaging test
is to measure the performance of the handheld microscope with a
laser beam profiler with a 100 objective lens as shown in Fig. 3.
An optical fiber with 50:50 coupling ratio is used to split the
laser power into half and feeding both ends into both input and
output collimators. The intersection of both input and out beams
defines the microscope resolution. The center wavelength used in
this experiment is 660 nm. The axial response is measured at the
output of collection collimator with a power meter while
translating a perfect mirror in axial direction through the focal
imaging plane. Resolution results are shown in Fig 4. The
full-width-at-half-maximum (FWHM) of transverse (X and Y
directions) resolutions are 4.1 and 3.6 m, respectively (Fig. 3a)
while FWHM of axial (Z direction or depth) resolution is 7.2 m
(Fig. 3b).
4.2 Resolution and biological samples The reflection image
acquired from the handheld confocal microscope shows group 7,
element 6 of the United States Air Force (USAF) standard resolution
target (Fig. 5). The porcine colon images are shown in Fig. 6. The
tissue sections are rinsed for a few times with phosphate buffer
saline (PBS) solution pH 7.2, and subsequently immersed in 5 mg/ml
of ICG in PBS pH
2014 International Computer Science and Engineering Conference
(ICSEC)
141
-
7.4 for 1 hour. After that, the tissue sections are rinsed for
several times to remove excess dye, and are captured by both the
standard microscope (Fig. 6a) and the handheld dual-axis confocal
microscope (Fig. 6b) for comparison. The maximum FOV is 550 m 500
m. Image gain level can be control through the software to balance
the image contrast over the entire FOV.
4.3 Image stitching
To demonstrate concept of our image stitching
algorithms, we used high-resolution images obtained from a
microscope with cellular definitions, we captured a high-resolution
image (1228 pixel 802 pixel) from a high-resolution camera
(Olympus, Inc., Model: DP72) attached to a microscope as shown in
Fig. 7(a). A sequence of image (120 frames) will be randomly
cropped and saved along x- and y-axes from this high-resolution
image into many synthetic small images to be used with our optical
flow mosaicing technique. Each cropped image will be exported as a
video file with frame rate of 10 Hz. After we have processed the
video data, the result is shown in Fig. 7(b). The mosaicing
alignment accuracy is calculated to be at over 87 percent with 20%
to 40% overlapping features between each frame. The maximum input
video frame rate to be processed with our algorithms is estimated
to be 30 Hz.
(a)
Figure 3: Microscope x-y resolution measurement setup, scale bar
is 4 cm.
Figure 4: Measured resolutions of handheld dual-axis confocal
microscope. (a) FWHM transverse (X-Y direction) resolutions are 4.1
and 3.6 m. (b) FWHM axial (Z direction) resolution is 7.2
Figure. 5: The reflection image shows group 7 of USAF standard
resolution target, scale bar is 20 m.
Figure 6: Porcine colon images. (a) a 40 optical image from the
Olympus microscope model BX63. (b) an image from the handheld
dual-axis confocal microscope, scale bar are 20 m.
2014 International Computer Science and Engineering Conference
(ICSEC)
142
-
(b) Figure 7: (a) A high-resolution image of hypo-plastic tissue
sample acquired with a high-resolution camera attached to a
microscope. (b) An implemented result of mosaicing algorithms.
1. SUMMARY
We have demonstrated a MEMS scanner based cervical confocal
imaging probe and successfully image ex vivo tissues of porcine
colon with cellular resolution. Performance of the handheld
microscope has been characterized. Real-time cancer diagnosis is
capable with high-speed imaging (upto 14 Hz). Image real-time
mosaicing program will be implemented in the experiments to enable
FOV enlarge for ease of use. We anticipate broad sets of in vivo
and ex vivo experiments with this imaging tool in the near
future.
5. ACKNOWLEDGEMENT
The work is support in part by the Fraunhofer-Bessel Research
Award from the Alexander von Humboldt Foundation, Germany.
6. REFERENCES [1] W. Piyawattanametha and T. D. Wang, MEMS-
Based Dual Axes Confocal Microendoscopy (Invited Paper), The
IEEE Journal of Selected Topics in Quantum Electronics (JSTQE),
July-August 2010, Vol. 16, Issue 4, pp.804-814.
[2] N. Khemthongcharoen, S. Rattanavarin, R. Jolivot, and W.
Piyawattanametha, Advanced in imaging probes and optical
microendoscopic imaging techniques for early in vivo cancer
assessment (invited paper), Journal of Advanced Drug Delivery
Reviews, October 9, 2013, 10.1016/j.addr.2013.09.012.
[3] World Health Organization, Cervical cancer screening in
developing countries: report of a WHO consultation, WHO Library
Cataloguing-in-Publication Data, 2002.
[4] J. S. Mandelblatt, W. F. Lawrence, S. M. Womack, D.
Jacobson, B. Yi, Y. T. Hwang, K. Gold, J. Barter, and K. Shah,
"Benefits and costs of using HPV testing to screen for cervical
cancer", Journal
of the American Medical Association, Vol. 287, pp.2372-2381,
2002.
[5] W. Piyawattanametha, H. Ra, Z. Qiu, S. Friedland, J. T. C.
Liu, K. Loewke, G. S. Kino, O. Solgaard, T. D. Wang, M. J.
Mandella, and C. H. Contag, In Vivo Near-infrared Dual-Axis
Confocal Microendoscopy in the Human Lower Gastrointestinal Tract,
Journal of Biomedical Optics 17(2), February 2012, 021102:1-4.
[6] J. S. Tan, E. S. Lukacz, S. A. Menefee, K. M. Luber, M. E.
Albo, and C. W. Nager, "Determinants of vaginal length", American
journal of obstetrics and gynecology, Vol. 195, pp.1846-50,
2006.
[7] Cobzas, D., Jagersand, M. and Zhang, H., A Panoramic Model
for Remote Robot Environment Mapping and Predictive Display,
International Journal of Robotics and Automation, 20(1):25-34,
2005.
[8] W. Piyawattanametha, M. Mandella, H. Ra, J. Liu, E. Garai,
G. Kino, O. Solgaard, and C. Contag, MEMS base dual-axes confocal
clinical endoscope for real time in vivo imaging, Proceedings of
Optical MEMS and Nanophotonics, pp. 42-43, 2008.
[9] R. Szeliski, Video mosaics for virtual environments, IEEE
Computer Graphics and Applications, vol. 16, no. 2, pp. 22-30,
1996.
2014 International Computer Science and Engineering Conference
(ICSEC)
143
/ColorImageDict > /JPEG2000ColorACSImageDict >
/JPEG2000ColorImageDict > /AntiAliasGrayImages false
/CropGrayImages true /GrayImageMinResolution 200
/GrayImageMinResolutionPolicy /OK /DownsampleGrayImages true
/GrayImageDownsampleType /Bicubic /GrayImageResolution 300
/GrayImageDepth -1 /GrayImageMinDownsampleDepth 2
/GrayImageDownsampleThreshold 1.50000 /EncodeGrayImages true
/GrayImageFilter /DCTEncode /AutoFilterGrayImages false
/GrayImageAutoFilterStrategy /JPEG /GrayACSImageDict >
/GrayImageDict > /JPEG2000GrayACSImageDict >
/JPEG2000GrayImageDict > /AntiAliasMonoImages false
/CropMonoImages true /MonoImageMinResolution 400
/MonoImageMinResolutionPolicy /OK /DownsampleMonoImages true
/MonoImageDownsampleType /Bicubic /MonoImageResolution 600
/MonoImageDepth -1 /MonoImageDownsampleThreshold 1.50000
/EncodeMonoImages true /MonoImageFilter /CCITTFaxEncode
/MonoImageDict > /AllowPSXObjects false /CheckCompliance [ /None
] /PDFX1aCheck false /PDFX3Check false /PDFXCompliantPDFOnly false
/PDFXNoTrimBoxError true /PDFXTrimBoxToMediaBoxOffset [ 0.00000
0.00000 0.00000 0.00000 ] /PDFXSetBleedBoxToMediaBox true
/PDFXBleedBoxToTrimBoxOffset [ 0.00000 0.00000 0.00000 0.00000 ]
/PDFXOutputIntentProfile (None) /PDFXOutputConditionIdentifier ()
/PDFXOutputCondition () /PDFXRegistryName () /PDFXTrapped
/False
/CreateJDFFile false /Description >>>
setdistillerparams> setpagedevice