MICROSCOPY IMAGE REGISTRATION, SYNTHESIS AND SEGMENTATION A Dissertation Submitted to the Faculty of Purdue University by Chichen Fu In Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy May 2019 Purdue University West Lafayette, Indiana
124
Embed
MICROSCOPY IMAGE REGISTRATION, SYNTHESIS AND … · Charles William Harrison Distinguished Professorship at Purdue University. The images used in section 3 for microscopy image registration
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
MICROSCOPY IMAGE REGISTRATION, SYNTHESIS AND SEGMENTATION
A Dissertation
Submitted to the Faculty
of
Purdue University
by
Chichen Fu
In Partial Fulfillment of the
Requirements for the Degree
of
Doctor of Philosophy
May 2019
Purdue University
West Lafayette, Indiana
ii
THE PURDUE UNIVERSITY GRADUATE SCHOOL
STATEMENT OF DISSERTATION APPROVAL
Dr. Edward J. Delp, Chair
School of Electrical and Computer Engineering
Dr. Paul Salama
School of Electrical and Computer Engineering
Dr. Mary L. Comer
School of Electrical and Computer Engineering
Dr. Fengqing M. Zhu
School of Electrical and Computer Engineering
Approved by:
Dr. Pedro Irazoqui
Head of the School Graduate Program
iii
ACKNOWLEDGMENTS
First of all, I would like to thank my doctoral advisor Professor Edward J. Delp
for offering me the opportunity to join his research lab, Video and Image Processing
Laboratory (VIPER), and under his supervision. I am grateful to him for his guidance,
support, advice, and criticism. I am especially thankful for his trust in me and his
encouragement to me to challenge myself, to overcome obstacles and to explore in
new dimensions.
I would like to thank Professor Paul Salama for his inspiration and involvement in
microscopy image analysis. I appreciate all his invaluable time and efforts for helping
me with my research and paper. I would like to thank Professor Fengqing Zhu for her
insightful suggestions on research ideas and my future career. I would like to thank
Professor Mary Comer for her advice, support and encouragement.
I would like to thank Professor Kenneth W. Dunn for sharing his knowledge in
biology. His feedback helped me to have a new understanding of the goals of my
project.
I would like to thank all of my microscopy project team members, Dr. Neeraj
Gadgil, Mr. Soonam Lee, Mr. David J. Ho and Ms. Shuo Han. It is truly an honor
to work in this team. I would like to thank David for being a great friend and co-
worker. We have been working through so many challenge together. I would like
to thank Soonam for being a friend and helping me with my paper. I would like to
thank Shuo for being a friend and involving in my research. I could not count how
many nights we have been working together. Those will be the precious memory in
my life. I would like to thank again for their support, encouragement and heartful
advices for my research and my personal life.
iv
I would like to specially thank Dr. Neeraj Gadgil for being a great mentor at my
first year of PhD. I would like to specially thank Dr. Khalid Tahboub for helping me
with writing my first paper.
Studying and working in VIPER have been a great experience. I would like to
thank all my talented colleagues: Mr. Shaobo Fang, Mr. Yuhao Chen, Mr. Daniel
Mas, Mr. Javier Ribera, Mr. David Guera Cobo, Dr. Albert Parra, Ms. Qingchaung
2This is joint work with Ms. Shuo Han and Mr. Soonam Lee3This is joint work with Ms. Shuo Han, Mr. Soonam Lee and Dr. David J. Ho
viii
LIST OF TABLES
Table Page
2.1 Average SSD per pixel of different sample time volumes before and afterregistration and percentage of improvement. . . . . . . . . . . . . . . . . . 26
4.1 Accuracy, Type-I and Type-II errors for other methods and our methodon the Data-I . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
4.2 Accuracy, Type-I and Type-II errors for known methods and our methodon subvolume 1 of Data-I . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
4.3 Accuracy, Type-I and Type-II errors for known methods and our methodon subvolume 2 of Data-I . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
4.4 Accuracy, Type-I and Type-II errors for known methods and our methodon subvolume 3 of Data-I . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
4.5 True positive, False positive, False negative, Precision, Recall and F1Scores for known methods and our method on Data-I . . . . . . . . . . . . 77
2.2 Grayscale versions of the four different spectral channels of the 6th focalslice of the 1st time volume of the original dataset. (a) Green channel, (b)Yellow channel, (c) Red channel, (d) Blue channel. . . . . . . . . . . . . . 20
2.3 YZ view of the green channel of the original and the interpolated sampleimages. (a) Original, (b) Interpolated. . . . . . . . . . . . . . . . . . . . . 21
2.4 Sample images of our 3D non-rigid registration. (a) MIP of the sampleoriginal volume projected on XY plane, (b) MIP of the sample result of3D non-rigid registration projected on XY plane, (c) MIP of the sampleoriginal volume projected on YZ plane, (d) MIP of the sample result of3D non-rigid registration projected on YZ plane. . . . . . . . . . . . . . . . 22
2.6 MIPs of the original time volumes and registered time volumes at timesample 1,11,21,31,41,51, and 61. (a) MIP of the original volumes projectedon XY plane, (b) MIP of the result of 4D rigid registered volumes projectedon XY plane, (c) MIP of the original volumes projected on YZ plane, (d)MIP of the result of 4D rigid registered volumes projected on YZ plane. . 24
2.7 Views of MIP volumes (using ImageJ 3D viewer). (a) XY view of originalMIP volume, (b) XY view of 4D rigid registered MIP volume, (c) YZ viewof original MIP volume, (d) YZ view of 4D rigid registered MIP volume. . 25
x
Figure Page
2.8 3D spherical histograms of motion vectors using time volume 9 as themoving volume and time volume 8 as the reference volume. (a) histogramof original volume in the view from top, (b) histogram of registered vol-ume in the view from top, (c) histogram of original volume in the viewfrom bottom, (d) histogram of registered volume in the view from bottom,(e) histogram of original volume in +XY view, (f) histogram of registeredvolume in +XY view, (g) histogram of original volume in -XY view, (h)histogram of registered volume in -XY view, (i) histogram of original vol-ume in XZ view, (j) histogram of registered volume in XZ view. . . . . . . 27
2.9 3D spherical histograms of motion vectors using time volume 30 as themoving volume and time volume 29 as the reference volume. (a) histogramof original volume in the view from top, (b) histogram of registered vol-ume in the view from top, (c) histogram of original volume in the viewfrom bottom, (d) histogram of registered volume in the view from bottom,(e) histogram of original volume in +XY view, (f) histogram of registeredvolume in +XY view, (g) histogram of original volume in -XY view, (h)histogram of registered volume in -XY view, (i) histogram of original vol-ume in XZ view, (j) histogram of registered volume in XZ view. . . . . . . 28
4.8 3D visualization of Volume-I of Data-I using Voxx [106] (a) original vol-ume (b) 3D ground truth volume, (c) 3D active surfaces from [62], (d)3D Squassh from [69, 70], (e) segmentation result before refinement, (f)segmentation result from after refinement. . . . . . . . . . . . . . . . . . . 56
4.9 Nuclei count using watershed (a) original image, Iorigz175, (b) segmentation
result from our method, Isegz175, (c) watershed result, I labelz175
. . . . . . . . . . 57
4.10 Nuclei segmentation on different rat kidney data (a) Iorigz16of Data-II, (b)
4.13 Slices of the original volume, the synthetic microscopy volume, and thecorresponding synthetic binary volume for Data-I and Data-II (a) originalimage of Data-I, (b) synthetic microscopy image of Data-I, (c) syntheticbinary image of Data-I, (d) original image of Data-II, (e) synthetic mi-croscopy image of Data-II, (f) synthetic binary image of Data-II . . . . . . 64
4.14 3D visualization of subvolume 1 of Data-I using Voxx [106] (a) originalvolume, (b) 3D ground truth volume, (c) 3D active surfaces from [62],(d) 3D active surfaces with inhomogeneity correction from [108], (e) 3DSquassh from [69,70], (f) 3D encoder-decoder architecture from [43], (g) 3Dencoder-decoder architecture with CycleGAN, (h) 3D U-Net architecturewith SpCycleGAN (Proposed method) . . . . . . . . . . . . . . . . . . . . 67
xii
Figure Page
4.15 Original images and their color coded segmentation results of Data-I andData-II (a) Data-I Iorigz66
, (b) Data-II Iorigz31, (c) Data-I Isegz66
using [43], (d)Data-II Isegz31
using [43], (e) Data-I Isegz66using 3D encoder-decoder archi-
tecture with CycleGAN, (f) Data-II Isegz31using 3D encoder-decoder ar-
chitecture with CycleGAN, (g) Data-I Isegz66using 3D U-Net architecture
with SpCycleGAN (Proposed method), (h) Data-II Isegz31using 3D U-Net
4.18 Sample results of different stages of our proposed method. (a) Iseg (b)Iheat (c) dilated Ict (d) Imarkseg (e) Imarkct (f) Ifinal (g) color result . . . . 76
4.19 Sample results of Data-I (a) Original microscopy images (b) Segmentationsof Squassh (c) Segmentations of method [53] (d) Segmentations of method[53] + Quasi 3D watershed (e) Segmentations of MTU-Net . . . . . . . . . 79
4.20 Sample results of Data-II (a) Original microscopy images (b) Segmenta-tions of Squassh (c) Segmentations of method [53] (d) Segmentations ofmethod [53] + Quasi 3D watershed (e) Segmentations of MTU-Net . . . . 80
4.21 Sample results of Data-III (a) Original microscopy images (b) Segmenta-tions of Squassh (c) Segmentations of method [53] (d) Segmentations ofmethod [53] + Quasi 3D watershed (e) Segmentations of MTU-Net . . . . 81
4.22 Sample results of Data-IV (a) Original microscopy images (b) Segmenta-tions of Squassh (c) Segmentations of method [53] (d) Segmentations ofmethod [53] + Quasi 3D watershed (e) Segmentations of MTU-Net . . . . 82
4.23 Sample results of Data-V (a) Original microscopy images (b) Segmenta-tions of Squassh (c) Segmentations of method [53] (d) Segmentations ofmethod [53] + Quasi 3D watershed (e) Segmentations of MTU-Net . . . . 83
4.24 3D visualization of different methods of subvolume of Data-I. (a) Origi-nal volume (b) Groundtruth volume (c) Otsu + Quasi 3D watershed (d)CellProfiler (e) Squassh (f) Method [110] (g) Method [110] + Quasi 3Dwatershed (h) MTU-Net (Proposed) . . . . . . . . . . . . . . . . . . . . . . 84
during segmentation. Also, our proposed method has reasonably low Type-II errors
compared to other segmentation methods. Moreover, in this table, we show that our
proposed SpCycleGAN creates better paired synthetic volumes which reflects in seg-
mentation accuracy. Instead of 3D encoder-decoder structure, we use 3D U-Net which
leads to better results since 3D U-Net has skip connections that can preserve spatial
information. In addition, the combination of two loss functions such as the Dice loss
and the BCE loss turns out to be better for the segmentation task in our application.
In particular, the Dice loss constrains the shape of the nuclei segmentation whereas
the BCE loss regulates voxelwise binary prediction. It is observed that training with
more synthetic volumes can generalize our method to achieve better segmentation
accuracy. Finally, the postprocessing (PP) that eliminates small components helps
to improve segmentation performance.
To make this clear, segmentation results were color coded using 3D connected
component labeling and overlaid on the original volumes. The method from [43]
cannot distinguish between nuclei and non-nuclei structures including noise. This is
especially recognizable from segmentation results of Data-I in which multiple nuclei
and non-nuclei structures are colored with the same color. As can be observed from
Figure 4.15(e) and 4.15(f), segmentation masks are smaller than nuclei size and suf-
fered from location shifts. Conversely, our proposed method shown in Figure 4.15(g)
and 4.15(h) segments nuclei with the right shape at the correct locations.
71
4.4 MTU-Net 2
Fig. 4.16.: Block diagram of our method
Figure 4.16 shows a block diagram of our method. We denote I as a 3D image
volume of size X×Y ×Z. Note that Izp is a pth focal plane image, of size X×Y , along
the z-direction in a volume, where p ∈ {1, . . . , Z}. In addition, let I(qi:qf ,ri:rf ,pi:pf) be
a subvolume of I, whose x-coordinate is qi ≤ x ≤ qf , y-coordinate is ri ≤ y ≤ rf ,
z-coordinate is pi ≤ z ≤ pf , where qi, qf ∈ {1, . . . , X}, ri, rf ∈ {1, . . . , Y }, pi, pf ∈
{1, . . . , Z}, qi ≤ qf , ri ≤ rf , and pi ≤ pf . For example, Iseg(241:272,241:272,131:162) is a
subvolume of a segmented volume, Iseg, where the subvolume is cropped between
241st slice and 272nd slice in x-direction, between 241st slice and 272nd slice in y-
direction, and between 131st slice and 162nd slice in z-direction.
As shown in Figure 4.16, our proposed method is a two-stage method that consists
of synthetic volume generation and MTU-Net segmentation. We first train a spatially
constrained CycleGAN (SpCycleGAN) with synthetic binary volumes, I labelcyc, and
a subvolume of the original image volumes, Iorigcyc to obtain a generative model de-
noted as model G. To create MTU-Net training volumes, a new set of synthetic
binary volume, I label, and its corresponding heat map, Iheatlabel, are generated. A set
of synthetic microscopy volumes Isyn are generated using model G with I label. Note
that I label is a binary segmentation mask whereas Iheatlabel indicates the centroids of
nuclei. Here, I label and Iheatlabel serve as the segmentation labels and heat map labels
of Isyn. A multi-task network, MTU-Net, is trained with Isyn, I label and Iheatlabel to
2This is joint work with Ms. Shuo Han and Mr. Soonam Lee
72
obtain a model M . Also, Iorig is the original fluorescence microscopy volume. The
corresponding segmented volume, Iseg, and heat map, Iheat of Iorig can be obtained
using model M on Iorig. Finally, a nuclei separation method marker-controlled wa-
tershed [65] is used to separate overlapped nuclei in Iseg. This produces the final
segmentation Ifinal of Iorig.
4.4.1 3D Convolutional Neural Network
Fig. 4.17.: Architecture of our MTU-Net
Figure 4.17 shows architecture of our network. Our network is a multi-task U-
Net that outputs a 3D heatmap of the location of the nuclei and a probability map
of binary volumetric segmentation. The 3D heatmap is used to separate overlapped
nuclei in the binary volumetric segmentation and the detail is described in 4.4.2. After
separating overlapped nuclei, our method is able to produce instance segmentation of
the nuclei. The binary segmentation branch is the same as described in our previous
work [53]. Additionally, we extract spatial information of each layer of the decoder
and concatenate them together to form a branch that estimates the 3D heatmap of the
nuclei. A mean squared error is used to measure the difference between the predicted
3D heatmap and the label of the 3D heatmap while a combination of the Dice loss
73
and binary cross-entropy loss is used to measure the difference between the predicted
binary volumetric segmentation and the label of the segmentation. Therefore, the
total training loss of our network can be expressed as a linear combination of the
Dice loss, the binary cross entropy loss and the mean squared error such that
Lseg(T, S, C,D) = µ1LDice(T, S) + µ2LBCE(T, S)
+ µ3LMSE(C,D) (4.4)
where
LDice(T, S) =2(∑P
i=1 tisi)∑P
i=1 t2i +
∑P
i=1 s2i
LBCE(T, S) = −1
P
P∑
i=1
ti log(si) + (1− ti) log(1− si)
LMSE(C,D) =1
P
P∑
i=1
(ci − di)2,
respectively [101]. Note that T is the set of the groundtruth values of volumetric
binary segmentation while S is a prediction of binary volumetric segmentation. ti ∈ T
and si ∈ S are a groundtruth value at ith voxel location and a value of prediction at
ith voxel location. Also, C is the set of groundtruth values of the 3D heatmap and
ci ∈ C is a groundtruth value of the 3D heatmap at ith voxel location. Similarly, D
is a predicted 3D heatmap and di ∈ D is a value of the predicted 3D heatmap at ith
voxel location. Lastly, P is the number of entire voxels and µ1, µ2, and µ3 serve as
the weight coefficient between to loss terms in Equation (4.4). Our proposed network
produces an volumetric binary segmentation and 3D heatmap with the same size of
a input grayscale volume of size of 64 × 64 × 64. To train our model M , V pairs of
Isyn, I label, and Iheatlabel are used.
For the inference of our network, a moving inference window with size of 64×64×64
is slided through the entire volume starting from top to bottom and from left to right.
First, a symmetric padding was peformed to pad the original volume Iorig by 16 voxels
in x, y and z-direction. Since partial included nuclei structures may create artifacts
near the boundaries of the moving window, the stride of the sliding window was set
74
to 32 in x, y and z-directions and only the segmentation of size of 32× 32× 32 at the
center of the window is used to generate the corresponding subvolume of Iseg. More
details were described in our previous work [43].
4.4.2 Nuclei Separation
To achieve instance segmentation, our network uses the 3D heatmap with the
binary volumetric segmentation. An additional nuclei separation step is employed on
binary volumetric segmentation to separate overlapped nuclei. Here, we describe two
different approaches to separate overlapped nuclei such as quasi 3D watershed and
marker controlled watershed.
Quasi 3D Watershed
Our previous method [53] achieves promising segmentation results in terms of
voxel accuracy but fail to identify overlapped nuclei. We use watershed which is
a well-known and widely used technique to solve this problem. Since our goal is
to produce a volumetric segmentation, a 3D watershed is prefered. However 3D
watershed is computationally expensive when the input volume is large. Instead of
using a 3D, A 2D watershed [63] is used on the 3D segmentation in three different
direction sequentially to separate overlapped nuclei in a quasi 3D manner.
Marker Controlled Watershed
Watershed algoritm tends to oversegment objects into multiple small pieces. Here,
a marker controlled watershed is used to minimize oversegmentation problems in the
nuclei separation [65]. First, a non-maximum suppression is used on the heatmap
followed by a 3D connected components analysis to extract the centroids of the nuclei,
Ict. More specifically, the non-maximum supression uses a ball shape sliding window
with radius of R. R is selected according to the real size of the nuclei. Then, Ict is
75
dilated to a ball with radius of R3. In order to reduce over segmentation of watershed
technique, we only use marker controlled watershed on the components in Iseg that
contain no less than two centroids in Ict. A marker map Imarkseg can be generated
by finding the centroids objects in Iseg that contain no less than two centroids in
Ict. Imarkct are generated by finding the centroids that overlapped with Imarkseg. We
use [65] to separate overlapped nuclei from Imarkseg according to marker map Imarkct.
The final segmentation Ifinal is obtained by adding the output of marker controlled
watershed. The sample results of different stages of our proposed method are shown
in Figure 4.18.
4.4.3 Experimental Results
We tested our proposed method on four different rat kidney data sets and one rat
cardiomyocytes data set. Data-I contains grayscale images with size X = 512 × Y
= 512 × Z = 512. Data-II contains grayscale images with size X = 512 × Y = 512
× Z = 415. Data-III contains grayscale images with size X = 512 × Y = 512 × Z
= 32. Data-IV contains grayscale images with size X = 512 × Y = 512 × Z = 300.
Data-V contains grayscale images with size X = 512 × Y = 512 × Z = 157. Note
that Data-I, II, III, and V are obtained from rat kidney whereas Data-IV is obtained
from rat cardiomyocytes.
Synthetic Generation
Our SpCycleGAN is implemented in PyTorch using the Adam optimizer with
constant learning of 0.0002 in the first 100 epochs and gradually decayed learning
rate from 0.0002 to 0 in the second 100 epochs. We use Resnet 9 blocks for both
network G, F and H. For each of the data, the sizes of I labelcyc and Iorigcyc were both
128 × 128 × 128. Here, Iorigcyc is a subvolume of Iorig. A 64 × 64 2D random
cropping was used to augment training images before training. For Data-I, Data-II,
and Data-III, SpCycleGAN generative models GData−I ,GData−II and GData−III were
76
(a) (b) (c)
(d) (e) (f)
(g)
Fig. 4.18.: Sample results of different stages of our proposed method. (a) Iseg (b)
Iheat (c) dilated Ict (d) Imarkseg (e) Imarkct (f) Ifinal (g) color result
trained individually using λ1 = λ2 = 10. For Data-IV, SpCycleGAN generative model
GData−IV was trained using λ1 = λ2 = 50 to penalize more on spatial constrains since
Data-IV contains more directional pattern. For each data, 80 sets of Isyn, I label and
77
Iheatlabel were generated using its own generative model. The size of each volume of
Isyn, I label and Iheatlabel is 64 × 64 × 64.
MTU-Net Segmentation
Table 4.5.: True positive, False positive, False negative, Precision, Recall and F1
Scores for known methods and our method on Data-I
Data-I
Method NTP NFP NFN P R F1
Otsu [55] + Quasi 3D watershed 151 22 132 87.28% 53.36% 66.23%
CellProfiler [109] 59 14 223 80.82% 20.92% 33.24%
Squassh [69,70] 109 12 174 90.08% 38.52% 53.96%
Method [53] 228 22 50 91.20% 82.01% 86.36%
Method [53] + Quasi 3D watershed 261 31 13 89.38% 95.26% 92.23%
MTU-Net (Proposed) 260 20 17 92.86% 93.86% 93.36%
Table 4.6.: Voxel Accuracy, Type-I and Type-II for known methods and our method
on Data-I
Data-I
Method Voxel Accuracy Type-I Type-II
Otsu [55] + Quasi 3D watershed 81.89% 17.88% 0.23%
CellProfiler [109] 78.02% 21.67% 0.31%
Squassh [69,70] 86.48% 11.87% 1.65%
Method [53] 95.68% 1.33% 2.99%
Method [53] + Quasi 3D watershed 95.73% 1.49% 2.78%
MTU-Net (Proposed) 95.68% 1.86% 2.46%
78
Table 4.7.: True positive, False positive, False negative, Precision, Recall and F1
Scores for known methods and our method on Data-III
Data-III
Method NTP NFP NFN P R F1
Otsu [55] + Quasi 3D watershed 223 47 69 82.59% 76.37% 79.36%
CellProfiler [109] 218 37 78 85.49% 73.65% 79.13%
Squassh [69,70] 243 22 79 91.70% 75.47% 82.79%
Method [53] 321 92 3 92.18% 83.38% 87.56%
Method [53] + Quasi 3D watershed 317 47 5 87.09% 98.45% 92.42%
MTU-Net (Proposed) 303 30 18 91.27% 94.41% 92.82%
Table 4.8.: Voxel Accuracy, Type-I, and Type-II for known methods and our method
on Data-III
Data-III
Method Voxel Accuracy Type-I Type-II
Otsu [55] + Quasi 3D watershed 93.95% 2.53% 3.51%
CellProfiler [109] 93.95% 2.66% 3.39%
Squassh [69,70] 94.84% 4.46% 0.70%
Method [53] 92.19% 1.93% 5.88%
Method [53] + Quasi 3D watershed 92.29% 1.79% 5.92%
MTU-Net (Proposed) 92.69% 1.41% 5.90%
79
(a) (b)
(c) (d)
(e)
Fig. 4.19.: Sample results of Data-I (a) Original microscopy images (b) Segmentations
of Squassh (c) Segmentations of method [53] (d) Segmentations of method [53] +
Quasi 3D watershed (e) Segmentations of MTU-Net
80
(a) (b)
(c) (d)
(e)
Fig. 4.20.: Sample results of Data-II (a) Original microscopy images (b) Segmenta-
tions of Squassh (c) Segmentations of method [53] (d) Segmentations of method [53]
+ Quasi 3D watershed (e) Segmentations of MTU-Net
81
(a) (b)
(c) (d)
(e)
Fig. 4.21.: Sample results of Data-III (a) Original microscopy images (b) Segmenta-
tions of Squassh (c) Segmentations of method [53] (d) Segmentations of method [53]
+ Quasi 3D watershed (e) Segmentations of MTU-Net
82
(a) (b)
(c) (d)
(e)
Fig. 4.22.: Sample results of Data-IV (a) Original microscopy images (b) Segmenta-
tions of Squassh (c) Segmentations of method [53] (d) Segmentations of method [53]
+ Quasi 3D watershed (e) Segmentations of MTU-Net
83
(a) (b)
(c) (d)
(e)
Fig. 4.23.: Sample results of Data-V (a) Original microscopy images (b) Segmenta-
tions of Squassh (c) Segmentations of method [53] (d) Segmentations of method [53]
+ Quasi 3D watershed (e) Segmentations of MTU-Net
84
(a) (b) (c)
(d) (e) (f)
(g) (h)
Fig. 4.24.: 3D visualization of different methods of subvolume of Data-I. (a) Original
volume (b) Groundtruth volume (c) Otsu + Quasi 3D watershed (d) CellProfiler (e)
Squassh (f) Method [110] (g) Method [110] + Quasi 3D watershed (h) MTU-Net
(Proposed)
Our MTU-Net is also implemented in PyTorch using Adam optimizer with learn-
ing rate of 0.001. For each of the Data-I, Data-II, Data-III and Data-IV, MTU-Net
models MData−I ,MData−II ,MData−III and MData−IV were trained individually with
85
80 sets of Isyn, I label, Iheatmap. The weights of MTU-Net loss function were used as
µ1 = 1 and µ2 = µ3 = 10. We tested Data-I, Data-II, Data-III and Data-IV with
model MData−I , MData−II , MData−III , and MData−IV respectively. Additionally, we
tested Data-V with the model MData−II since they shares similar characteristic of
nuclei. For nuclei separation step, we RData−I = 5, RData−II = 7, RData−III = 13,
RData−IV = 5, and RData−V = 6. For the convenience of visualization, we used 3D
connected components to identify individual nuclei and assigned them with different
color. Small 3D connected components that less than 20 voxels are removed at the
end.
We evaluate our segmentation on Data-I and Data-III. Two groundtruth volumes,
Igt,Data−I and Igt,Data−III , are manually anotated using ITK-SNAP [111]. Igt,Data−I
is 128 × 128 × 64 and corresponds to Iorig,Data−I
(193:320,193,320,31:94). Igt,Data−III is 512 × 512
× 32 and corresponds to the entire Iorig,Data−III . To evaluate the segmentation,
both voxel-based evaluation and object-based evaluation are used. For voxel-based
evaluation, Type-I and Type-II error metric was used. voxel accuracy = nTP+nTN
ntotal
,
Type-I error = nFP
ntotal
, Type-II error = nFN
ntotal
, where nTP, nTN, nFP, nFN, ntotal are
defined to be the number of true-positives (voxels segmented as nuclei correctly), true-
negatives (voxels segmented as background correctly), false-positives (voxels falsely
segmented as nuclei), false-negatives (voxels falsely segmented as background), and
the total number of voxels in a volume, respectively. For object-based evaluation, F1
score (F1), Precision (P) and Recall (R) [112,113] were obtained as:
P =NTP
NTP +NFP
, R =NTP
NTP +NFN
, F1 =2PR
P +R, (4.5)
where NTP is the number of true-positive, NFP is the number of false-positive, NTN
is the number of true-negative,and NFN is the number of false-negative. Here, a true-
postive is defined as the segmentation of a nucleus overlap more than 50% with corre-
sponding nucleus in the groundtruth. Otherwise, it a false-positive. A true-negative
is defined as the segmentation of a nucleus overlap less than 50% with corresponding
nucleus in the groundtruth or no corresponding nucleus presents in the groundtruth.
86
Our method was compared to 6 different methods including Otsu [55] + quasi
3D watershed, CellProfiler [109], Squassh [69,70], our previous work [53], and [53] +
quasi 3D watershed. Otsu cannot separate overlapped nuclei so the quasi 3D water-
shed was used on the results of Otsu. CellProfiler is a cell image analysis tool that
are commonly used in biological researches. We used CellProfiler for nuclei segmen-
tation that includes contrast enhancement, median filtering, Otsu thresholding, hole
removal, and watershed. For Squassh, the default parameters were used for testing.
Our previous work [53] is trained with the same synthetic data that MTU-Net used
for training.
The best four of the compared methods of five different data sets are shown in
Figure 4.19, 4.20, 4.21, 4.22 and 4.23. As shown in the Figure 4.19, 4.20, 4.21, 4.22
and 4.23, Squassh is able to segment nuclei as individual objects if the original volume
is sparse and clear but failed otherwise, especially when non-nuclei structure and noise
are presented. Our previous work [53] is able to segment nuclei accurately but not
able to separate overlapped nuclei. With a quasi 3D watershed used after [53], the
overlapped nuclei are observed to be identified as indivdual nuclei for the most of the
situations. However, if multiple nuclei are overlapped with each others, this method
may fail to separate the overlapped objects accurately. Our proposed method uses
a heatmap of centroids to locate the nuclei in a overlapped objects and uses mark
controlled watershed to separate them accurately. We also visualized the results
of each methods in 3D using ImageJ Volume Viewer [114]. A comparison of 3D
visualization were also shown in 4.24.
In Table 4.5 and 4.7, it was shown that our proposed method reduces the number
of false-positive in the object-based evaluation of both of the data sets. It means
our proposed method is able to separate nuclei more accurately compared to others.
However, due to the limitaion of non-maximum suppression, the increasing number
of false-negative is also observed. Our proposed method also achieved high voxel
accuracy since our proposed method can segment the shape of the nuclei accurately.
87
5. DISTRIBUTED AND NETWORKED ANALYSIS OF
VOLUMETRIC IMAGE DATA (DINAVID)
5.1 System Overview1
Fig. 5.1.: System diagram of DINAVID
We designed and developed a web-based microscopy image analysis system. We
call this system the Distributed and Networked Analysis of Volumetric Image Data
(DINAVID). This system is designed for fast and accurate analysis of large scale
microscopy volumes. As shown in Figure 5.1, our system consists of web-based user
interface and computing clusters that contains high performance GPUs. User will be
able to upload and download data using web-based user interface. Also, built in image
previewer and built in 3D volume visualization tools are integrated for visualizing data
before and after processes.
In Figure 5.2, user will need to login to our system using our issued credential.
Currently, we only issue credential upon request. As shown in Figure 5.3, users will
see the tutorial of our system and our project information once they login. In the
”Tool” tap, a upload function will allow user to uploaded they data into system using
the blue ”Upload Images” button. As shown in Figure 5.4, at the right top of the
page, user can delete all the images using the red ”Delete Uploaded Images” button.
Currently, our system only support 2D image slices and will support 3D image in the
future.
1This is joint work with Ms. Shuo Han, Mr. Soonam Lee and Dr. David J. Ho
88
Fig. 5.2.: Login page of DINAVID
Fig. 5.3.: Home page of DINAVID
89
Fig. 5.4.: Data upload page of DINAVID
Fig. 5.5.: Segmentation tool page of DINAVID
90
Fig. 5.6.: Subvolume selecting functionality
91
A deep learning based nuclei segmentation method, deep 3D+ [53], is implemented
in our system. In Figure 5.5, five different segmentation models that trained with
different microscopy images are provided. User can process their uploaded data with
these models. Also, a image preview window shows the uploaded image. As shown in
Figure 5.6, user can also process on a subvolume of the data by specifying a region of
interest in the preview window. By pushing the blue ”Process” button, our system
will process the data at our computing clusters. Once our system finished the process,
the web page will automatically redirected to the result download page. As shown
in Figure 5.7, user can download the result or visualize the result immediately in our
built in 3D visualization tool. In Figure 5.8, our visualization tool can also provide
subvolume visualization and 2D slices visualization.
Fig. 5.7.: Download page of DINAVID
92
Fig. 5.8.: 3D visualization of DINAVID
93
6. SUMMARY AND FUTURE WORK
6.1 Summary
In this thesis, we focused on the image analysis on microscopy images including
image registration, image synthesis and image segmentation. A 4D image registration
that uses combination of rigid and non-rigid registration was described. A Quasi 2D
nuclei segmentation was developed convolutional neural networks. We investigated in
nuclei image synthesis to solve lack of training data problem. A nuclei image synthesis
technique spatially constrained cycle-consistent adversarial network was proposed to
generate nuclei image. A 3D segmentation using a combination of binary cross entropy
loss and dice loss was presented later. Finally, a multi-task U-Net was described to
segmentation nuclei as individual instance. The main contribution of this thesis are
as follows:
• 4D Image Registration
We extended previous work of 3D image registration method to a 4D registration
method. The 4D registration method enables fixing motion artifacts in depth
of the live tissue and motion artifacts in time dimension. Three dimensional
spherical histograms of motion vectors were used to validate our method.
• Image Synthesis
We proposed a spatial constrained cycle-consistent adversarial network for nu-
clei image synthesis. This method generates realistic nuclei images with cor-
responding segmentation labels. This method requires no segmentation labels
for training. This method enabled the training of machine learning based tech-
niques for nuclei segmentation.
94
• 2D Nuclei Segmentation
We described a 2D CNN segmentation method to segmentation only nuclei
from the 3D image volumes that also contains different non-nuclei biological
structures. We are able to accurately segment nuclei from 3D image volumes
by using our system. Watershed based nuclei counting was able to separate
overlapped nuclei and count them.
• 3D Nuclei Segmentation
We described a 3D CNN segmentation method to segmentation 3D nuclei from
the 3D image volumes. A combination of dice loss and binary cross entropy
loss were used to train a modifed U-Net. With our SpCycleGAN nuclei data
generation, we were able to training our 3D U-Net in a large scale. A Quasi-
3D watershed was applied on the segmentation to separate overlapping nuclei.
This method achieves promising results in terms of object-based evaluation and
voxel-based evaluation
• 3D Nuclei Segmentation
We also proposed a instance segmentation method, multi-task U-Net. This
method generates segmentation mask with corresponding nuclei location map.
Using marker-controlled watershed, our method was able to separate overlap-
ping nuclei and minimize over-segmentation of watershed-based technique.
• Distributed and Networked Analysis of Volumetric Image Data (DINAVID)
We create a Distributed and Networked Analysis of Volumetric Image Data
(DINAVID) system. DINAVID is web-based microscopy image analysis system.
This system enables biologists to do fast and accurate analysis on microscopy
images. After analysis, a 3D visualization of the results can also be viewed in
our system.
95
6.2 Future Work
• Image Registration
Currently, our registration method is limited to 4D rigid registration due to the
need of preserving the original motion of cells in our dataset. In many other
microscopy image registration problems, a 4D non-rigid registration method
would be used to generate the best results. In the future, we plan to generalize
our method to a 4D non-rigid registration technique that can cancel the non-
rigid motion artifacts in temporal 3D images.
• Image Synthesis
Our image SpCycleGAN is able to generate nuclei images without using any
manually labeled data. The generated 2D images can be stacked to form 3D
volume. Although the characteristic of nuclei is realistic in 2D, shape of the
structures are not perfectly defined in 3D. In the future, we would like to ex-
panding our current method to a 3D technique. Also, our current method can
be used on other applications such as image de-noising and image restoration.
• Nuclei Segmentation
Although our nuclei segmentation achieves high accuracy in terms of object-
based and voxel-based evaluation, the generalization of our model remains a
problem. The characteristic of biological structures varied from different organs
and different data acquisitions. A generalized model is hard to obtain due to
lack of labeled data. Since our SpCycleGAN can be used to cheaply generate
training data for our segmentation training, our current approach is to generate
segmentation models for different groups of microscopy images. In the future,
we would like to explore more on how to generalize our techniques.
• Distributed and Networked Analysis of Volumetric Image Data (DINAVID)
We will continue to develop our web-based image analysis system with more
features based on the feedback of biologist.
96
6.3 Publication Resulting From This Work
Journal Papers
1. C. Fu, S. Han, S. Lee, D. J. Ho, P. Salama, K. W. Dunn and E. J. Delp, ”Three
Dimensional Nuclei Synthesis and Instance Segmentation”, To be Submitted,
IEEE Transactions on Medical Imaging.
2. D. J. Ho, C. Fu, D. M. Montserrat, P. Salama and K. W. Dunn and E. J. Delp,
”Sphere Estimation Network: Three Dimensional Nuclei Detection of Fluores-
cence Microscopy Images”, To be Submitted, IEEE Transactions on Medical
Imaging.
Conference Papers
1. C. Fu, N. Gadgil, K. K Tahboub, P. Salama, K. W. Dunn and E. J. Delp,
”Four Dimensional Image Registration For Intravital Microscopy”, Proceedings
of the Computer Vision for Microscopy Image Analysis workshop at Computer
Vision and Pattern Recognition, July 2016, Las Vegas, NV.
2. C. Fu, D. J. Ho, S. Han, P. Salama, K. W. Dunn, E. J. Delp, ”Nuclei segmen-
tation of fluorescence microscopy images using convolutional neural networks”,
Proceedings of the IEEE International Symposium on Biomedical Imaging, pp.
704-708, April 2017, Melbourne, Australia. DOI: 10.1109/ISBI.2017.7950617
3. C. Fu, S. Han, D. J. Ho, P. Salama, K. W. Dunn and E. J. Delp, ”Three dimen-
sional fluorescence microscopy image synthesis and segmentation”, Proceedings
of the Computer Vision for Microscopy Image Analysis workshop at Computer
Vision and Pattern Recognition, June 2018, Salt Lake City, UT.
4. D. J. Ho, C. Fu, P. Salama, K. W. Dunn, and E. J. Delp, ”Nuclei Segmen-
tation of Fluorescence Microscopy Images Using Three Dimensional Convolu-
tional Neural Networks,” Proceedings of the Computer Vision for Microscopy
97
Image Analysis (CVMI) workshop at Computer Vision and Pattern Recognition
(CVPR), July 2017, Honolulu, HI. DOI: 10.1109/CVPRW.2017.116
5. D. J. Ho, C. Fu, P. Salama, K. W. Dunn, and E. J. Delp, ”Nuclei Detection
and Segmentation of Fluorescence Microscopy Images Using Three Dimensional
Convolutional Neural Networks”, Proceedings of the IEEE International Sym-
posium on Biomedical Imaging, pp. 418-422, April 2018, Washington, DC. DOI:
10.1109/ISBI.2018.8363606
6. S. Lee, C. Fu, P. Salama, K. W. Dunn, and E. J. Delp, ”Tubule Segmentation
of Fluorescence Microscopy Images Based on Convolutional Neural Networks
with Inhomogeneity Correction,” Proceedings of the IS&T Conference on Com-
putational Imaging XVI, February 2018, Burlingame, CA.
7. D. J. Ho, S. Han, C. Fu, P. Salama, K. W. Dunn, and E. J. Delp, ”Center-
Extraction-Based Three Dimensional Nuclei Instance Segmentation of Fluores-
cence Microscopy Images,” Submitted To, Proceedings of the IEEE International
Symposium on Biomedical Imaging, April 2019, Venice, Italy.
8. S. Han, S. Lee, C. Fu, P. Salama, K. W. Dunn, and E. J. Delp, ”Nuclei Count-
ing in Microscopy Images with Three Dimensional Generative Adversarial Net-
works, To, Appear, Proceedings of the SPIE Conference on Medical Imaging,
February 2019, San Diego, California.
REFERENCES
98
REFERENCES
[1] D. B. Murphy and M. W. Davidson, Fundamentals of Light Microscopy andElectronic Imaging. Wiley-Blackwell, 2012.
[2] J. W. Lichtman and J.-A. Conchello, “Fluorescence microscopy,” Nature Meth-ods, vol. 2, no. 12, pp. 910–919, December 2005.
[3] M. Chalfie, Y. Tu, G. Euskirchen, W. Ward, and D. C. Prasher, “Green fluo-rescent protein as a marker for gene expression,” Science, vol. 263, no. 5148,pp. 802–805, February 1994.
[4] T. Stearns, “Green flourescent protein: The green revolution,” Current Biology,vol. 5, no. 3, pp. 262–264, March 1995.
[5] O. Shimomura, “The discovery of aequorin and green fluorescent protein,” Jour-nal of Microscopy, vol. 217, no. 1, pp. 3–15, January 2005.
[6] M. Minsky, “Memoir on inventing the confocal scanning microscope,” Scanning,vol. 10, no. 4, pp. 128–138, 1988.
[7] E. Wang, C. M. Babbey, and K. W. Dunn, “Performance comparison betweenthe high-speed Yokogawa spinning disc confocal system and single-point scan-ning confocal systems,” Journal of Microscopy, vol. 218, no. 2, pp. 148–159,May 2005.
[8] W. Denk, J. H. Strickler, and W. W. Webb, “Two-photon laser scanning fluo-rescence microscopy,” Science, vol. 248, no. 4951, pp. 73–76, April 1990.
[9] D. W. Piston, “Imaging living cells and tissues by two-photon excitation mi-croscopy,” Trends in Cell Biology, vol. 9, no. 2, pp. 66–69, February 1999.
[10] K. W. Dunn, R. M. Sandoval, K. J. Kelly, P. C. Dagher, G. A. Tanner, S. J.Atkinson, R. L. Bacallao, and B. A. Molitoris, “Functional studies of the kidneyof living animals using multicolor two-photon microscopy,” American Journalof Physiology-Cell Physiology, vol. 283, no. 3, pp. C905–C916, September 2002.
[11] F. Helmchen and W. Denk, “Deep tissue two-photon microscopy,” Nature Meth-ods, vol. 2, no. 12, pp. 932–940, December 2005.
[12] K. Svoboda and R. Yasuda, “Principles of two-photon excitation microscopyand its applications to neuroscience,” Neuron, vol. 50, no. 6, pp. 823–839, 2006.
[13] E. H. Hoover and J. A. Squier, “Advances in multiphoton microscopy technol-ogy,” Nature Photonics, vol. 7, pp. 93–101, February 2013.
[14] J. Peti-Peterdi, I. Toma, A. Sipos, and S. L. Vargas, “Multiphoton imaging ofrenal regulatory mechanisms,” Physiology, vol. 24, no. 2, pp. 88–96, April 2009.
99
[15] C. Sumen, T. R. Mempel, I. B. Mazo, and U. H. von Andrian, “Intravitalmicroscopy: visualizing immunity in context,” Immunity, vol. 21, no. 3, pp.315–329, September 2004.
[16] M. J. Hickey and P. Kubes, “Intravascular immunity: the host–pathogen en-counter in blood vessels,” Nature Reviews Immunology, vol. 9, no. 5, pp. 364–375, May 2009.
[17] L. Qu, F. Long, and H. Peng, “3-D registration of biological images and mod-els: Registration of microscopic images and its uses in segmentation and anno-tation,” IEEE Signal Processing Magazine, vol. 32, no. 1, pp. 70–77, January2015.
[18] B. Zitova and J. Flusser, “Image registration methods: A survey,” Image andVision Computing, vol. 21, no. 11, pp. 977–1000, October 2003.
[19] K. S. Arun and K. S. Sarath, “An automatic feature based registration algo-rithm for medical images,” International Conference on Advances in RecentTechnologies in Communication and Computing, pp. 174–177, October 2010.
[20] P. Matula, M. Kozubek, and V. Dvorak, “Fast point-based 3-D alignment of livecells,” IEEE Transactions on Image Processing, vol. 15, no. 8, pp. 2388–2396,August 2006.
[21] S. Chang, F. Cheng, W. Hsu, and G. Wu, “Fast algorithm for point patternmatching: Invariant to translations rotations and scale changes,” Pattern Recog-nition, vol. 30, no. 2, pp. 311–320, February 1997.
[22] K. Mkrtchyan, A. Chakraborty, and A. Roy-Chowdhury, “Optimal landmarkselection for registration of 4D confocal image stacks in arabidopsis,” To appear,IEEE/ACM Transactions on Computational Biology and Bioinformatics, 2016.
[23] A. Myronenko and X. Song, “Intensity-based image registration by minimizingresidual complexity,” IEEE Transactions on Medical Imaging, vol. 29, no. 11,pp. 1882–1891, June 2010.
[24] G. P. Penney, J. Weese, J. A. Little, P. Desmedt, D. L. G. Hill, and D. J.hawkes, “A comparison of similarity measures for use in 2-D-3-D medical imageregistration,” IEEE Transactions on Medical Imaging, vol. 17, no. 4, pp. 586–595, August 1998.
[25] F. Maes, A. Collignon, D. Vandermeulen, G. Marchal, and P. Suetens, “Mul-timodality image registration by maximization of mutual information,” IEEETransactions on Medical Imaging, vol. 16, no. 2, pp. 187–198, April 1997.
[26] K. S. Lorenz, P. Salama, K. W. Dunn, and E. J. Delp, “Digital correction ofmotion artefacts in microscopy image sequences collected from living animalsusing rigid and nonrigid registration,” Journal of Microscopy, vol. 245, no. 2,pp. 148–160, February 2012.
[27] K. S. Lorenz, “Registration and segmentation based analysis of microscopyimages,” Ph.D. dissertation, Purdue University, West Lafayette, IN, August2012.
100
[28] P. Thevenaz, U. E. Ruttimann, and M. Unser, “A pyramid approach to subpixelregistration based on intensity,” IEEE Transactions on Image Processing, vol. 7,no. 1, pp. 27–41, January 1998.
[29] M. Jenkinson, P. Bannister, M. Brady, and S. Smith, “Improved optimizationfor the robust and accurate linear registration and motion correction of brainimages,” Neuroimage, vol. 17, no. 2, pp. 825–841, 2002.
[30] C. A. Wilson and J. A. Theriot, “A correlation-based approach to calculaterotation and translation of moving cells,” IEEE Transactions on Image Pro-cessing, vol. 15, no. 7, pp. 1939–1951, July 2006.
[31] S. Yang, D. Kohler, K. Teller, T. Cremer, P. L. Baccon, E. Heard, R. Eils, andK. Rohr, “Nonrigid registration of 3-D multichannel microscopy images of cellnuclei,” IEEE Transactions on Image Processing, vol. 17, no. 4, pp. 493–499,April 2008.
[32] I. H. Kim, Y. C. M. Chen, D. L. Spector, R. Eils, and K. Rohr, “Nonrigid regis-tration of 2-D and 3-D dynamic cell nuclei images for improved classification ofsubcellular particle motion,” IEEE Transactions on Image Processing, vol. 20,no. 4, pp. 1011–1022, September 2010.
[33] T. Du and M. Wasser, “3D image stack reconstruction in live cell microscopyof Drosophila muscles and its validation,” Cytometry Part A, vol. 75, no. 4, pp.329–343, April 2009.
[34] R. G. Keys, “Cubic convolution interpolation for digital image processing,”IEEE Transactions on Acoustics, Speech and Signal Processing, vol. 29, no. 6,pp. 1153–1160, December 1981.
[35] K. S. Lorenz, P. Salama, K. W. Dunn, and E. J. Delp, “Digital correction ofmotion artifacts in microscopy image sequences collected from living animalsusing rigid and nonrigid registration,” Journal of Microscopy, vol. 245, no. 2,pp. 148–160, February 2012.
[36] K. W. Dunn, K. S. Lorenz, P. Salama, and E. J. Delp, “IMART softwarefor correction of motion artifacts in images collected in intravital microscopy,”IntraVital, vol. 3, no. 1, pp. e28 210:1–10, February 2014.
[37] D. C. Liu and J. Nocedal, “On the limited memory BFGS method for largescale optimization,” Mathematical Programming, vol. 45, no. 1, pp. 503–528,August 1989.
[38] C. G. Broyden, “The convergence of a class of double-rank minimization algo-rithms,” IMA Journal of Applied Mathematics, vol. 6, no. 1, pp. 76–90, 1970.
[39] R. Fletcher, “A new approach to variable metric algorithms,” The ComputerJournal, vol. 13, no. 3, pp. 317–322, 1970.
[40] D. Goldfarb, “A family of variable-metric methods derived by variationalmeans,” Mathematics of Computation, vol. 24, no. 109, pp. 23–26, January1970.
[41] D. F. Shanno, “Conditioning of quasi-Newton methods for function minimiza-tion,” Mathematics of Computation, vol. 24, no. 111, pp. 647–656, July 1970.
101
[42] B. Schmid, J. Schindelin, A. Cardona, M. Longair, and M. Heisenberg, “A high-level 3D visualization API for Java and ImageJ,” BMC Bioinformatics, vol. 11,no. 274, May 2010.
[43] D. J. Ho, C. Fu, P. Salama, K. W. Dunn, and E. J. Delp, “Nuclei segmentationof fluorescence microscopy images using three dimensional convolutional neuralnetworks,” Proceedings of the Computer Vision for Microscopy Image Analysisworkshop at Computer Vision and Pattern Recognition, pp. 834–842, July 2017,Honolulu, HI.
[44] S. Rajaram, B. Pavie, N. E. F. Hac, S. J. Altschuler, and L. F. Wu, “SimuCell: Aflexible framework for creating synthetic microscopy images,” Nature methods,vol. 9, no. 7, pp. 634–635, June 2012.
[45] D. Svoboda and V. Ulman, “Generation of synthetic image datasets for time-lapse fluorescence microscopy,” International Conference Image Analysis andRecognition, pp. 473–482, June 2012.
[46] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair,A. Courville, and Y. Bengio, “Generative adversarial nets,” Proceedings of theAdvances in Neural Information Processing Systems, pp. 2672–2680, December2014, Montreal, Canada.
[47] A. Radford, L. Metz, and S. Chintala, “Unsupervised representation learn-ing with deep convolutional generative adversarial networks,” arXiv preprintarXiv:1511.06434v2, January 2016.
[48] M. Arjovsky, S. Chintala, and L. Bottou, “Wasserstein gan,” arXiv preprintarXiv:1701.07875v3, December 2017.
[49] A. Osokin, A. Chessel, R. E. C. Salas, and F. Vaggi, “Gans for biological im-age synthesis,” Proceedings of the IEEE International Conference on ComputerVision, pp. 2252–2261, October 2017, venice, Italy.
[50] P. Isola, J. Y. Zhu, T. Zhou, and A. A. Efros, “Image-to-image translationwith conditional adversarial networks,” Proceedings of the IEEE Conference onComputer Vision and Pattern Recognition, pp. 5967–5976, July 2017, Honolulu,HI.
[51] J. Y. Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired image-to-image translation using cycle-consistent adversarial networks,” arXiv preprintarXiv:1703.10593, pp. 1–16, March 2017.
[52] Y. Huo, Z. Xu, S. Bao, A. Assad, R. G. Abramson, and B. A. Landman,“Adversarial synthesis learning enables segmentation without target modalityground truth,” arXiv preprint arXiv:1712.07695, pp. 1–4, December 2017.
[53] C. Fu, S. Lee, D. J. Ho, S. Han, P. Salama, K. W. Dunn, and E. J. Delp, “Threedimensional fluorescence microscopy image synthesis and segmentation,” arXivpreprint arXiv:1801.07198, pp. 1–9, April 2018.
[54] C. Vonesch, F. Aguet, J. Vonesch, and M. Unser, “The colored revolution ofbioimaging,” IEEE Signal Processing Magazine, vol. 23, no. 3, pp. 20–31, May2006.
102
[55] N. Otsu, “A threshold selection method from gray-level histograms,” IEEEtransactions on systems, man, and cybernetics, vol. 9, no. 1, pp. 62–66, January1979.
[56] W. Niblack, An introduction to digital image processing. Prentice-Hall, 1986,vol. 34.
[57] J. Sauvola and M. Pietikainen, “Adaptive document image binarization,” Pat-tern recognition, vol. 33, no. 2, pp. 225–236, 2000.
[58] M. Kass, A. Witkin, and D. Terzopoulos, “Snakes: Active contour models,”International Journal of Computer Vision, vol. 1, no. 4, pp. 321–331, January1988.
[59] B. Li and S. T. Acton, “Automatic active model initialization via Poisson inversegradient,” IEEE Transactions on Image Processing, vol. 17, no. 8, pp. 1406–1420, August 2008.
[60] B. Li and S. T. Acton, “Active contour external force using vector field convolu-tion for image segmentation,” IEEE Transactions on Image Processing, vol. 16,no. 8, pp. 2096–2106, August 2007.
[61] T. F. Chan and L. A. Vese, “Active contours without edges,” IEEE Transactionson Image Processing, vol. 10, no. 2, pp. 266–277, February 2001.
[62] K. S. Lorenz, P. Salama, K. W. Dunn, and E. J. Delp, “Three dimensionalsegmentation of fluorescence microscopy images using active surfaces,” Proceed-ings of the IEEE International Conference on Image Processing, pp. 1153–1157,September 2013, Melbourne, Australia.
[63] L. Vincent and P. Soille, “Watersheds in digital spaces: an efficient algorithmbased on immersion simulations,” IEEE Transactions on Pattern Analysis andMachine Intelligence, vol. 13, no. 6, pp. 583–598, June 1991.
[64] R. C. Gonzalez and R. E. Woods, Digital Image Processing, 2nd ed., Boston,MA, 2001.
[65] X. Yang, H. Li, and X. Zhou, “Nuclei segmentation using marker-controlled wa-tershed, tracking using mean-shift, and Kalman filter in time-lapse microscopy,”IEEE Transactions on Circuits and Systems I: Regular Papers, vol. 53, no. 11,pp. 2405–2414, November 2006.
[66] A. Dufour, V. Shinin, S. Tajbakhsh, N. Guillen-Aghion, J. C. Olivo-Marin,and C. Zimmer, “Segmenting and tracking fluorescent cells in dynamic 3-Dmicroscopy with coupled active surfaces,” IEEE Transactions on Image Pro-cessing, vol. 14, no. 9, pp. 1396–1410, September 2005.
[67] O. Dzyubachyk, W. A. van Cappellen, J. Essers, W. J. Niessen, and E. Mei-jering, “Advanced level-set-based cell tracking in time-lapse fluorescence mi-croscopy,” IEEE Transactions on Image Processing, vol. 29, no. 3, pp. 852–867,March 2010.
[68] J. Cardinale, G. Paul, and I. F. Sbalzarini, “Discrete region competition for un-known numbers of connected regions,” IEEE Transactions on Image Processing,vol. 21, no. 8, pp. 3531–3545, August 2012.
103
[69] G. Paul, J. Cardinale, and I. F. Sbalzarini, “Coupling image restoration andsegmentation: A generalized linear model/Bregman perspective,” InternationalJournal of Computer Vision, vol. 104, no. 1, pp. 69–93, March 2013.
[70] A. Rizk, G. Paul, P. Incardona, M. Bugarski, M. Mansouri, A. Niemann,U. Ziegler, P. Berger, and I. F. Sbalzarini, “Segmentation and quantificationof subcellular structures in fluorescence microscopy images using Squassh,” Na-ture Protocols, vol. 9, no. 3, pp. 586–596, February 2014.
[71] G. Srinivasa, M. C. Fickus, Y. Guo, A. D. Linstedt, and J. Kovacevic, “Activemask segmentation of fluorescence microscope images,” IEEE Transactions onImage Processing, vol. 18, no. 8, pp. 1817–1829, August 2009.
[72] S. C. Zhu and A. Yuille, “Region competition: Unifying snakes, region growing,and Bayes/MDL for multiband image segmentation,” IEEE Transactions onPattern Analysis and Machine Intelligence, vol. 18, no. 9, pp. 884–900, Septem-ber 1996.
[73] Y. Shi and W. C. Karl, “A real-time algorithm for the approximation of level-set-based curve evolution,” IEEE Transactions on Image Processing, vol. 17,no. 5, pp. 645–656, May 2008.
[74] J. O. Cardinale, “Unsupervised segmentation and shape posterior estimationunder Bayesian image models,” Ph.D. dissertation, Swiss Federal Institute ofTechnology in Zurich, Zurich, Switzerland, January 2013.
[75] S. Mallat and S.Zhong, “Characterization of signals from multiscale edges,”IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 14, no. 7,pp. 710–732, July 1992.
[76] L. Zhang and P. Bao, “Edge detection by scale multiplication in wavelet do-main,” Pattern Recognition Letters, vol. 23, no. 14, pp. 1771–1784, December2002.
[77] Z. Zhang, S. Ma, H. Liu, and Y. Gong, “An edge detection approach based ondirectional wavelet transform,” Computers and Mathematics with Applications,vol. 57, no. 8, pp. 1265–1271, April 2009.
[78] S. Arslan, T. Ersahin, R. Cetin-Atalay, and C. Gunduz-Demir, “Attributedrelational graphs for cell nucleus segmentation in fluorescence microscopy im-ages,” IEEE Transactions on Medical Imaging, vol. 32, no. 6, pp. 1121–1131,June 2013.
[79] N. Gadgil, P. Salama, K. Dunn, and E. Delp, “Nuclei segmentation of fluo-rescence microscopy images based on midpoint analysis and marked point pro-cess,” Proceedings of the IEEE Southwest Symposium on Image Analysis andInterpretation, pp. 37–40, March 2016, Santa Fe, NM.
[80] Y. He, Y. Meng, H. Gong, S. Chen, B. Zhang, W. Ding, Q. Luo, and A. Li, “Anautomated three-dimensional detection and segmentation method for touchingcells by integrating concave points clustering and random walker algorithm,”PLoS ONE, vol. 9, no. 8, pp. e104 437 1–15, August 2014.
104
[81] O. Cicek, A. Abdulkadir, S. Lienkamp, T. Brox, and O. Ronneberger, “3DU-Net: Learning dense volumetric segmentation from sparse annotation,” Pro-ceedings of the Medical Image Computing and Computer-Assisted Intervention,pp. 424–432, October 2016, Athens, Greece.
[82] Q. Dou, H. Chen, L. Yu, L. Zhao, J. Qin, D. Wang, V. Mok, L. Shi, and P.-A. Heng, “Automatic detection of cerebral microbleeds from MR images via3D convolutional neural networks,” IEEE Transactions on Medical Imaging,vol. 35, no. 5, pp. 1182–1195, May 2016.
[83] H. Chen, Q. Dou, L. Yu, and P.-A. Heng, “VoxResNet: Deep voxelwise residualnetworks for volumetric brain segmentation,” arXiv preprint arXiv:1608.05895,August 2016.
[84] G. E. Hinton, “Training products of experts by minimizing contrastive diver-gence,” Neural Computation, vol. 14, no. 8, pp. 1771–1800, August 2002.
[85] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learningapplied to document recognition,” Proceedings of the IEEE, vol. 86, no. 11, pp.2278–2324, November 1998.
[86] Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” Nature, vol. 521, pp.436–444, May 2015.
[87] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet classification withdeep convolutional neural networks,” Proceedings of the Neural InformationProcessing Systems, pp. 1097–1105, December 2012, Lake Tahoe, NV.
[88] J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for se-mantic segmentation,” Proceedings of the IEEE Conference on Computer Visionand Pattern Recognition, pp. 3431–3440, June 2015, Boston, MA.
[89] V. Badrinarayanan, A. Kendall, and R. Cipolla, “SegNet: A deep convolu-tional encoder-decoder architecture for image segmentation,” arXiv preprintarXiv:1511.00561, 2015.
[90] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” arXiv preprint arXiv:1409.1556v6, April 2015.
[91] D. Ciresan, A. Giusti, L. M. Gambardella, and J. Schmidhuber, “Deep neuralnetworks segment neuronal membranes in electron microscopy images,” Pro-ceedings of the Neural Information Processing Systems, pp. 1–9, December 2012,Lake Tahoe, NV.
[92] B. Dong, L. Shao, M. D. Costa, O. Bandmann, and A. F. Frangi, “Deep learningfor automatic cell detection in wide-field microscopy zebrafish images,” Proceed-ings of the IEEE International Symposium on Biomedical Imaging, pp. 772–776,April 2015, Brooklyn, NY.
[93] M. Kolesnik and A. Fexa, “Multi-dimensional color histograms for segmenta-tion of wounds in images,” Proceedings of the International Conference ImageAnalysis and Recognition, pp. 1014–1022, September 2005, Toronto, Canada.
105
[94] O. Ronneberger, P. Fischer, and T. Brox, “U-Net: Convolutional networks forbiomedical image segmentation,” Proceedings of the Medical Image Comput-ing and Computer-Assisted Intervention, pp. 231–241, October 2015, Munich,Germany.
[95] S. E. A. Raza, L. Cheung, D. Epstein, S. Pelengaris, M. Khan, and N. Rajpoot,“MIMO-Net: A multi-input multi-output convolutional neural network for cellsegmentation in fluorescence microscopy images,” Proceedings of the IEEE In-ternational Symposium on Biomedical Imaging, pp. 337–340, April 2017, Mel-bourne, Australia.
[96] J. Yi, P. Wu, D. J. Hoeppner, and D. Metaxas, “Pixel-wise neural cell instancesegmentation,” pp. 373–377, April 2018.
[97] A. Arbelle and T. R. Raviv, “Microscopy cell segmentation via adversarial neu-ral networks,” pp. 645–648, April 2018.
[98] F. Xing, Y. Xie, and L. Yang, “An automatic learning-based framework forrobust nucleus segmentation,” IEEE Transactions on Medical Imaging, vol. 35,no. 2, pp. 550–566, February 2016.
[99] K. Sirinukunwattana, S. E. A. Raza, Y.-W. Tsang, D. R. J. Snead, I. A. Cree,and N. M. Rajpoot, “Locality sensitive deep learning for detection and classifi-cation of nuclei in routine colon cancer histology images,” IEEE Transactionson Medical Imaging, vol. 35, no. 5, pp. 1196–1206, May 2016.
[100] A. Prasoon, K. Petersen, C. Igel, F. Lauze, E. Dam, and M. Nielsen, “Deepfeature learning for knee cartilage segmentation using a triplanar convolutionalneural network,” Proceedings of the Medical Image Computing and Computer-Assisted Intervention, pp. 246–253, September 2013, Nagoya, Japan.
[101] F. Milletari, N. Navab, and S. A. Ahmadi, “V-Net: Fully convolutional neuralnetworks for volumetric medical image segmentation,” Proceedings of the IEEE2016 Fourth International Conference on 3D Vision, pp. 565–571, October 2016,Stanford, CA.
[102] D. J. Ho, C. Fu, P. Salama, K. W. Dunn, and E. J. Delp, “Nuclei detectionand segmentation of fluorescence microscopy images using three dimensionalconvolutional neural networks,” pp. 418–422, April 2018.
[103] F. Meyer, “Topographic distance and watershed lines,” Signal Processing,vol. 38, no. 1, pp. 113–125, July 1994.
[104] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep networktraining by reducing internal covariate shift,” arXiv preprint arXiv:1502.03167,March 2015.
[105] Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadar-rama, and T. Darrell, “Caffe: Convolutional architecture for fast feature em-bedding,” arXiv preprint arXiv:1408.5093, 2014.
[106] J. L. Clendenon, C. L. Phillips, R. M. Sandoval, S. Fang, and K. W. Dunn,“Voxx: a PC-based, near real-time volume rendering system for biological mi-croscopy,” American Journal of Physiology-Cell Physiology, vol. 282, no. 1, pp.C213–C218, January 2002.
106
[107] D. P. Kingma and J. L. Ba, “Adam: A method for stochastic optimization,”arXiv preprint arXiv:1412.6980, pp. 1–15, December 2014.
[108] S. Lee, P. Salama, K. W. Dunn, and E. J. Delp, “Segmentation of fluorescencemicroscopy images using three dimensional active contours with inhomogeneitycorrection,” Proceedings of the IEEE International Symposium on BiomedicalImaging, pp. 709–713, April 2017, Melbourne, Australia.
[109] A. E. Carpenter, T. R. Jones, M. R. Lamprecht, C. Clarke, I. H. Kang,O. Friman, D. A. Guertin, J. H. Chang, R. A. Lindquist, J. Moffat, P. Golland,and D. M. Sabatini, “CellProfiler: Image analysis software for identifying andquantifying cell phenotypes,” Genome Biology, vol. 7, no. 10, pp. R100–1–11,October 2006.
[110] C. Fu, D. J. Ho, S. Han, P. Salama, K. W. Dunn, and E. J. Delp, “Nucleisegmentation of fluorescence microscopy images using convolutional neural net-works,” Proceedings of the IEEE International Symposium on Biomedical Imag-ing, pp. 704–708, April 2017, Melbourne, Australia.
[111] P. A. Yushkevich, J. Piven, H. C. Hazlett, R. G. Smith, S. Ho, J. C. Gee, andG. Gerig, “User-guided 3D active contour segmentation of anatomical struc-tures: Significantly improved efficiency and reliability,” NeuroImage, vol. 31,no. 3, pp. 1116–1128, July 2006.
[112] D. M. W. Powers, “Evaluation: from Precision, Recall and F-measure toROC, Informedness, Markedness and Correlation,” Journal of Machine Learn-ing Technologies, vol. 2, no. 1, pp. 37–63, December 2011.
[113] T. Fawcett, “An introduction to ROC analysis,” Pattern Recognition Letters,vol. 27, no. 8, pp. 861–874, June 2006.
[114] K. U. Barthel, “Volume viewer,” https://imagej.nih.gov/ij/plugins/volume-viewer.html.
VITA
107
VITA
Chichen Fu was born in Nanchang, Jiangxin Province, China. He received the
Bachelor of Science in Electrical Engineering from Purdue University, West Lafayette,
Indiana in 2014.
Chichen Fu then joined the Ph.D program at the School of Electrical and Com-
puter Engineering at Purdue University in August 2014. He worked as a research
assistant at the Video and Image Processing Laboratory (VIPER) under supervision
of Professor Edward J. Delp. Chichen Fu’s research interests include image process-
ing, computer vision and deep learning.
He is a student member of the IEEE, the IEEE Signal Processing Society.