1 Re-localisation of Microscopic Lesions in their Macroscopic Context for Surgical Instrument Guidance Baptiste Allain A dissertation submitted for the degree of Doctor of Philosophy of the University College London - UCL Centre for Medical Image Computing - CMIC Department of Medical Physics and Bioengineering University College London Submitted in July 2011
208
Embed
Re-localisation of microscopic lesions in their ...discovery.ucl.ac.uk/1338127/1/1338127.pdf · Macroscopic Context for Surgical Instrument Guidance ... Caroline Rodier, ... MICROSCOPIC
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
1
Re-localisation of Microscopic Lesions in their
Macroscopic Context for Surgical Instrument
Guidance
Baptiste Allain
A dissertation submitted for the degree of
Doctor of Philosophy
of the
University College London - UCL
Centre for Medical Image Computing - CMIC
Department of Medical Physics and Bioengineering
University College London
Submitted in July 2011
2
I, Baptiste Allain, confirm that the work presented in this thesis is my own. Where
information has been derived from other sources, I confirm that this has been indicated in the
thesis.
3
Abstract Optical biopsies interrogate microscopic structure in vivo with a 2mm diameter miniprobe
placed in contact with the tissue for detection of lesions and assessment of disease
progression. After detection, instruments are guided to the lesion location for a new optical
interrogation, or for treatment, or for tissue excision during the same or a future examination.
As the optical measurement can be considered as a point source of information at the surface
of the tissue of interest, accurate guidance can be difficult. A method for re-localisation of the
sampling point is, therefore, needed.
The method presented in this thesis has been developed for biopsy site re-localisation
during a surveillance examination of Barrett’s Oesophagus. The biopsy site, invisible
macroscopically during conventional endoscopy, is re-localised in the target endoscopic
image using epipolar lines derived from its locations given by the tip of the miniprobe visible
in a series of reference endoscopic images. A confidence region can be drawn around the re-
localised biopsy site from its uncertainty that is derived analytically. This thesis also presents
a method to improve the accuracy of the epipolar lines derived for the biopsy site re-
localisation using an electromagnetic tracking system.
Simulations and tests on patient data identified the cases when the analytical
uncertainty is a good approximation of the confidence region and showed that biopsy sites
can be re-localised with accuracies better than 1mm. Studies on phantom and on porcine
excised tissue demonstrated that an electromagnetic tracking system contributes to more
accurate epipolar lines and re-localised biopsy sites for an endoscope displacement greater
than 5mm. The re-localisation method can be applied to images acquired during different
endoscopic examinations. It may also be useful for pulmonary applications. Finally, it can be
combined with a Magnetic Resonance scanner which can steer cells to the biopsy site for
tissue treatment.
4
The patient data collection was undertaken with the ethical approval 08/H0808/08 in the
Department of Gastroenterology of University College London Hospitals, UCLH NHS.
5
Acknowledgements I would like to thank my supervisors and employers Dr. Richard J. Cook, Dr. Mingxing Hu,
and Professor David J. Hawkes. They welcomed me in the department of Biomaterials,
Dental Institute, King’s College London, and in the Centre for Medical Image Computing,
University College London. They encouraged and guided me, but they also helped me find
my own solutions and make contributions. They put a lot of effort to correct and to suggest
improvements to my publications and my thesis. I am grateful to Dr. Richard Cook who
provided me with my salary through the Department of Health, UK, in order to accomplish
my work and my thesis and who gave me full access to his Fibered Confocal Microscope
1.2 BACKGROUND: DETECTION OF LESIONS STARTING AT THE SUPERFICIAL LAYERS OF TISSUE BY
OPTICAL BIOPSY .................................................................................................................................29 1.2.1 Development of cancers ......................................................................................................30 1.2.2 In vivo and in situ detection of lesions by optical biopsy ....................................................31
1.3 EXAMPLES OF INFORMATION EXTRACTED BY OPTICAL BIOPSY.....................................................32 1.3.1 Optical biopsy for the study of the spectrum of light after interaction with the tissue ........32 1.3.2 Optical biopsy for the study of the morphology of the cells................................................32 1.3.3 Detection of lesions based on functional imaging ...............................................................34
1.4 MOTIVATION ................................................................................................................................36 1.4.1 Extensions of the field of view of microscopic images .......................................................36 1.4.2 Need for accurate re-localisation of microscopic lesions detected by optical biopsy in the
macroscopic space of the organ of interest ...................................................................................37
1.5 STATEMENT OF CONTRIBUTION ....................................................................................................38
1.6 STRUCTURE OF THE THESIS...........................................................................................................39
CHAPTER 2 INITIAL PILOT WORK TO ASSESS THE IN VIVO USE OF THE FIBERED
CONFOCAL MICROSCOPE AND ITS USE IN COMBINATION WITH MRI.................................43
2.2 FIBERED CONFOCAL MICROSCOPY...............................................................................................43 2.2.1 Confocal microscopy ...........................................................................................................43 2.2.2 Fibered confocal microscopy...............................................................................................45 2.2.3 Experiment: Level of details reached by the fibered confocal microscope .........................46
2.2.3.1 Materials and method...................................................................................................................46 2.2.3.2 Results and discussion .................................................................................................................47
2.3 POTENTIAL COMBINATION WITH MAGNETIC RESONANCE IMAGING FOR THE MONITORING OF THE
DELIVERY OF MAGNETIC CELLS TOWARDS A SITE OF INTEREST ..........................................................49 2.3.1 Delivery of cells using a magnetic resonance imaging system............................................49
2.3.1.1 Materials and method...................................................................................................................49 2.3.1.2 Results and discussion .................................................................................................................50
2.3.2 Localisation of the fibered confocal microscope miniprobe in high-field Magnetic
Resonance images.........................................................................................................................52 2.3.2.1 Materials and method...................................................................................................................52 2.3.2.2 Results..........................................................................................................................................53
2.3.3 Tracking of the FCM miniprobe in an MR scanner.............................................................54
3.2 RE-LOCALISING MICROSCOPIC LESIONS IN THEIR MACROSCOPIC CONTEXT ..................................57 3.2.1 Re-localising lesions within a pre-operative image .............................................................57 3.2.2 Re-localising lesions in endoscopic images.........................................................................59 3.2.3 Re-localising lesions in interventional Magnetic Resonance Images ..................................62
3.3 A CLINICAL APPLICATION: DETECTION OF CANCERS IN BARRETT’S OESOPHAGUS........................64 3.3.1 Cancers in Barrett’s Oesophagus and conventional diagnosis.............................................64 3.3.2 Detection of the biopsy sites in BO by optical biopsy .........................................................66 3.3.3 Need for accurate re-localisation of biopsy sites during a surveillance examination of BO....
8 3.4 COMPUTATION OF A MAPPING BETWEEN ENDOSCOPIC IMAGES.....................................................69
3.4.1 Endoscopic images acquired during a surveillance examination of BO..............................70 3.4.2 Possible mappings ...............................................................................................................72 3.4.3 Computation of a mapping by recovery of the epipolar geometry ......................................74
3.5 REVIEW OF THE METHODS FOR THE RECOVERY OF THE EPIPOLAR GEOMETRY ..............................76 3.5.1 Endoscope camera calibration and correction of image distortions.....................................76 3.5.2 Feature detection and matching ...........................................................................................78
3.5.2.1 Feature trackers: the example of the Lucas-Kanade tracker.........................................................78 3.5.2.2 Feature detection in the image scale-space and matching of descriptors: the example of the Scale
Invariant Feature Transform ....................................................................................................................81 3.5.3 Computation of the fundamental matrix ..............................................................................84
3.5.3.1 Properties of the fundamental matrix ...........................................................................................84 3.5.3.2 Computation of the fundamental matrix from a minimal set of matches .....................................85 3.5.3.3 Robust estimations of the fundamental matrix.............................................................................86 3.5.3.4 Optimisation of the computation of the fundamental matrix........................................................91 3.5.3.5 Summary of the computation of the fundamental matrix for a pair of images .............................92
4.2 ANALYSIS OF FEATURES IN ENDOSCOPIC IMAGES .........................................................................96 4.2.1 Feature detection in the image scale-space and matching of descriptors.............................96 4.2.2 Experiment: study of the error for the localisation of the features.....................................100
4.2.2.1 Materials and method.................................................................................................................101 4.2.2.2 Results........................................................................................................................................101
4.3 ANALYSIS OF THE METHODS FOR THE ESTIMATION OF THE CAMERA MOVEMENT .......................102 4.3.1 Experiment: comparison of the estimations of the fundamental matrix with LMedS,
RANSAC, and MAPSAC...........................................................................................................103 4.3.1.1 Materials and method.................................................................................................................103 4.3.1.2 Results and discussion ...............................................................................................................104
4.3.2 Experiment: number of matches for the computation of the fundamental matrix .............104 4.3.2.1 Materials and method.................................................................................................................105 4.3.2.2 Results and discussion ...............................................................................................................106
5.2 RE-LOCALISATION PRINCIPLE .....................................................................................................109 5.2.1 Re-localisation with 2 epipolar lines..................................................................................110 5.2.2 Limits of the re-localisation with 2 epipolar lines due to their uncertainty .......................111 5.2.3 Extension of the re-localisation with N epipolar lines .......................................................112
5.3 EXPERIMENT 1: STUDY BY SIMULATIONS OF THE RE-LOCALISATION PRECISION AND BIAS WITH THE
LOCATIONS OF THE MATCHES PERTURBED BY A GAUSSIAN NOISE AND WITH THE PRESENCE OF
5.4 EXPERIMENT 2: STUDY OF THE INFLUENCE OF THE ANGLE OF THE EPIPOLAR LINES ON THE
ACCURACY OF THE RE-LOCALISED BIOPSY SITE USING PATIENT DATA..............................................126 5.4.1 Materials and method ........................................................................................................126 5.4.2 Results ...............................................................................................................................127
6.2 EXPERIMENTAL AND ANALYTICAL COMPUTATIONS OF THE UNCERTAINTY OF A VECTOR ...........131 6.2.1 Confidence ellipse and precision .......................................................................................131 6.2.2 Experimental estimation of the uncertainty and of the precision.......................................131 6.2.3 Error propagation for the analytical estimation of the uncertainty ....................................132
6.3 DERIVATION OF THE UNCERTAINTY OF THE RE-LOCALISED BIOPSY SITE ....................................135 6.3.1 Discussion of the hypotheses of the experimental and analytical derivation of the
uncertainty in the case of the re-localised biopsy site computed with N epipolar lines..............135
10 6.3.2 Analytical estimation of the uncertainty of the biopsy site re-localised with N > 2 epipolar
6.4 EXPERIMENT: COMPARISON OF THE UNCERTAINTIES DERIVED ANALYTICALLY AND
STATISTICALLY BY SIMULATIONS.....................................................................................................138 6.4.1 Method...............................................................................................................................139 6.4.2 Results and discussion .......................................................................................................140
6.4.2.1 Results of the simulations ..........................................................................................................140 6.4.2.2 Results on patients .....................................................................................................................143
CHAPTER 7 TEST OF THE RE-LOCALISATION METHODS ON PHANTOM AND PATIENT
DATA ..................................................................................................................................................145
8.2 RE-LOCALISATION WITH AN EM TRACKING SYSTEM ..................................................................155 8.2.1 Context, hypotheses, and description of an EM tracking system.......................................155 8.2.2 Combination of the EM tracking system with the re-localisation algorithm .....................158
8.2.3 Computation of ( ( )TIF ,i)EM during the first step of the hybrid method.............................160
8.3 EXPERIMENTS AND RESULTS ......................................................................................................161 8.3.1 Experiment 1: error of an EM tracking system for the determination of the displacement
and of the orientation of the EM sensor......................................................................................162 8.3.1.1 Materials and method.................................................................................................................163 8.3.1.2 Results and discussion ...............................................................................................................164
11 8.3.2 Experiment 2: error of the positioning of the epipolar lines derived with the re-localisation
system.........................................................................................................................................165 8.3.2.1 Methods and materials ...............................................................................................................165 8.3.2.2 Results and discussion ...............................................................................................................168
8.3.3 Experiment 3: test of the method on excised organs from pigs .........................................177 8.3.3.1 Materials and method.................................................................................................................177 8.3.3.2 Results........................................................................................................................................179
Fig. 5-1: Method for biopsy site re-localisation with 2 reference images I1 and I2, a target image T and
two epipolar lines ( )1Iel and ( )2Iel : the biopsy site is seen under 2 different viewpoints of the
endoscope camera. This results in two distinct epipolar lines ( )1Iel and ( )2Iel that form an intersection
at the location of the biopsy site in T. ..................................................................................................110
Fig. 5-2: Uncertainty of the epipolar line: it corresponds to a confidence region which is a hyperbola.
The matched point is located on the epipolar line where the two arms narrow to a minimum. ...........112
Fig. 5-3: Condition of triangulation: the two axes passing respectively through Camera centre i and ( )iIp , and through Camera centre T and p, must meet at the position of the biopsy site P in the 3D
Fig. 6-3: Analytical and experimental 99% confidence ellipses for N > 2 epipolar lines in the target
image: each row corresponds to a sequence acquired on a patient and presents first the target image
with the location of the biopsy site, secondly the analytical ellipse (green) and the experimental ellipse
(blue), and finally the KL divergences. ................................................................................................144
Fig. 7-1: Endoscopic image of a) the phantom with a white-light endoscope: the blue point corresponds
to the ground truth of the biopsy site and b) a patient’s oesophagus with an NBI endoscope: an APC
burn indicates the ground truth of the biopsy site. ...............................................................................146
21 Fig. 7-2: Two gastroscopic sequences acquired with an NBI endoscope: a) to d) are images extracted
from a sequence where the biopsy site (APC mark) was observed under various viewpoints; e) to h) are
from a sequence with a miniprobe in the camera FOV. .......................................................................146
Fig. 7-3: Failure case of the re-localisation method: the camera moved along the endoscope central
axis. In a), b), and c), the epipole derived from 3 reference images is displayed in the target image. It is
the intersection of the epipolar lines (red) derived from each feature of the reference image. The
epipole does not move much. This results in d): the bundle of epipolar lines (blue lines) used for the re-
localisation of the biopsy site subtend very small angles. The yellow point indicates the ground-truth
position of the biopsy site. The red point indicates the re-localised biopsy site...................................150
Fig. 7-4: Movement of the epipole in the target image T: figures a), b), c), and d) show the position of
the epipole in T derived from a series of consecutive reference images Ii. The epipole moves towards
the centre of the image T which is the result of the rotation of the endoscope tip. ..............................151
Fig. 7-5: Examples of re-localised biopsy sites: for each sequence, the four images are the target image
of each sequence with the features displayed (green dots) or the epipolar lines (blue lines) derived from
the previous images or the confidence region (green ellipse). These are the results obtained for feature
detection and matching with the LK tracker. For the two first sequences, the fourth image is an
enlargement around the confidence region...........................................................................................151
Fig. 8-1: Critical cases for a good performance of the LK tracker: two sequences of endoscopic images
aquired during a surveillance examination of Barrett’s Oesophagus (BO) are presented as examples.
For each sequence, 3 endoscopic images are extracted to illustrate the problems that may be
encountered during endoscopy. For sequence 1 and sequence 2, the oesophagus surface is interrogated
by optical biopsy (top row), air/water bubbles may obstruct the endoscope Field Of View (FOV) or the
endoscope may move too fast when the miniprobe is removed (middle row), and the endoscopic
images are clear again (bottom row). ...................................................................................................154
Fig. 8-2: Main components of an EM tracking system (medSAFE system by Ascension Technology
Corporation): a) the emitter which generates electromagnetic waves in order to help measure the
position of b) the EM sensor which may be attached to the tip of the endoscope. The position and the
orientation of the EM sensor in the emitter coordinate system are measured. .....................................157
Fig. 8-3: Description of the EM sensor coordinate system (S, xS, yS, zS) in the EM emitter coordinate
system (O, xem, yem, zem) with spherical coordinates: the azimuth ψ, elevation θ, and roll Φ angles. ..158
Fig. 8-4: Hybrid method for biopsy site re-localisation: information from the EM tracking system helps
recover approximately the epipolar geometry formed by each pair of endoscopic images Ii ↔ T. This
22
returns a set of fundamental matrices ( ( )TIF ,i)EM which help constrain the matching process of the SIFT
features. Once the features have been matched, the estimation of the epipolar geometries is refined
which returns a set of more accurate fundamental matrices ( ( )TIF ,i)fused and the re-localisation method
can be applied.......................................................................................................................................159
Fig. 8-5: Spatial constraint during SIFT feature matching: the search in the target image T for the
feature matching the feature ( )ijIp in Ii is constrained in a bounded region (blue dotted lines) centred on
the epipolar line ( ( )ijIel )EM derived from ( )i
jIp . The green dots are the features. .................................160
Fig. 8-6: Relations between the coordinate systems of the camera, of the EM sensor, and of the EM
The RANSAC finds the fundamental matrix that maximises the number of inliers and
minimises ( )TI ,iS defined as (Torr and Murray, 1997; Torr and Zisserman, 2000):
( )( )
( )
( )
∑=
⎟⎟
⎠
⎞
⎜⎜
⎝
⎛=
1..Lj
ji
ii
i
eρS 2
2
,,,
minminσ
I
FTIF TITI
with ( ) ( )
( ) ⎪⎭
⎪⎬⎫
⎪⎩
⎪⎨⎧
≥
<=⎟
⎟
⎠
⎞
⎜⎜
⎝
⎛
TeT
Teeρ
i
ii
j
jj
I
II
if
if02
2
σ.
(3-24)
The threshold is set such that T = 1.96σ where σ is the standard deviation of the residuals ( )ije I . The standard deviation is estimated with the LMedS with a training image before
applying the RANSAC (Zhang, 1998).
The Maximum A Posteriori SAmple Consensus (MAPSAC):
The MAPSAC was introduced by Torr (2002). The matches { ( )ijIp , pj}j=1..L in the endoscopic
image Ii and in the target image T are linked with a 2 view relation R with parameters θ =
{ ( )TIF ,i, β , γ}, β are the corrected locations of the features { ( )i
jIp , pj}j=1..L corrupted by the
noise in the images, and γ indicates whether the matches are inliers or outliers. The noise
model and R define a hypothesized model M for the set of matches noted D = { ( )ijIp , pj}j=1..L.
The parameter θ is optimally estimated such that:
( )
( ) ( )( )IM|D
IMIM|D
IMD
,Pr,|Pr,,Prmaxarg
,,|Prmaxargˆ
θθ
θθ
θ
θ
=
=
.
(3-25)
where I is the information upon which all the probabilities are conditioned. The first term Pr(D
| θ, M, I) is the likelihood term, the second term Pr(θ | M, I) is the prior term, and the
denominator Pr(D | M, I) is called the evidence which is constant for a fixed M (Torr, 2002).
The prior is supposed sufficiently diffuse to be constant (Torr, 2002).
For an estimated set of parameters θ = { ( )TIF ,i , β , γ}, the residuals ( )i
je I that the
matches form with ( )TIF ,i to fit the model M follow a normal distribution of zero mean and
91
variance σ2 (Torr, 2002). The outliers follow a uniform distribution of parameter v1 where v is
the volume of the space where outliers are located. Indeed, outliers can lie anywhere within
the endoscopic image. Therefore, the likelihood of a model is given as follows:
( )( )
( )∏=
⎟⎟
⎠
⎞
⎜⎜
⎝
⎛−+⎟
⎟
⎠
⎞
⎜⎜
⎝
⎛−⎟⎟
⎠
⎞⎜⎜⎝
⎛=
Ljj
jj v
e i
..12
24
2
112
exp2
1,,,,|Pr γσπσ
γγβαI
IMD .
(3-26)
The set of parameters θ = { ( )TIF ,i , β , γ} needs to be optimally estimated which is equivalent
to maximising equations (3-25) or (3-26) or to minimising a cost function ( )TI ,iS defined as:
( )( )
( )
( )
∑=
⎟⎟
⎠
⎞
⎜⎜
⎝
⎛=
1..Lj
ji
ii
i
eρS 2
2
,,,
minminσ
I
FTIF TITI
with ( )
( ) ( )
( ) ⎪⎪⎭
⎪⎪⎬
⎫
⎪⎪⎩
⎪⎪⎨
⎧
≥
<=⎟
⎟
⎠
⎞
⎜⎜
⎝
⎛
TeT
Tee
eρ
i
ii
i
j
jjj
2
2
2
2
2
2
2
if
if
I
III
σσσ
and
⎟⎟⎠
⎞⎜⎜⎝
⎛= 2
2
2log
πσWT .
(3-27)
where W is the dimension of the search window within which the matched feature may be in
the target image T and σ is the standard deviation of the residuals ( )ije I (Torr, 2002; Torr and
Davidson, 2003).
3.5.3.4 Optimisation of the computation of the fundamental matrix
The robust methods require setting a number of parameters in order to optimise the
estimation of ( )TIF ,i, like:
- The number of samples m of seven matches or eight matches with the seven-point or
eight-point algorithm.
- The selection of the seven or eight matches in such a way that the Monte Carlo
technique is efficient.
Ideally, all of the samples of matches from D = { ( )ijIp , pj}j=1..L have to be tested, but this is
usually computationally infeasible. Thus, the number of samples needs to be set such that
there is a statistical significance. Fischler and Bolles (1981) and Rousseeuw and Leroy (1987)
presented slightly different means of calculations, but both return similar numbers (Zhang,
1998). Rousseeuw’s approach consists of choosing the number of samples m sufficiently high
to give the probability Γ in excess of 99% that a good sample is selected. The probability that
at least one of the m samples is good is given by:
92
( )[ ]mpε−−−=Γ 111 . (3-28)
The parameter p is the number of feature matches, and ε is the assumed fraction of outliers.
Thus, the number of samples is given by:
( )( )( )pm
ε−−Γ−
=11log
1log.
(3-29)
The values of m as functions of ε are given in Table 3-1.
The number of samples m is set such that there is at least one good sample of
matches. However, Zhang (1998) noticed that the location of the features of the seven or
eight matches of the samples may be very close to each other in both images. As stated in
Luong and Faugeras (1996), such a situation leads to unstable and imprecise fundamental
matrices. Therefore, Zhang (1998) suggested a method to achieve higher stability and
efficiency for the recovery of the epipolar geometry. Each image is divided in bxb buckets
which group together features that are spatially close (Fig. 3-21 a). During the Monte Carlo
process, the principle consists of selecting randomly 7 or 8 mutually different buckets and
then randomly choosing one match in each selected bucket. However, the number of features
in one bucket may be very different from that in another bucket. It is preferable to select a
bucket having many matches and the selection of the buckets cannot be entirely random. If
there are in total l buckets, the range [0,1] is divided into l intervals such that the width of the
ith interval is equal to ni/Σini, where ni is the number of matches in the ith bucket (Fig. 3-21 b)).
A random number is generated in the interval [0,1] and falls in the ith interval which means
that the ith bucket is selected.
3.5.3.5 Summary of the computation of the fundamental matrix for a pair of images
The main steps of the computation of the fundamental matrix were presented in Fig. 3-16.
Fig. 3-22 presents the steps in more details by integrating the algorithms and methods
presented in the previous sections. The remaining questions are:
- The choice of a technique for feature detection and matching
- The number of features that need to be detected for the computation of the
fundamental matrix
- The choice of the Random Sampling Consensus to determine the fundamental matrix
These choices are discussed in the next chapter Chapter 4 ‘Feature Analysis in Endoscopic
Images and Endoscope Camera Movement’.
93
a) b) Fig. 3-21: The process of matches’ selection: a) the image is divided in buckets to group together the
features that are spatially close; b) the buckets are selected based on the number of features.
Table 3-1: Minimum number m of matches to draw for the estimation of the fundamental matrix:
Percentage of outliers
10% 20% 30% 40% 50% 60% 70% 80% 90%
m
for p = 7 8 20 54 163 588 2809 21 055 359 780 5x107
m
for p = 8 9 26 78 272 1 177 7 025 70 188 1.8x106 4.6x108
inputs Image IiImage T
Feature detection and matching with the LK tracker or the SIFT
Fundamental matrix computation:1. Translate and scale the coordinates of the features2. Apply the 7-point algorithm in a random sampling
consensus (LMedS, RANSAC, or MAPSAC):a. The features are drawn from 7 different bucketsb. The cost to minimise is a function of the Sampson’s
residuals3. Denormalise the first estimation of the fundamental matrix4. Refine this first estimation over the set of inliers by
constrained nonlinear optimisation
Correction of the endoscopic image distortionspreliminary step
F( , T)output
Ii Fig. 3-22: Summary of the framework and possible algorithms for the computation of the fundamental
matrix.
Image not displayed for Copyright reasons
94
3.6 Conclusion
Lesions detected by optical biopsy during gastroscopic, colonoscopic, or pulmonary
examinations can be re-localised using either pre-operative images and EM tracking systems,
or endoscopic images, or interventional MR images, or a combination of these modalities. A
clinical application of optical biopsies is the detection of biopsy sites invisible
macroscopically in the oesophagus during a surveillance examination of BO. Other
applications of optical biopsies can be the detection of flat polyps and microscopic colitis in
colon. These examinations make use of interventional endoscopic images. Thus, the re-
localisation of the lesions developed in this thesis for these clinical applications makes use of
endoscopic images only. The epipolar geometry formed by a reference image where the
biopsy site location is known and a target image where the biopsy site needs to be re-
localised has to be recovered for the specific case of the endoscopic images. The choice of the
algorithms is discussed in the next chapter.
95
Chapter 4 Feature Analysis in Endoscopic Images and
Endoscope Camera Movement
4.1 Introduction
The biopsy site is re-localised in a target endoscopic image T using the epipolar geometries
formed by this image with each reference image Ii. For each pair of images Ii ↔ T, the
epipolar geometry can be described algebraically by the fundamental matrix ( )TIF ,i. The
recovery of the epipolar geometry requires (Fig. 4-1):
- The correction of the distortions in the endoscopic images.
- The detection of features in the endoscopic images Ii and T and matching the features
together.
- The estimation of the fundamental matrix ( )TIF ,i from the set of matched features.
As the epipolar lines derived from the biopsy site locations in the reference images Ii
are used to re-localise the biopsy site in the target image T, they have to be computed with
high accuracy. This accuracy depends on:
- The number and proportion of good matched features termed inliers (Hartley and
Zisserman, 2004).
inputs Image I1Image T
Image IiImage T
Image INImage T
Feature detection and matching
Fundamental matrix computation
… …
Correction of the endoscopic image distortionspreliminary step
F( ,T)outputs F( ,T) F( ,T)I1 Ii IN Fig. 4-1: Framework for the estimation of the fundamental matrices that each reference image Ii forms
with the target image T.
96
- The wide spread of the features on the whole physical surface observed in the
endoscopic image (Luong and Faugeras, 1996).
- The method used to compute the fundamental matrix (Torr and Murray, 1997; Zhang,
1998; Torr and Zisserman, 2000; Hartley and Zisserman, 2004).
This chapter aims to present a method to compute an accurate fundamental matrix in
the specific context of endoscopic images. Methods for feature detection and matching are
discussed in this context. Algorithms for the recovery of the epipolar geometry as well as
their hypotheses are also discussed since their practical use can help determine which feature
detection and matching technique is well-adapted and how many features have to be detected
for the derivation of accurate epipolar lines.
4.2 Analysis of features in endoscopic images
Two main methods for feature detection and matching were presented in Chapter 3
‘Literature Review: Possible Approaches for Biopsy Site Re-localisation and Application for
the Surveillance Examination of Barrett’s Oesophagus’. These were the Lucas-Kanade (LK)
tracker which tracks features through endoscopic images and the Scale Invariant Feature
Transform which detects features independently in the images and which matches them using
their descriptors. The pros and cons of using these methods are discussed. Besides, the
algorithms used for the estimation of the fundamental matrix make the assumption that the
features are localised with an error that follows a Gaussian distribution. This assumption is
studied for the case of endoscopic images.
4.2.1 Feature detection in the image scale-space and matching of descriptors
The SIFT and the LK tracker have been used for feature detection and matching in
endoscopic images, for example by Mountney et al. (2006), Hu et al. (2007, 2009), and Luo
et al. (2010). The use of one method mainly depends on the type of sequences that need
processing.
SIFT feature matching has the advantage of detecting features independently in the
reference images and in the target image. Therefore, there is no tracking through consecutive
images to find the matches. The SIFT may be appropriate to sequences where intermediate
endoscopic images are blurred, or corrupted by the presence of air/water bubbles or by
specularities (Mori et al., 2005). Indeed, for this type of sequences, the LK tracker loses track
of the features and a method to start tracking again needs to be developed.
The neighbourhoods of the features that commonly correspond to vessel curvatures
and intersections look similar within an image and their visual content can change
97
significantly when the viewpoint of the endoscope camera changes (Atasoy et al., 2009). This
is illustrated with two examples: for this thesis, features were detected and matched for 2
pairs of reference and target images acquired during a surveillance examination of BO. The
pairs were processed first with the LK tracker, and secondly with the SIFT. Anatomical
points corresponding to vessel intersections or curvatures were detected as feature points with
both the LK tracker and the SIFT. One feature for each pair was considered as an example of
the limits of the SIFT. Both examples show that the LK tracker returned the right match
while SIFT returned the wrong match (Fig. 4-2 and Fig. 4-3 present the results for the first
pair, and Fig. 4-4 and Fig. 4-5 present the results for the second pair). This is mainly due to
the spatial constraint that the LK tracker imposes when it tracks a feature while the SIFT does
not take into account this constraint. The effect of the matching method on the re-localisation
will be discussed in more details in Chapter 7 ‘Test of the Re-localisation Methods on
Phantom and Patient Data’.
In the remainder of this thesis, the LK tracker will be used for non critical endoscopic
sequences for which the endoscope camera movement is controlled and for which the
acquired images are neat. The SIFT will be used for the critical endoscopic sequences for
which the camera moves too quickly or for which air/water bubbles appear.
a) b)
c) d) Fig. 4-2: Results of the matching of features using the LK tracker for the first pair of endoscopic
images acquired during an endoscopy examination for the surveillance of BO: a) a feature (green
point) was detected in the reference image and was located at the intersection of blood vessels; b)
zoomed image on the feature; c) the LK tracker tracked the feature successfully and the resulting
matched feature in the target image was located at the intersection of the same blood vessels; d)
zoomed image on the matched feature.
98
a) b)
c) d)
e) f)
Fig. 4-3: Illustration of a mismatch with the SIFT for the first pair of endoscopic images acquired
during a surveillance examination of Barrett’s Oesophagus: a) a feature has been detected in the
reference image of the oesophagus at a given scale and with a given orientation: the feature is at the
centre of the circle whose radius is proportional to the scale of the feature and the drawn radius
indicates the orientation of the feature; b) zoomed image on the detected feature in a); c) the drawn
feature in the target image is the actual match of the feature drawn in a) but the matching process does
not match these two features; d) the zoomed image on the detected feature in c) shows that the
orientation of the feature is not the same as that of the feature in a); e) the matching process matches
the feature in a) with the feature in e) which does not correspond to the same anatomical point; f)
zoomed image on the feature detected in e).
99
a) b)
c) d)
e) f) Fig. 4-4: Illustration of a mismatch with the SIFT for the second pair of endoscopic images: a) a
feature has been detected in the reference image of the oesophagus at a given scale and with a given
orientation; b) zoomed image on the detected feature in a); c) the drawn feature in the target image is
the actual match of the feature drawn in a) but the matching process does not match these two features;
d) the zoomed image on the detected feature in c) shows that the orientation of the feature is not the
same as that of the feature in a); e) the matching process matches the feature in a) with the feature in e)
which does not correspond to the same anatomical point; f) zoomed image on the feature detected in
e).
100
a) b)
c) d) Fig. 4-5: Results of the matching of features using the LK tracker for the second pair of endoscopic
images acquired during an endoscopy examination for the surveillance of BO: a) a feature (green
point) was detected in the reference image and was located at the intersection of blood vessels; b)
zoomed image on the feature; c) the LK tracker tracked the feature successfully and the resulting
matched feature in the target image was located at the intersection of the same blood vessels; d)
zoomed image on the matched feature.
Feature 1Feature 4
Feature 2
Feature 3
Feature 1Feature 4
Feature 2
Feature 3
a) b) c) Fig. 4-6: Illustration of the error of the localisation of 4 detected features in an endoscopic image: a) a
255 pixels x 165 pixels endoscopic image of the pattern corresponding to an endoscopic image
acquired on a patient during a surveillance examination of Barrett’s Oesophagus; b) the green points
correspond to the 100 estimations of the features detected with the LK tracker in 100 endoscopic
images of the pattern; c) the green points correspond to the 100 estimations of the features detected
with SIFT in 100 endoscopic images of the pattern.
4.2.2 Experiment: study of the error for the localisation of the features
Features are detected with an error due to the noise in endoscopic images. Many authors
made the classical assumption that the errors of the localisation of features are identically and
independently distributed with a Gaussian distribution in order to use computer vision
101
algorithms, especially those for the computation of the fundamental matrix (Luong and
Faugeras, 1996; Csurka et al., 1997; Torr and Murray, 1997; Zhang, 1998; Hartley and
Zisserman, 2004). This experiment aimed to study the distribution of the error of the
localisation of features detected with the LK tracker and with SIFT in endoscopic images
whose distortions were corrected.
4.2.2.1 Materials and method
An image from an endoscopy sequence acquired during a surveillance examination of
Barrett’s Oesophagus was printed out. This image was glued to a flat rigid box, and placed in
front of a digital high resolution endoscope mounted with a video colour CCD camera
(Pentax Ltd.). The box and the endoscope were maintained still in order to acquire the same
endoscopic images and to evaluate the uncertainty of the feature localisation. Fig. 4-6 a)
presents an example of the acquired endoscopic images. Features were detected first with the
LK tracker and secondly with SIFT in 100 acquired images of the pattern stuck on the box. A
separate acquired image was used to set the parameters of the LK tracker and of SIFT for the
detection of features. These parameter values were used for the 100 other images. Gaussian
distribution of the error for 4 different feature locations was evaluated as presented by
Johnson and Wichern (1998). Features were selected such that they differed in contrast and
such that they corresponded to very different patterns. The feature of interest can be
considered as a random vector pj whose location ( )ijIp is estimated in each image Ii. If the
error on pj were from a bivariate Gaussian distribution, the probability for pj to satisfy (pj -
μj)trΛpj-1(pj - μj) ≤ χ2
2(0.5) should be 0.5 where μj and Λpj are the mean and the covariance
matrix of the Gaussian distribution, and χ2 is the chi-square distribution with 2 degrees of
freedom (Johnson and Wichern, 1998). The test of normality of pj consists, therefore, of
computing the mean pjmean and the covariance matrix Sj of the observations ( )ijIp , and of
counting the percentage of sample observations ( )ijIp that satisfy (pj - pjmean)trSj
-1(pj - pjmean) ≤
χ22(α) for α = [0.25 , 0.5 , 0.75 , 0.99]. Statistical tests, such as the Kolmogorov-Smirnov test,
provide an alternative way to test the normality of the localisation of the features. However,
such tests are designed to demonstrate the non-normality of a distribution.
4.2.2.2 Results
The locations of the features of interest were displayed as green points in one of the
100 acquired endoscopic images (Fig. 4-6). These locations varied for both detections with
102
the LK tracker and SIFT. The percentage of sample observations was approximately the same
as the value of α except for a few cases especially for the features detected with the LK
tracker (Table 4-1 and Table 4-2). The reason is that the implementation of the LK tracker
used in this thesis detects feature locations at the pixel resolution. It results from this
implementation that no feature is in the 25% confidence contour or that almost all the features
are in the 75% confidence contour. Although this test was run for only four features in an
endoscopic image, the classical assumption of Gaussian distribution of the error on the
feature location was accepted for the rest of the thesis.
4.3 Analysis of the methods for the estimation of the camera
movement
Three main methods were presented in Chapter 3 ‘Literature Review: Possible Approaches
for Biopsy Site Re-localisation and Application for the Surveillance Examination of Barrett’s
Oesophagus’. These are the Least Median of Squares (LMedS), the RANdom SAmple
Consensus (RANSAC), and the Maximum A Posteriori SAmple Consensus (MAPSAC).
These methods are compared in this chapter for the specific case of endoscopic images and
the choice for MAPSAC is explained. Moreover, the results of an experiment are presented
and help decide how many feature matches are used to compute the fundamental matrix.
Table 4-1: Percentage of sample observations after feature detection with the LK tracker for various α:
α 25% 50% 75% 99%
Feature 1 0 41 96 100
Feature 2 34 34 78 100
Feature 3 22 43 61 100
% of sample
observations
Feature 4 0 35 95 100
Table 4-2: Percentage of sample observations after feature detection with the SIFT for various α:
α 25% 50% 75% 99%
Feature 1 0 44 93 100
Feature 2 20 40 67 100
Feature 3 4 51 86 100
% of sample
observations
Feature 4 4 57 86 100
103
4.3.1 Experiment: comparison of the estimations of the fundamental matrix with
LMedS, RANSAC, and MAPSAC
Torr and Murray (1997) and Torr (2002) investigated the performances of the LMedS,
RANSAC, and MAPSAC methods. They found that the MAPSAC outperformed the other
two methods as it searches for the fundamental matrix ( )TIF ,i which best fits the set of
detected inliers while the other two methods tend to return the solution which maximises the
number of inliers. The following experiment also illustrates why the MAPSAC is chosen for
the recovery of the epipolar geometry between endoscopic images.
4.3.1.1 Materials and method
The overall goal of the biopsy site re-localisation is the determination of accurate
epipolar lines. Therefore, this experiment compared the accuracy of the epipolar lines in the
target image T derived from the biopsy site location in the reference images Ii with the 3
methods LMedS, RANSAC, and MAPSAC. The epipolar geometry was recovered by
following the process presented in Fig. 3-22. One sequence from the same surveillance
examination of Barrett’s Oesophagus (BO) of a patient was processed. Image dimensions
were 339 pixels x 216 pixels and the sequence included 101 images. Images were
undistorted.
The first image was used to tune the parameters for feature detection and tracking
with the LK tracker. The number of inliers was counted which helped set the number m of
repeats of the random sampling consensus for the computation of the fundamental matrix.
The fundamental matrix was recovered between this first image and the last image of the
sequence with the LMedS in order to find an estimate of the residuals ( )ije I and of their
standard deviation σ. This standard deviation was used to set the threshold T for the
distinction of inliers and outliers with RANSAC and MAPSAC. Given the image dimensions,
the size of the buckets to group the features that are spatially close was set at 20 pixels.
For the 100 other pairs of images Ii ↔ T:
- The feature matches were found with the LK tracker
- The fundamental matrices were recovered, afterwards, with either the LMedS, or the
RANSAC, or the MAPSAC.
Table 4-3: RMS errors in pixels of the distances from the epipolar lines to the ground-truth of the
biopsy site in the target image T when the lines are computed with LMedS, RANSAC, or MAPSAC:
3. The features were grouped into buckets as described in section ‘3.5.3.4 Optimisation
of the computation of the fundamental matrix’. The bucket size took the values as
follows: 8, 10, 12, 15, 20, 25, 30, and 35 pixels. For each bucket size, the
fundamental matrix was computed as presented in Fig. 3-22 using MAPSAC and the
values of m and T found in steps 1 and 2. The size of the bucket was, therefore, used
to control the number of matches and to guarantee a good spread of the features over
the whole 3D surface observed with the endoscope camera.
4. A point was selected manually in the reference image and tracked manually in the
target image to be the biopsy site. This biopsy site corresponded to a salient blob well
identified in these images. The location of the biopsy site in the target image was the
ground truth and noted p0 = [p0x, p0y, 1] tr. After the computation of the fundamental
matrix for each bucket size, the epipolar line el = [elx, ely, elm] tr was derived in the
target image from the biopsy site location in the reference image. The accuracy of the
epipolar line was measured in pixels as the distance from the epipolar line to the
ground truth in the target image:
22
00accuracyyx
myyxx
elel
elpelpel
+
+⋅+⋅= .
(4-2)
4.3.2.2 Results and discussion
Approximately 30% outliers were detected manually during the counting process.
According to Table 3-1, only 54 tests of the MAPSAC could be applied to compute a correct
fundamental matrix. However, in order to improve the chance to compute an accurate
fundamental matrix, 3000 tests were run. When the number of matches was high which
corresponded to small bucket sizes, for example 63, 86, or 98, the epipolar line was accurate
(Table 4-4). The corresponding fundamental matrices were recovered from a high number of
inliers, respectively 40, 55, and 66. For fewer matches, the accuracy was getting worse. For
107
these cases, outliers corrupted the sets of matches and the fundamental matrix could have
been wrongly computed by detecting an outlier as an inlier. Thus, in practice for the rest of
this thesis, the LK tracker and the SIFT were tuned such that approximately 100 matches
could be detected.
4.4 Conclusion
This chapter discussed the method to recover accurate epipolar geometry for the pairs of
reference images Ii and target image T where the re-localised biopsy site needs to be
computed. In practice, sequences were processed offline after the endoscopy examination.
Before the examination and image acquisition, the endoscope camera was calibrated in order
to estimate the barrel distortions. Each acquired sequence was corrected for distortions after
the examination. A sequence refers to a series of images acquired around the detected biopsy
site. The last image of the sequence was used as the target image where the biopsy site
needed to be re-localised. The first image of the sequence was used to tune the parameters of
the algorithms for feature detection and matching and for epipolar geometry recovery.
For the tuning, features were preferably detected and matched with the LK tracker
applied to the first image of the sequence and to the target image. The LK tracker failed when
the endoscope camera moved too quickly and generated blurred images, or when air/water
bubbles obstructed the camera field of view. In these cases, SIFT features were detected and
matched, but the matching could generate a great number of outliers since it does not impose
a spatial constraint. Approximately 100 matches could be detected. The number of outliers
was counted visually in order to set the number of estimations necessary for the estimation of
the epipolar geometry. The epipolar geometry was recovered using a combination of the 7-
point algorithm with the LMedS. The 7-point algorithm was used since it imposes the
constraint that the determinant of ( )TIF ,i is null. The LMedS was used in order to estimate the
standard deviation of the Sampson’s residuals that the matches formed with the estimated
fundamental matrix. Features in the training image and in the target image were grouped in
buckets. For images whose dimensions were of the order of 200 pixels x 200 pixels, the
dimensions of the buckets were 20 pixels x 20 pixels.
The other endoscopic images were processed after parameter tuning. Features were
detected and matched using the LK tracker or SIFT. The 7-point algorithm was run in
combination with MAPSAC in order to estimate the epipolar geometries. The number of tests
for MAPSAC was set at 3000 in order to guarantee a great chance of computing accurate
fundamental matrix. Features in the images were grouped within buckets as for the tuning
step.
108
The next chapters demonstrate how accurate epipolar lines can be used to re-localise
the biopsy site accurately and precisely, and to determine the confidence of the re-
localisation. Moreover, another chapter presents a method to improve the accuracy of the
epipolar lines when they are recovered from SIFT matches.
109
Chapter 5 Re-localisation of Biopsy Sites during Endoscopy
Examinations
5.1 Introduction
Biopsy sites detected by optical biopsy during a surveillance examination of Barrett’s
Oesophagus (BO) need to be re-localised in the endoscopic images acquired during the same
examination in order to guide instruments to the biopsy sites for tissue excision, or for
treatment, or for a new tissue interrogation by optical biopsy.
The approach for the biopsy site re-localisation is based on the computation of the
mapping from one reference image Ii where the biopsy site location is known to the target
image T. In the previous chapters, it was shown that this mapping can be the fundamental
matrix ( )TIF ,i which is estimated during the recovery of the epipolar geometry formed by the
pair of images Ii ↔ T. In order to recover the epipolar geometry, features in both images
need to be detected and matched, which can be done with the Lucas Kanade (LK) tracker or
with the Scale Invariant Feature Transform (SIFT). Bad matches due to failures of the LK
tracker or the SIFT corrupt the accuracy of the epipolar geometry. They are, therefore,
detected when ( )TIF ,i is estimated. The Maximum A Posteriori Sample Consensus (MAPSAC)
finds a first estimate of ( )TIF ,i that fits best the set of inliers or good feature matches. The
estimated ( )TIF ,i is refined by non-linear optimisation over the whole set of detected inliers.
The fundamental matrix ( )TIF ,i transforms a point of the reference image Ii in a line
termed epipolar line in the target image T. This line indicates the locus of the image point in
T. In this chapter, a method is presented to re-localise the biopsy site in T using epipolar
lines. Either 2 epipolar lines or N > 2 epipolar lines are used for the re-localisation. This was
first published in Allain et al. (2009a.) and in Allain et al. (2010). The results of studies of the
influence on the re-localised biopsy site of the accuracy of the epipolar lines and of the angles
they subtend are discussed in this chapter.
5.2 Re-localisation principle
Epipolar lines derived from the biopsy site in the reference images Ii indicate a direction in
the target image T along which the biopsy site is. Geometric information from the epipolar
lines can be combined in order to determine the location of the biopsy site in T.
110
Biopsy site P at the tissue surface
Camera centre 1
Camera centre T
I1TI2
Camera centre 2
Re-localised biopsy site p
pp
e
e(I2)
elel
(I1)
(I1)
(I 1)
(I2)
(I 2)
Fig. 5-1: Method for biopsy site re-localisation with 2 reference images I1 and I2, a target image T and
two epipolar lines ( )1Iel and ( )2Iel : the biopsy site is seen under 2 different viewpoints of the
endoscope camera. This results in two distinct epipolar lines ( )1Iel and ( )2Iel that form an intersection
at the location of the biopsy site in T.
5.2.1 Re-localisation with 2 epipolar lines
During an endoscopy examination for the surveillance of BO, a biopsy site can be seen from
various viewpoints with an endoscope (Fig. 5-1). If a selected point of interest, for example a
biopsy site, is visible in two images I1 and I2, termed reference images, it can be re-localised
in a third subsequent image T, termed target image, when acquired for example after a small
movement of the endoscope camera.
Let P be the biopsy site location in the 3D space, and ( )1Ip and ( )2Ip be the locations
of the biopsy site in images I1 and I2. A possible approach would consist of computing P
from ( )1Ip and ( )2Ip and of projecting P back onto T. The two images I1 and I2 have to show
the physical surface under two very different viewpoints in order to estimate accurately the
3D position P (Hartley and Zisserman, 2004). As the endoscope camera has a limited motion
in the oesophagus, I1, I2, and T do not show significantly different viewpoints and therefore
this approach was not chosen for the re-localisation. The fundamental matrices ( )TIF ,1 and
( )TIF ,2 can be computed between images I1 and T and images I2 and T respectively. The axes
formed with camera centre 1 and camera centre T, and camera centre 2 and camera centre T,
have an intersection with the image plane T, which are termed the epipoles. Let ( )1Ie and ( )2Ie be the two epipoles of this configuration (Fig. 5-1). As presented in Chapter 3,
( )TIF ,1. ( )1Ip is a vector and defines the epipolar line ( )1Iel , which passes through the projection
of ( )1Ip onto T. By geometric construction, the intersection of the plane formed by camera
centre 1, camera centre T, and P with the image plane T is the epipolar line ( )1Iel (Fig. 5-1).
This plane passes through ( )1Ie . Thus, the epipolar line ( )1Iel passes through the projection of ( )1Ip onto T and ( )1Ie (Fig. 5-1). The epipolar line ( )2Iel can be defined similarly for
111
( )TIF ,2. ( )2Ip . The two epipolar lines indicate the locus of the possible images of ( )1Ip and
( )2Ip . As ( )1Ip and ( )2Ip correspond to the same biopsy site location in the 3D-space, the
intersection of ( )1Iel and ( )2Iel returns the location of the biopsy site in the target image T
(Allain et al., 2009a). A similar method was first proposed by Faugeras and Robert (1994).
However, they applied it to real world images representing the same object seen under
various viewpoints without the presence of instruments obstructing the camera field of view
and positioned at various locations in several images.
During endoscopy, the biopsy site location is defined and issued at that point imaged
with an optical biopsy miniprobe passed via the working channel of the endoscope. The
imaging axis is commonly arranged around the central axis of the endoscope. Thus, twisting
the head of the endoscope creates rotations and translations of the camera, while the probe
remains at its location at the tissue surface. This motion helps generate different views of the
biopsy site and different epipolar lines can be derived.
5.2.2 Limits of the re-localisation with 2 epipolar lines due to their uncertainty
The epipolar geometry formed by the pair of reference image Ii and target image T is
recovered from a set of matched features D = { ( )ijIp , pj}j=1..L with ( ) ( ) ( )[ ] tr
1,, iiijyjxj pp IIIp = , a
feature in Ii which matches pj = [pjx, pjy, 1] tr, a feature in T. The endoscopes that were used
for this thesis were mounted with a CCD camera. Noise appears in the images acquired with
a CCD camera and has an influence on the detection of features, on the determination of their
locations, and propagates to the fundamental matrix ( )TIF ,i and to the epipolar lines ( )iIel .
As stated in Chapter 4, Section 4.2.2 Experiment: study of the error for the
localisation of the features’), the features ( )ijIp and pj detected with the LK tracker or with the
SIFT are localised with an error. Many authors assumed that the errors of the localisations of
features ( )ijIp are identically and independently distributed with a Gaussian distribution and
they approximated the covariance matrix of these errors as ( )ijIp
Λ = σ12 where 12 is the 2x2
identity matrix (Luong and Faugeras, 1996; Csurka et al., 1997; Torr and Murray, 1997;
Zhang, 1998; Hartley and Zisserman, 2004). The experiments in Chapter 4 section ‘4.2.2
Experiment: study of the error for the localisation of the features’ confirmed that this is a
reasonable model for the images acquired with an endoscope and for the features extracted
with the LK tracker or with the SIFT.
112
matchedpoint
Ii
point
T
el i
enve
lope
FiT
Ii
( )TIF ,iel
(I i)j
Fig. 5-2: Uncertainty of the epipolar line: it corresponds to a confidence region which is a hyperbola.
The matched point is located on the epipolar line where the two arms narrow to a minimum.
A feature ( )ijIp in the reference image Ii is transformed by ( )TIF ,i
as an epipolar line
( )ijIel = ( )TIF ,i
. ( )ijIp = ( ) ( ) ( )[ ] tr
,, iiijmjyjx elelel III in T. Csurka et al. (1997), Zhang (1998), and
Hartley and Zisserman (2004) demonstrated that the errors on the localisation of the features ( )ijIp propagate to the fundamental matrix ( )TIF ,i
and to the epipolar line ( )ijIel . The uncertainty
or error of the localisation of an epipolar line can be represented visually as a confidence
region in which the line is likely to lie (Csurka et al., 1997; Zhang, 1998; Hartley and
Zisserman, 2004). This region represents the range of directions and positions that the line
may have with a given probability. Csurka et al. (1997), Zhang (1998), and Hartley and
Zisserman (2004) showed that this region is a hyperbola (Fig. 5-2). The two arms narrow to a
minimum at the point in T that is the image of the point in Ii from which the epipolar line was
derived (Fig. 5-2).
As the epipolar lines are determined with uncertainty, they may pass a few pixels
away from the true location of the biopsy site, and the intersection of 2 epipolar lines which
corresponds to the re-localised biopsy site may not be in coincidence with the true location of
the biopsy site. Furthermore, if the two epipolar lines subtend a small angle, their intersection
may lie far away from the true location of the biopsy site.
In order to improve accuracy and precision of the re-localisation, N epipolar lines
from N different views of the biopsy site may be used (Allain et al., 2010).
5.2.3 Extension of the re-localisation with N epipolar lines
The epipolar lines are derived with uncertainty and do not pass exactly through the biopsy
site in the target image T. Therefore, the re-localised biopsy site is computed by minimisation
of a criterion function taking into account information about the location of the epipolar lines.
113
The re-localised biopsy site p in the target image T must satisfy the condition of
triangulation with its match ( )iIp in the reference image Ii (Hartley, 1997). This condition
means that the two axes passing respectively through Camera centre i and ( )iIp , and through
Camera centre T and p, must meet at the position of the biopsy site P in the 3D space (Fig.
5-3). Longuet-Higgins (1981) demonstrated that the two axes corresponding to the matching
pair of points ( )iIp ↔ p will meet in space if and only if the algebraic residual that the points
form with the fundamental matrix is null:
( )( ) 0,
tr =⋅⋅ i
i
ITI pFp . (5-1)
As discussed in section ‘5.2.2 Limits of the re-localisation with 2 epipolar lines due to their
uncertainty’, the fundamental matrix ( )TIF ,i is determined with uncertainty. Given the biopsy
site location ( )iIp in the reference image Ii and its image p in the target image T, the
algebraic residual for the pair of points ( )iIp ↔ p is not null. Therefore, the re-localised
biopsy site p can be computed by minimisation of the algebraic residuals with the least
squares method (Bjorck, 1996):
( )( )( )∑
=
⋅⋅N
i
i1
2,
trmini
ITIp
pFp . (5-2)
Minimising the sum of squared algebraic residuals may not return a good estimation
of the re-localised biopsy site p:
− Torr (1995), Hartley and Zisserman (2004), and Hu et al. (2008b) stated that the
algebraic residual does not have a geometric meaning. A better measure to minimise is the
perpendicular distance from the feature matches to the fitting ellipse that corresponds to the
fundamental matrix.
Biopsy site P at the tissue surface
Camera centre i
Camera centre T
IiT
Re-localised biopsy site pF( ,T)
el
ep(Ii)
Ii
(I i)
(Ii)
Fig. 5-3: Condition of triangulation: the two axes passing respectively through Camera centre i and
( )iIp , and through Camera centre T and p, must meet at the position of the biopsy site P in the 3D
space.
114
− Luong and Faugeras (1996), Torr and Murray (1997), and Zhang (1998)
demonstrated that an optimal fundamental matrix ( )TIF ,i is obtained by dividing the algebraic
residuals with their standard deviation (equation (3-19)). This operation corresponds to a
minimisation of the perpendicular distance of the feature matches to the ellipse corresponding
to ( )TIF ,i. Thus, given the biopsy site ( )iIp in Ii, the re-localised biopsy site p may be searched
such that the perpendicular distance from the pair of matches ( )iIp ↔ p to the ellipse is
minimal. As it is assumed that the biopsy site location ( )iIp in image Ii is known, the standard
deviation of the algebraic residuals ( )( ) ( ) ( ) ( ) ( )iiiii
ieelelpelp myyxx
IIIIITI pFp =+⋅+⋅=⋅⋅ ,
tr
depends only on the uncertainty of the re-localised biopsy site p. By application of the
formula of error propagation, the standard deviation of the algebraic residuals is:
( )
( ) ( )
pp
II
I∂∂
⎥⎦
⎤⎢⎣
⎡∂∂
=ii
i
eepypxpy
pxpypx
e varcovcovvartr
2σ . (5-3)
where varpx and varpy are the variances of p for the components px and py, and cov is the
covariance of the components px and py. Equation (5-3) gives:
( )( ) ( ) ( ) ( )
pxpyyxpyypxxeiiii
i elelelel cov2varvar 222 ⋅⋅⋅+⋅+⋅= IIIIIσ . (5-4)
For features in the images, Luong and Faugeras (1996), Torr and Murray (1997), and Zhang
(1998) made the approximation that the variances of the feature point components along the x
and y directions of the image are equal and that the covariance terms are too small compared
to the variances. A similar approximation is done for the re-localised biopsy site p and the
standard deviation of the algebraic residual is proportional to the sum of the squares of the
first and second components of the epipolar line vector:
( )( ) ( )222 ii
i yxeelel II
I +∝σ . (5-5)
Thus, the algebraic residuals are divided by their standard deviation and the biopsy site is re-
localised as the point that minimises its perpendicular distances to the epipolar lines.
− Another approach to justify the minimisation of the algebraic residuals divided by
their standard deviation may be inspired from Luong and Faugeras (1996), Torr and Murray
(1997), and Zhang (1998) who introduced the concept of minimisation of the perpendicular
distance from an image feature to its epipolar line. Indeed, a feature has to be as close as
possible to its epipolar line. The perpendicular distance from the searched re-localised biopsy
site p = [px, py, 1]tr to its epipolar line ( ) ( ) ( ) ( )[ ] tr,, iiii
myx elelel IIIIel = is defined as (Fig. 5-4):
115
( )( )( ) ( ) ( )
( ) ( )
( )( )
( )( )( ) ( )
( )( ) 2
2,2
1,
,tr
22,
i
i
i
i
i
i
ii
iii
i
yx
myyxx
elel
elpelpeld
ITI
ITI
ITI
II
IIII
pFpF
pFp
elp
⋅+⋅
⋅⋅=
+
+⋅+⋅=
.
(5-6)
where ( ( )TIF ,i. ( )iIp )1 = ( )i
xel I is the first component of the vector ( )TIF ,i. ( )iIp . This distance is a
signed measure.
For these three reasons, the re-localised biopsy site p in the target image T is
computed such that it minimises by the linear least squares method the sum of all of its
distances from the epipolar lines { ( )iIel }i=1..N:
( ) ( )( ) ( )( )∑∑==
===N
i
N
ii
ii dCCC1
2
1
2min ,min,minmin I
p
I
ppelpelpp .
(5-7)
el(If)
el (In)
el(Iw)
el(I f)
el (In )
el(Iw)
Fig. 5-4: The definition of the re-localised biopsy site p: it is defined such that it minimises the sum of
the perpendicular distances to the epipolar lines.
inputs
output
Image I1Image T
Image IiImage T
Image INImage T
Feature detection and matching
Fundamental matrix computation
- If N = 2, p is at the intersection of the two epipolar lines- If N > 2, solve for p such that it minimises the sum of its perpendicular distances to the epipolar lines
Re-localised biopsy site p
… …
Correction of the endoscopic image distortionspreliminary step
Fig. 5-5: Framework for the biopsy site re-localisation in the target image T.
116
C is the cost function to minimise. This solution makes the assumption that no epipolar line
in { ( )iIel }i=1..N is an outlier. This assumption is valid since the epipolar geometries formed for
each pair of reference and target images Ii ↔ T are computed initially from sets of inliers
among the matches. Any epipolar line derived from these geometries is, therefore, accurate.
For future developments, in case an epipolar line is an outlier, equation (5-7) can be solved
using a random sampling consensus which consists of selecting the epipolar lines that
minimise Cmin.
The re-localisation of the biopsy site is integrated into a whole algorithm which
firstly recovers the epipolar geometries for each pair of endoscopic images Ii ↔ T (Fig. 5-5).
It is assumed that these images are corrected for distortions. As discussed in Chapter 4
section ‘4.2 Analysis of features ’, features can be detected and matched either with the Lucas
Kanade tracker when the camera movement is smooth or SIFT when the movement is quick
or when air/water bubbles obstruct the camera field of view. The fundamental matrix ( )TIF ,i is
estimated with the Maximum A Posteriori Sample Consensus. Once all the fundamental
matrices are estimated, the biopsy site is re-localised either with 2 epipolar lines or N epipolar
lines.
5.3 Experiment 1: study by simulations of the re-localisation
precision and bias with the locations of the matches perturbed by a
Gaussian noise and with the presence of outliers
The epipolar lines are computed with uncertainty due to the error of the localisation of the
features in the endoscopic images (Csurka et al., 1997; Zhang, 1998; Hartley and Zisserman,
2004). This error or noise is assumed to be independently and identically Gaussianly
distributed. The lines are also determined with uncertainty since outliers corrupt the matches
(Hartley and Zisserman, 2004). The uncertainty of the epipolar lines propagates to the re-
localised biopsy site.
5.3.1 Method
An experiment was performed in order to study the impact of Gaussian noise that affects the
features on the re-localised biopsy site. It was based on simulations in order to have control
of the standard deviation of the Gaussian noise and the percentage of outliers. Simulations
were run on a virtual endoscopic scene.
117
0.3 units= 0.75 cm
b)C
amer
a z-
axis
Camera x-axis
Cameray-axis
3D pointsCameraBiopsy site
Cam
era
z-ax
is
Camera x-axis Camera y-axis
3D pointsCameraBiopsy site
d)
x-axis of the image plane
y-ax
is o
f the
imag
e pl
ane
Fig. 5-6: Creation of a virtual endoscopy scene and generation of the images: a) a virtual 3D surface
mimics a tubular organ; b) and c) 3D points are extracted from the surface and a camera is simulated;
d) the 3D points are projected onto the image plane of the simulated camera moving along the surface.
118
The virtual scene reproduced a section of a hollow organ, such as an oesophagus,
which can be described by a cylinder of diameter 2.5cm (Fig. 5-6 a)). A virtual camera
representing an endoscope camera was placed within this virtual scene at a distance of
approximately 0.75cm from the tissue surface and had an inclination of 30° in order to
observe only an extent of the cylinder. The inclination was set at 30° since this was the angle
that could be assumed from the endoscopic images acquired on real patients. Moreover, the
camera was placed at a distance of 0.75cm from the surface since sequences extracted from
real endoscopy examinations showed that the camera is commonly at this distance using the
miniprobe as a scale within the image.
Three-D points were extracted at regular distances from this cylinder in order to
represent the features of the oesophagus and perturbed with an isotropic Gaussian noise in the
x, y, and z directions in order to obtain an irregular distribution of points in the 3D space
(Fig. 5-6 b) and c)). During an optical biopsy and a biopsy excision with forceps in real
endoscopic sequences in Barrett’s Oesophagus (BO), the camera usually observes a small
region of the oesophagus which can be its bottom half, for example. Features detected in this
region correspond to tissue points on an extent of 2cm x 2cm according to real endoscopic
images. Thus, only half of the 3D points were used for the simulations, and the 3D-points
were extracted only along a 2.5cm height of the cylinder. In Chapter 4 section ‘4.3.2
Experiment: number of matches for the computation of the fundamental matrix’, it was
mentioned that approximately 100 feature matches would be used for the recovery of the
epipolar geometries in real endoscopic images of 300 pixels x 300 pixels. In these
simulations, 200 3D points were created such that various samples of 100 features could be
used for the recovery of the epipolar geometries.
The camera was translated and rotated within this scene in a neighbourhood of a
1.5cm diameter centred on the initial camera position. During real endoscopy examinations of
the oesophagus, it was possible to twist the endoscope up to 45° which is why the camera was
rotated up to this angle. For each camera pose, the 3D points were projected onto the image
plane Ii of the camera (Fig. 5-6 d)).
A 3D point was selected randomly among the 200 3D points to be the biopsy site
(Fig. 5-6 b) and c)). This point had, therefore, a known location ( )iIp in the reference images
Ii. The last image was considered as the target image T where the biopsy site needed to be re-
localised. The ground-truth position of this site was defined as p0.
For each pair of images Ii ↔ T, α% of the features in the target image T were
displaced anywhere within its field of view to create outliers. A Gaussian noise of standard
deviation varying from 0.1 pixel to 4 pixels was added to the inliers. These figures were used
119
by Hartley (1997) and Csurka et al. (1997) for images or simulated images that had a field of
view of hundreds of pixels. The re-localised biopsy site was computed for each Gaussian
noise. The impact of the noise and of the outliers on the re-localised biopsy site was studied
by estimating the root mean squared error (RMS) of the re-localisation for each standard
deviation of the Gaussian noise. The definition of the error was inspired from West et al.
(1999) who measured the Euclidean distance between the ground-truth of a point and its
estimate. In these simulations, the error was defined as the Euclidean distance of the re-
localised biopsy site to the ground-truth p0. For each standard deviation of the Gaussian
noise, the re-localised biopsy site was computed 1000 times and the RMS error was defined
as (West et al., 1999):
( )∑=
−⋅=1000
1
2
21-10001RMS
k
k0pp .
(5-8)
where p(k) is the kth estimate of the biopsy site. By development:
( )
( ) 2
2mean
1000
1
2
2mean
1000
1
2
2
2
1-10001000
1-10001
1-10001RMS
0
0
pppp
pp
−⋅+−⋅=
−⋅=
∑
∑
=
=
k
k
k
k
.
(5-9)
The first term of the sum is the experimental precision:
( )∑=
−⋅=1000
1
2
2mean1-10001precision
k
k pp . (5-10)
where pmean = [xmean, ymean] tr is the mean of the biopsy sites p(k). The second term is the bias:
2
2mean2
1-10001000 bias 0pp −⋅= .
(5-11)
The generation of a virtual 3D scene and the corresponding analysis were repeated
twice:
- the first study was performed with 30% outliers among the matches which corresponds to
the proportion counted manually in the last experiment in Chapter 4 section ‘4.3.2
Experiment: number of matches for the computation of the fundamental matrix’. The re-
localised biopsy site was computed with 2 epipolar lines, first, and with 50 epipolar lines,
secondly. The number 50 was chosen since for some real endoscopic sequences a similar
order of images was used for biopsy site re-localisation.
- the second study was performed with 20% outliers among the matches. The re-localised
biopsy site was computed with 2 epipolar lines, first, and with 10 epipolar lines, secondly,
since for other real endoscopic sequences approximately 5 to 15 images were used for the
biopsy site re-localisation.
120
The images generated from the projections of the 3D surface onto the camera image
plane for each viewpoint were free of noise and outliers. For this experiment, each pair of
images Ii ↔ T was modified as follows:
1. 100 matches were selected randomly among the 200 matches since this number
corresponds approximately to that used for the estimation of the epipolar geometry
on patient data.
2. For 1000 repeats:
a. In image T select randomly α% of the matches and move them by a random
displacement within the image in order to create outliers. Indeed, a feature in
T forms an outlier with a feature in Ii if the two features do not correspond to
the same point at the surface of the object.
b. Apply a Gaussian noise sample of standard deviation σ to the features in T.
The experiments in section ‘4.2.2 Experiment: study of the error for the
localisation of the features’ demonstrated that these are strong assumptions.
Nevertheless, these simulations just aimed to generate uncertainty of the
epipolar lines in order to study the variations of the location of the re-
localised biopsy site p. Thus, the noise could be modelled freely.
c. Apply a noise sample of standard deviation σ to the features in the images Ii.
The biopsy site ( )iIp was perturbed by the noise as well.
d. Recover the epipolar geometry formed by the pair of images Ii ↔ T. Derive
the epipolar lines in T from the biopsy site ( )iIp in Ii.
Thus, for each pair of images Ii ↔ T, there were 1000 estimations of the epipolar
line. Steps 1. and 2. were repeated for a standard deviation of the Gaussian noise σ varying
from 0.1 pixels to 4 pixels.
3. For each value σ of the standard deviation of the noise:
a. For each repeat, re-localise the biopsy site either with 2 epipolar lines or with
N epipolar lines. Two sets of 1000 re-localised biopsy sites p(k) = [px(k), py
(k)] tr
in the target image T were computed: there was one set for the re-localisation
with 2 epipolar lines and one set for the re-localisation with N epipolar lines.
b. For each set of re-localised biopsy sites, the experimental precision and bias
of the re-localisation method were computed (equations (5-10) and (5-11)).
The re-localised biopsy site p was estimated 1000 times. It was assumed that the re-
localised biopsy sites computed with N epipolar lines were close to the true location of p and
there was no need to detect outliers for the computation of the precision and of the bias.
However, the re-localisation with 2 epipolar lines could return biopsy sites that were very far
121
from p. Indeed, p can be anywhere within the overlap of the two envelopes of the epipolar
lines (Fig. 5-7). The true location of the biopsy site or ground truth was located where the
arms of the hyperbolas narrow to a minimum. The simulations aimed to study the variations
of the precision and bias of the re-localised biopsy site with the noise on the matches.
Therefore, only the re-localised biopsy sites located around the ground truth were used for
this study. Fifty percent of these sites were used as inliers and it was assumed that their
distribution around the ground truth was Gaussian. These inliers were detected by the
minimum volume ellipsoid algorithm (Rousseeuw and Leroy, 1987). This algorithm is as
follows:
Draw a subsample of 3 different re-localised biopsy sites p(k):
1. Compute the mean ps_mean of these 3 sites and their covariance matrix Cs_mean.
2. Compute the median m2 of the Mahalanobis distances of each re-localised biopsy site
with Cs_mean: ( ) ( ) tr)(1_
)(2meanmeansmean ppCpp −⋅⋅−= − kkmedm .
3. The volume of the resulting ellipsoid is proportional to (det(m2C))1/2.
The subsample drawing is repeated. The mean and the covariance of the distribution of the
re-localised biopsy sites correspond to those that return the minimum volume over the
drawings.
Envelop
e 2Envelope 1
Envelop
e 2
Env
elop
e 1
a) b) Fig. 5-7: Intersection of two epipolar lines and locus of the possible re-localised biopsy sites: the
epipolar lines are characterised by their envelope whose thickness represents the confidence level
(50% confidence, for example). The lines can be anywhere within this envelope with the
corresponding probability. The re-localised biopsy site is in the region corresponding to the overlap of
the two envelopes. The overlap depends on the angle that the two epipolar lines subtend: a) the two
epipolar lines subtend a large angle; b) the two epipolar lines subtend a small angle.
122
5.3.2 Results
The matches between the images Ii and the target image T were contaminated with a
Gaussian noise and a certain percentage of matches was displaced in order to create outliers.
When the standard deviation of the noise increases, the features may lie far from their true
location. The resulting feature matches can be outliers for the epipolar geometry. Thus, the
epipolar geometry is recovered from less inliers. Luong and Faugeras (1996), Hartley and
Zisserman (2004), and Hu et al. (2008b) showed that the epipolar geometry and, therefore,
the epipolar lines are recovered with less precision when less inliers are used. Therefore, the
epipolar lines may be further away from the location of the ground-truth and their orientation
angle determined by the envelope is larger (Csurka, 1997; Hartley and Zisserman, 2004). In
order to illustrate this property of inaccuracy, an example of the behaviour of an epipolar line
( ) ( ) ( ) ( )[ ] tr,, iiii
myx elelel IIIIel = derived from the biopsy site ( )iIp for a pair of images Ii and T with
the increasing standard deviation of the Gaussian noise is shown in Fig. 5-8. As the epipolar
geometry from which this epipolar line was derived was estimated 1000 times for each
standard deviation of the noise, the distance of this epipolar line from the ground-truth was
computed 1000 times. The accuracy of the line was defined as the root mean squared error of
the perpendicular distance of each estimation of this line to the ground-truth p0 = [x0, y0, 1] tr
(West et al., 1999):
( )
( )( ) ( )( ) ( )( )
( )( ) ( )( )∑= ⎟⎟
⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
+
+⋅+⋅=
1000
1
2
22
00
10001accuracy
kk
yk
x
km
ky
kx
ii
iii
i
elel
elyelxelII
III
el I . (5-12)
In practice, it could happen that some estimates of the epipolar line out of the 1000 estimates
were wrong. These were detected using the LMedS where the error was the distance of the
estimate of the epipolar line to the ground truth of the biopsy site. When the standard
deviation of the Gaussian noise increased, the values of the accuracy of the epipolar line
increased (Fig. 5-8).
The biopsy site p is re-localised in the target image T either as the intersection of two
epipolar lines or by minimisation of a cost function C which is the sum of the perpendicular
distances from p to the epipolar lines (equation (5-7)).
For a re-localisation of the biopsy site with 2 epipolar lines, as the lines could be
further away from the ground-truth when the noise increased, their intersection could lie
further away from the ground-truth (Fig. 5-9 a)). Fifty percent of these re-localised biopsy
sites were used for the estimation of the precision (Fig. 5-9 b)). The disparity of the re-
localised biopsy sites was greater when the noise increased. Thus, for the simulations, when
the standard deviation of the Gaussian noise on the features increased, the re-localised biopsy
123
sites formed a larger point cloud around the ground truth and the value of the re-localisation
precision was higher. As the point cloud was larger, the mean of the re-localised biopsy sites
might be further away from the ground-truth p0 and the re-localisation bias might be higher
(Fig. 5-10 and Fig. 5-11).
For a re-localisation with N > 2 epipolar lines, the biopsy sites were close to the
ground-truth p0 (Fig. 5-9 c)). The value of the minimum Cmin of the cost function defined in
equation (5-7) increased as the epipolar lines were further away from the ground-truth (Fig.
5-12). As the biopsy site was estimated 1000 times, 1000 costs Cmin were determined and an
average of the costs could be computed. This cost tended to increase with an increasing noise.
Therefore, the re-localised biopsy sites were computed with less precision (Fig. 5-13 and Fig.
5-14). Their bias increased as well. For comparison, the re-localised biopsy site was also
computed by minimisation of the algebraic distance (equation (5-2)). The resulting biopsy
site was less precise than that obtained by minimisation of the perpendicular distances. The
biases were similar. Finally, precisions and biases had smaller values for re-localisation with
N > 2 epipolar lines.
This experiment consisted of adding noise and outliers among the feature matches in
order to create uncertainty of the epipolar lines. The location of the re-localised biopsy site
was less precise and more biased when the noise increased and when the epipolar lines were
less accurate. The resulting re-localised biopsy sites were less biased and more precise when
they were estimated from N > 2 epipolar lines than when they were estimated from 2 epipolar
The comparison of the experimental and analytical uncertainties was performed on patient
data. The results are presented in Fig. 6-3.
For N > 2 epipolar lines, the ellipses had similar orientations and they did not differ
much in size. The corresponding KL divergences were small and of similar orders. The
greater axis of the ellipses was smaller than 2mm (Fig. 6-3). The ellipses indicate the region
where the biopsy site is likely to lie. Because of approximations during the derivation of the
analytical uncertainties, the analytical and experimental ellipses did not overlap exactly.
Nevertheless, both ellipses define a region where the biopsy site needs to be taken. Large
ellipses in one direction indicate that the epipolar lines are too much in coincidence. If the
epipolar lines subtend sufficiently large angles, the confidence ellipses will have areas less
than 2mm which is less than half the size of the typical endoscopic biopsy forceps. In this
case, if the confidence ellipses are displayed to the operator during tissue sampling, there is a
reassurance offered that the desired tissue has indeed been sampled.
6.5 Conclusion
The display of a confidence interval around a previously marked biopsy site of interest is
extremely useful for the practicing clinician as the display of both point and regional data can
reassure that the desired sample has actually been taken in traditional forceps terms or
optically interrogated in optical biopsy terms. This confidence region is drawn from the
covariance matrix of the re-localised biopsy site that can be computed experimentally or
analytically. The analytical computation requires a single estimation of the re-localised
biopsy site and of its uncertainty. However, it makes a series of assumptions about the noise
144
on the matches and the independence of the epipolar lines. The experimental computation
does not make any assumption. However, its main disadvantage is the number of iterations
this method requires for the computation of the re-localised biopsy site and of the uncertainty.
A whole framework for biopsy site re-localisation has been proposed in Chapters 3, 4, 5, and
6. The next chapter presents the results of this method on patient data. Chapter 8 focuses on
the extension of this framework for cases where the endoscopist moves the camera too
quickly or where air/water bubbles obstruct the camera field of view.
Target image Analytical and experimental ellipses (zoom around the biopsy site)
DKL1 DKL2
0.14 0.61
0.15 1.92
0.10 0.17
2mm
2mm
2mm
Biopsy site
Biopsy site
Biopsy site
Fig. 6-3: Analytical and experimental 99% confidence ellipses for N > 2 epipolar lines in the target
image: each row corresponds to a sequence acquired on a patient and presents first the target image
with the location of the biopsy site, secondly the analytical ellipse (green) and the experimental ellipse
(blue), and finally the KL divergences.
145
Chapter 7 Test of the Re-localisation Methods on Phantom
and Patient Data
7.1 Introduction
The previous chapters presented a method to re-localise a biopsy site in a target endoscopic
image using epipolar geometry. The re-localised biopsy site can be determined either as the
point at the intersection of two epipolar lines or as the point that minimises the sum of
squared perpendicular distances to N > 2 epipolar lines. The re-localisation uncertainty can be
computed in order to draw a confidence ellipse around the re-localised biopsy site. This
uncertainty can be computed experimentally or analytically. If the re-localised biopsy site is
accurate, the experimental and analytical uncertainties are similar and the drawn ellipses
overlap well. If the re-localised biopsy site is inaccurate, the ellipse drawn from the analytical
uncertainty is large due to the inaccuracy of the epipolar lines used for the re-localisation.
This warns the endoscopist that the re-localisation may be inaccurate.
This chapter aims to test the re-localisation techniques and the analytical estimation
of the uncertainty. Tests were done on patient data acquired during a surveillance
examination of Barrett’s Oesophagus in the department of Gastroenterology in University
College London Hospitals. Data were collected with the ethical approval reference
08/H0808/08. Tests were also done on a rigid tube phantom. A frame of an endoscopic
sequence acquired on a patient was used to generate a texture that was stuck inside the tube in
order to reproduce an oesophagus. These tests aimed to assess the accuracy and the precision
of the re-localisation. The influence of the endoscope camera motion on the re-localisation
was studied. The data were processed with SIFT features and with LK features in order to
study the difference of accuracies. Finally, these test results demonstrate the importance of
the spatial distribution of the features over the physical surface in order to compute an
accurately re-localised biopsy site.
7.2 Method
A physical phantom was built from a rigid tube whose diameter was 2.5cm, which
corresponds approximately to the diameter of an oesophagus. A synthetic texture was rolled
into the tube. As it had to be realistic, it was generated from an endoscopic image acquired
during an endoscopic monitoring of Barrett’s Oesophagus. The image was repeated several
146
times to form a whole texture with the dimensions 8cm x 13cm (Fig. 7-1 a)). Some points and
lines were added to the texture to give gold standard positions for the biopsy sites and scales
for the determination of the field of view of the endoscopic images.
For the patient data, eight sequences were acquired on 4 patients during routine
procedures for endoscopic surveillance of Barrett’s Oesophagus. Images were acquired with
white-light endoscopes or with Narrow Band Imaging (NBI) endoscopes. The NBI
endoscopes differ from the white-light ones since they narrow the spectrum of wavelengths in
the red-channel image such that the resulting red-green-blue image highlights the superficial
vessels of the oesophagus visible in the green and blue channel images. Argon Plasma
Coagulation (APC) was used to create marks of a diameter of approximately 3mm or gold
standards of the biopsy site at the tissue surface (Fig. 7-1 b))
.
Fig. 7-1: Endoscopic image of a) the phantom with a white-light endoscope: the blue point corresponds
to the ground truth of the biopsy site and b) a patient’s oesophagus with an NBI endoscope: an APC
burn indicates the ground truth of the biopsy site.
Fig. 7-2: Two gastroscopic sequences acquired with an NBI endoscope: a) to d) are images extracted
from a sequence where the biopsy site (APC mark) was observed under various viewpoints; e) to h) are
from a sequence with a miniprobe in the camera FOV.
147
The re-localisation was tested on sequences showing a biopsy site (the mark
corresponding to the gold standard position) from various viewpoints. The last image of the
sequences was selected as the target image for re-localisation. The endoscope camera was
moved freely backward, forward, and sideways. It was rotated as well. This first type of
acquisition returned a whole sequence of images with the mark only (Fig. 7-2 a) to d)). For a
second type of acquisition, an optical biopsy miniprobe was passed via the working channel
of the endoscope. The miniprobe was placed in contact with the tissue and was maintained at
this location while the endoscope camera rotated and translated backwards (Fig. 7-2 e) to h)).
The miniprobe was removed and the endoscopic camera could move freely while keeping the
biopsy site in the FOV.
For each sequence, the biopsy site position was tracked manually from I1 to T for the
re-localisation in T which returned the gold standard in T. Matches for each pair of images Ii
and T were found with the LK tracker and also with SIFT. This allowed the performances of
the re-localisation with epipolar lines derived from various feature detectors and matching
techniques to be compared. Features in the blue points for the phantom and in the APC mark
for the patients could be detected with the LK tracker and with SIFT. These features were
removed to avoid influencing the accuracy of the fundamental matrix computation. A region
of interest around the mark in T was used as a mask. Given the matches, the epipolar
geometries were recovered with MAPSAC for each pair of images Ii and T. The epipolar
lines were derived from each biopsy site position ( )iIp , and the re-localisation method was
applied with N > 2 epipolar lines. For comparison, the re-localised biopsy site was also
computed as the intersection of 2 epipolar lines subtending a large angle. The precision of the
re-localised biopsy sites was computed analytically for the sequences where features were
detected and matched using the LK tracker. The accuracy was computed in the 2D target
image as the Euclidean distance between the re-localised biopsy site and the gold standard
position. This distance and the values of the precision were small enough to approximate the
corresponding tissue extent as a plane. Under this assumption, the precision and the accuracy
computed in the 2D image are representative of the true measures in 3D. The marks or the
size of the miniprobe provided a scale in the target image T for the conversion from pixels to
millimetres.
The influence of the camera movement on the re-localised biopsy site was studied.
This last test aimed to show that a movement of the endoscope camera along its optical axis
returns inaccurate re-localisations. A sequence was acquired on the phantom with the white-
light endoscope camera. An optical miniprobe was placed in contact with the texture surface
and was maintained at the same location while the endoscope was moving. The camera
148
moved along its optical axis and was such that the resulting epipolar lines should be almost
coincident. Four reference images were used to re-localise the biopsy site in the target image.
The resulting re-localised biopsy site was computed.
7.3 Results
The re-localisation results are presented in Table 7-1, Table 7-2, and Table 7-3. Biopsy sites
were re-localised in endoscopic images with an FOV of approximately 3cm x 3cm with
analytical precisions and accuracies of:
− 2.5mm or better with 2 epipolar lines after matching features with the LK tracker.
− 0.45mm or better with N > 2 epipolar lines after matching features with the LK tracker.
− 0.92mm or better with N > 2 epipolar lines after matching features with the SIFT.
The best precisions and accuracies were obtained for the sequences without a miniprobe in
the FOV.
The results presented in Table 7-1 demonstrate that a re-localisation with 2 epipolar
lines is less accurate than with N > 2 epipolar lines. The main reason is the uncertainty of the
epipolar lines. Their intersection may be, therefore, far from the true biopsy site. The use of N
> 2 epipolar lines allows the space in T, within which the re-localised biopsy site may be, to
be constrained.
Table 7-1: Results of the biopsy site re-localisation with 2 epipolar lines: for each sequence, features
were detected and matched using the LK tracker.
Sequence Miniprobe? FOV
(pixels ; cm)
Accuracy: measured distance from
the re-localised biopsy site to the gold
standard
(pixels ; mm)
Phantom Sequence 1 Yes (283 x 180 ; 3 x 4) (3.1 ; 0.20)
Phantom Sequence 2 Yes (384 x 288 ; 4 x 6) (11.9 ; 0.44)
Patient 1 Sequence 1 Yes (339 x 216 ; 2 x 2) (9.6 ; 0.96)
Patient 1 Sequence 2 No (339 x 216 ; 5 x 2) (3.3 ; 0.67)
Patient 2 Sequence 1 No (283 x 180 ; 3 x 2) (18.6 ; 1.01)
Patient 3 Sequence 1 No (283 x 180 ; 6 x 2) (19.2 ; 1.92)
Patient 3 Sequence 2 Yes (283 x 180 ; 5 x 1) (6.9 ; 0.46)
Patient 4 Sequence 1 Yes (376 x 280 ; 1 x 1) (15.5 ; 0.61)
Patient 4 Sequence 2 Yes (376 x 280 ; 7 x 2) (21.5 ; 2.15)
149
The use of N > 2 epipolar lines can make a significant difference if the lines subtend
a sufficiently large angle which depends on the endoscope camera movement. For example,
when the camera moved only along the central axis of the endoscope in the phantom study,
the locations of the epipoles computed in the target image T from each reference image
varied only by a few pixels (Fig. 7-3 a), b), and c)). The resulting epipolar lines used for the
biopsy site re-localisation were almost all coincident and the re-localised biopsy site was far
from the true biopsy site (Fig. 7-3 d)).
During a gastroscopy examination including an optical biopsy, a practical way to
generate multiple viewpoints of the biopsy site while the optical miniprobe is still in contact
with the tissue consists of twisting the camera around the central axis of the endoscope. The
locations of the epipoles computed in T vary around the centre of T as seen in the example of
Fig. 7-4. As in practice the true biopsy site is also located near the centre of the image, the
distance between the epipoles and the biopsy site is short. The resulting epipolar lines
subtend large angles and differ well from each other. This camera movement guarantees,
therefore, a variety of directions of the epipolar lines necessary for accurate and precise re-
localisations.
Table 7-2: Results of the biopsy site re-localisation with several epipolar lines: for each sequence,
features were detected and matched using the LK tracker.
Sequence Miniprobe? FOV
(pixels ;
cm)
Number
of
epipolar
lines
Analytical
precision
(pixels ;
mm)
Accuracy: measured distance from the
re-localised biopsy site to the gold
standard
(pixels ; mm)
Phantom
Sequence 1 Yes
(283 x 180
; 3 x 4) 41 (1.0 ; 0.07) (2 ; 0.14)
Phantom
Sequence 2 Yes
(384 x 288
; 4 x 6) 47 (1.0 ; 0.04) (3.5 ; 0.13)
Patient 1
Sequence 1 Yes
(339 x 216
; 2 x 2) 26 (1.5 ; 0.15) (5.2 ; 0.52)
Patient 1
Sequence 2 No
(339 x 216
; 5 x 2) 178 (0.3 ; 0.05) (0.4 ; 0.08)
Patient 2
Sequence 1 No
(283 x 180
; 3 x 2) 7 (1.0 ; 0.05) (1.3 ; 0.07)
Patient 3
Sequence 1 No
(283 x 180
; 6 x 2) 14 (0.6 ; 0.06) (0.9 ; 0.09)
Patient 3
Sequence 2 Yes
(283 x 180
; 5 x 1) 19 (3.0 ; 0.20) (6.3 ; 0.42)
Patient 4
Sequence 1 Yes
(376 x 280
; 1 x 1) 11 (2.6 ; 0.10) (4.1 ; 0.16)
Patient 4
Sequence 2 Yes
(376 x 280
; 7 x 2) 8 (4.4 ; 0.44) (4.5 ; 0.45)
150 Table 7-3: Results of the biopsy site re-localisation with several epipolar lines: for each sequence,
features were detected and matched using SIFT.
Sequence Miniprobe? FOV
(pixels ;
cm)
Number
of
epipolar
lines
Accuracy: measured distance from the
re-localised biopsy site to the gold
standard
(pixels ; mm)
Phantom
Sequence 1 Yes
(283 x 180 ;
3 x 4) 41 (8.2 ; 0.58)
Phantom
Sequence 2 Yes
(384 x 288 ;
4 x 6) 47 (16.2 ; 0.65)
Patient 1
Sequence 1 Yes
(339 x 216 ;
2 x 2) 26 (3.1 ; 0.31)
Patient 1
Sequence 2 No
(339 x 216 ;
5 x 2) 178 (1.8 ; 0.36)
Patient 2
Sequence 1 No
(283 x 180 ;
3 x 2) 7 (7.2 ; 0.39)
Patient 3
Sequence 1 No
(283 x 180 ;
6 x 2) 14 (1.9 ; 0.24)
Patient 3
Sequence 2 Yes
(283 x 180 ;
5 x 1) 19 (6.0 ; 0.40)
Patient 4
Sequence 1 Yes
(376 x 280 ;
1 x 1) 11 (10.7 ; 0.42)
Patient 4
Sequence 2 Yes
(376 x 280 ;
7 x 2) 8 (9.2 ; 0.92)
a) b)
c) d) Fig. 7-3: Failure case of the re-localisation method: the camera moved along the endoscope central
axis. In a), b), and c), the epipole derived from 3 reference images is displayed in the target image. It is
the intersection of the epipolar lines (red) derived from each feature of the reference image. The
epipole does not move much. This results in d): the bundle of epipolar lines (blue lines) used for the re-
localisation of the biopsy site subtend very small angles. The yellow point indicates the ground-truth
position of the biopsy site. The red point indicates the re-localised biopsy site.
151
a) b)
c) d) Fig. 7-4: Movement of the epipole in the target image T: figures a), b), c), and d) show the position of
the epipole in T derived from a series of consecutive reference images Ii. The epipole moves towards
the centre of the image T which is the result of the rotation of the endoscope tip.
Fig. 7-5: Examples of re-localised biopsy sites: for each sequence, the four images are the target image
of each sequence with the features displayed (green dots) or the epipolar lines (blue lines) derived from
the previous images or the confidence region (green ellipse). These are the results obtained for feature
detection and matching with the LK tracker. For the two first sequences, the fourth image is an
enlargement around the confidence region.
152
The epipolar geometries of each pair of reference and target images were estimated
from sets of features detected with the LK tracker or with SIFT. In practice, the LK tracker
could track on average 100 features in the endoscopic images processed for this experiment
and it returned approximately 30% outliers. The SIFT usually returned 70 matches with 50%
outliers. The LK tracker has the advantage of tracking features through successive images
which constrains spatially the search for the matching features and guarantees, therefore, high
proportion and number of inliers. The SIFT does not constrain spatially the feature matching
and may return, therefore, more outliers than the LK tracker. Epipolar geometry is more
accurate when recovered from high proportion and number of inliers and the epipolar lines
derived from this geometry are, therefore, more accurate (Luong and Faugeras, 1996; Hartley
and Zisserman, 2004). As the epipolar lines used for the re-localisation could be more
accurate for the case of LK features, the re-localised biopsy sites were more accurate.
The more precise and accurate results were obtained for Patient 1 Sequence 2, Patient
2 Sequence 1, and Patient 3 Sequence 1 (Table 7-2 and Table 7-3). These sequences were
acquired without a miniprobe in the FOV. Therefore, the detected features were well
distributed over the oesophagus surface observed with the endoscope (Fig. 7-5 a)). They were
detected at various depths and along the curvature of the oesophagus. Such a good
distribution guarantees accurate epipolar lines as stated by Luong and Faugeras (1996).
Finally, the endoscope could move freely with wide translations and rotations. The resulting
epipolar lines subtended large angles (Fig. 7-5 a)) unlike those for the sequences acquired
with a miniprobe in the FOV (Fig. 7-5 b) and c)). The variety of directions is taken into
account in the uncertainty matrix Λp of the re-localised biopsy site for the cases where the
features have been detected and matched with the LK tracker. Large differences between the
line directions contribute to small ellipses (Fig. 7-5 a)) and, therefore, to small values of the
analytical precision. Also, accurate epipolar lines contributed to the high precision of the re-
localisation. Thus, the combination of a good distribution of the features over the observed
surface with a variety of directions of the epipolar lines helped reach high re-localisation
precisions for these 3 sequences.
For some of the sequences where a miniprobe was introduced in the endoscope FOV,
the values of analytical precision and accuracy were larger than for the other sequences. For
Patient 3 Sequence 2 and Patient 4 Sequence 2, for example, the FOV of the target image T
was bigger. The size of a pixel in the neighbourhood of the re-localised biopsy site and of the
ground truth was, therefore, larger for these sequences. The resulting accuracy was large.
Also, the re-localised biopsy site was computed from a set of epipolar lines that did not
153
subtend large angles (Fig. 7-5 c) and d)) and the resulting precision was, therefore, large for
the cases where the features were detected and matched with the LK tracker.
7.4 Conclusion
This chapter presented the results of the re-localisation method for endoscopic images
acquired on phantom and on patients. A re-localisation with N > 2 epipolar lines returns more
accurate results than with only 2 epipolar lines. Moreover, as the LK tracker tended to return
a greater proportion of inliers among the matched features than the SIFT, the epipolar lines
derived for the re-localisations tended to be more accurate, and the resulting re-localised
biopsy site was also more accurate. Nevertheless, matching SIFT features may be useful
when the camera field of view is obstructed by air/water bubbles or when the biopsy site
needs to be re-localised during a future examination using reference images acquired during a
previous examination. The next chapter presents a method to increase the proportion of
inliers among the SIFT features.
154
Chapter 8 Combination of an Electromagnetic Tracking
System with the Re-Localisation Method
8.1 Introduction
Fig. 8-1: Critical cases for a good performance of the LK tracker: two sequences of endoscopic images
aquired during a surveillance examination of Barrett’s Oesophagus (BO) are presented as examples.
For each sequence, 3 endoscopic images are extracted to illustrate the problems that may be
encountered during endoscopy. For sequence 1 and sequence 2, the oesophagus surface is interrogated
by optical biopsy (top row), air/water bubbles may obstruct the endoscope Field Of View (FOV) or the
endoscope may move too fast when the miniprobe is removed (middle row), and the endoscopic
images are clear again (bottom row).
155
The previous chapters introduced a method to re-localise a biopsy site in a target endoscopic
image T. This method makes use of epipolar lines derived from at least two previously
acquired endoscopic images Ii where the biopsy site location is known. Therefore, the
epipolar geometry formed between each pair of images Ii ↔ T needs to be recovered. Once
the biopsy site is re-localised, its uncertainty can be computed in order to provide the
endoscopist with a measure of the re-localisation precision and with the display of a
confidence region around the re-localised biopsy site in the target image T.
As presented in Chapter 3, the recovery of the epipolar geometry requires the
detection of features in the endoscopic images Ii and T and their matching for each pair Ii ↔
T. During a surveillance examination of Barrett’s Oesophagus (BO), the endoscopist may
manage to move the endoscope slowly, while keeping the biopsy site in the camera Field Of
View (FOV). In that case, the Lucas-Kanade (LK) tracker can be applied in order to detect
features in a first reference image Ii and in order to track them through the next images until
the target image T. However, in some situations, the camera FOV may be obstructed by
air/water bubbles or the endoscope tip may move too fast (Fig. 8-1). In these two cases, the
features are lost by the LK tracker and an alternative way of matching features is necessary.
This chapter aims to present and discuss a method to re-localise biopsy sites when the
features have been lost by the LK tracker. An ElectroMagnetic (EM) tracking system is used
to guide the process of feature matching. This method contributes to more accurate epipolar
lines and re-localised biopsy sites. A first experiment was done in order to assess the
accuracy of an EM tracking system in the conditions of an endoscopy. The results of this
experiment are presented. The results of a second experiment on a phantom demonstrate that
the EM tracking system helps increase the number and the proportion of good feature
matches and compute more accurate epipolar lines. Finally, the results of a third experiment
on excised porcine tissues show that more accurate re-localised biopsy sites can be computed
using the EM tracking system.
8.2 Re-localisation with an EM tracking system
8.2.1 Context, hypotheses, and description of an EM tracking system
The LK tracker may lose track of the features when air/water bubbles obstruct the FOV of the
endoscope camera, or when the camera moves too quickly. As discussed in Chapter 4, the
Scale Invariant Feature Transform (SIFT) features may be an alternative to the LK tracker in
these critical cases. Indeed, the features are detected independently in images Ii and T using
the scale-space representations of the images. For each detected feature, a descriptor is
computed. It is a vector which gives information about the distribution of the magnitude and
156
orientation of the gradients in the neighbourhood of the corresponding feature. Features are
matched by searching for the pair of descriptors in images Ii and T whose Euclidean distance
is minimal. Features, especially SIFT features, in endoscopic images correspond to blood
vessel junctions. As discussed in Chapter 4, junctions all look similar as do the descriptors
(Atasoy et al., 2009). This may create ambiguity when the SIFT features are matched and
result in a lot of mismatches or outliers. It is, therefore, necessary to guide spatially the
matching of SIFT features. For example, a rough knowledge of the relative positions between
the endoscope cameras can help determine a region in the target image where the correct
match of a feature in the reference image is likely to be.
An estimation of the endoscope camera movement can be given by positional sensors
such as an EM sensor mounted at the tip of the endoscope (Mori et al., 2005, 2007, and
2008). Mori et al. (2005, 2006, 2007, and 2008) built up hybrid systems combining
endoscopes, CT scanners, and EM tracking systems to track the positions and movement of
an endoscopic camera during a bronchoscopy procedure in a 3D reconstruction of the lumen
of the lung from a CT pre-operative image using registration of real endoscopic images with
virtual endoscopic images extracted from the 3D reconstruction. Given information about the
relative movement of the real camera from the EM sensor and a transformation from the real
camera coordinate system to the virtual camera coordinate system, the position of the virtual
camera could be updated in the virtual view and this position was used to initialise the image
registration algorithms for a refinement of the virtual camera position. In the case of biopsy
site re-localisation, the relative movement between endoscope cameras given by the EM
sensor can help identify a region in the target image where the match of a feature is.
An EM tracking system is made up of an emitter which generates electromagnetic
waves and of a sensor whose position and orientation are computed in the emitter coordinate
system (Fig. 8-2). The emitter is a box fixed next to the patient, for example. A coordinate
system (O, xem, yem, zem) is assigned to it and is used as a reference coordinate system for the
sensor. The sensor is made up of 3 coils that are arranged such that they form a coordinate
system (S, xS, yS, zS). The electromagnetic field is measured at the origin S of the sensor
coordinate system and the distance of S from the origin O of the emitter coordinate system
can be computed. Thus, the position S of the EM sensor is known in real-time in the emitter
coordinate system (O, xem, yem, zem). At position S, the orientations of the unit vectors of the
sensor coordinate system xS, yS, and zS in the emitter coordinate system are given by the
azimuth ψ, elevation θ, and roll Φ angles (Fig. 8-3). Thus, the rotational component of the
transformation matrix from the emitter coordinate system to the sensor coordinate system is
given by:
157
⎥⎥⎥⎥⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢⎢⎢⎢⎢
⎣
⎡
−
+−
+
++−
=
)cos(*)cos()cos(*)sin()sin(
))sin(*)sin(*)(cos())cos(*)(sin(
))sin(*)sin(*)(sin()cos(*)cos(
)sin(*)cos(
))cos(*)sin(*)(cos())sin(*)(sin(
))cos(*)sin(*)(sin())sin(*)(cos(
)cos(*)cos(
θφθφθ
ψθφψφ
ψθφψφ
ψθ
ψθφψφ
ψθφψφ
ψθ
SOM
(8-1)
When the sensor is mounted at the tip of an endoscope, its position and orientation
help track the tip translations and rotations in the emitter coordinate system. In this chapter,
the EM tracking system is used under the hypothesis that the endoscopist manages to keep
the biopsy site in the FOV of the endoscopic camera. As the oesophagus is attached to the
spine, it is assumed that the oesophagus is well trapped and does not move up and down in
the chest. Finally, it is assumed that the oesophagus does not stretch along its longitudinal
axis.
xem
xs
zs
ys
yem
zem
a) b)
O
S
8cm
1cm
Fig. 8-2: Main components of an EM tracking system (medSAFE system by Ascension Technology
Corporation): a) the emitter which generates electromagnetic waves in order to help measure the
position of b) the EM sensor which may be attached to the tip of the endoscope. The position and the
orientation of the EM sensor in the emitter coordinate system are measured.
158
xemyem
zem
Φ
θψ
yS
xS
zS
Fig. 8-3: Description of the EM sensor coordinate system (S, xS, yS, zS) in the EM emitter coordinate
system (O, xem, yem, zem) with spherical coordinates: the azimuth ψ, elevation θ, and roll Φ angles.
8.2.2 Combination of the EM tracking system with the re-localisation algorithm
A re-localised biopsy site p can be computed in a target image T by recovery of the epipolar
geometry formed by T with each of at least two reference images Ii that each show the site
under different viewpoints. The epipolar geometry formed by the pair of images Ii ↔ T is
represented algebraically using the fundamental matrix ( )TIF ,i:
( ) ( )[ ] ( ) ( ) ⋅⋅⋅⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
−−
−⋅=⋅⋅⋅= 1-
,
12
13
23tr-1-
,,tr-
,
00
0KRKKRtKF TITITITI iiii
tttt
tt
(8-2)
K is the intrinsic matrix linking the endoscope camera coordinate system with the 2D image
plane coordinate system. K can be assumed constant during endoscopy investigations and
recovered from a calibration procedure (Bouguet, 2004). The fundamental matrix ( )TIF ,i can
be computed either with computer vision algorithms as presented in Chapter 3 or with the
EM tracking system when the sensor is mounted at the tip of the endoscope camera or with a
combination of the two previous methods.
The hybrid method for biopsy site re-localisation detects SIFT features and estimates
the fundamental matrix from the EM tracking system referred to ( ( )TIF ,i)EM for each pair of
images Ii ↔ T. The fundamental matrix ( ( )TIF ,i)EM is used to increase the percentage and
159
number of inliers among the matched SIFT features (Fig. 8-4). This step has the advantage of
preventing the matching process from comparing useless potential matches and of adding a
constraint about the spatial arrangement of the features. Given the jth SIFT feature ( )ijIp in Ii,
the epipolar line ( ( )ijIel )EM = ( ( )TIF ,i
)EM. ( )ijIp is derived in T (Fig. 8-5). This line indicates the
location of the feature matching ( )ijIp in T. As ( )TIR ,i
and ( )TIt ,i are measured from the EM
tracking system with experimental error, the feature matching ( )ijIp in T lies in a region
around ( ( )ijIel )EM (Fig. 8-5). Zhang et al. (1995) suggested a search for matching features
within the envelope of epipolar lines. Therefore, the search for the matching feature in the
hybrid method is spatially constrained within a bounded region around ( ( )ijIel )EM. In practice,
the width of this region is set manually using a training reference image Ii and the target
image T. This width could also be computed analytically by propagating the errors resulting
from the determination of the calibration matrix K and of the localisation of the EM sensor to
the computed epipolar line. However, the analytical determination of the width requires a far
more complex derivation which is not used in this thesis. This first step returns a set of
matched SIFT features (Fig. 8-4).
Re-localised biopsy site p and confidence region
…I1↔T
- Computation of ( )EM, … ( )EM, … ( )EM from the EM tracking system to define the spatial constraints- Feature extraction with SIFT- Feature matching under the spatial constraints to increase the percentage of inliers
…Inputs
Feature extraction and
matching
Refinement of the fundamental
matrices
Outputs
Ii↔T IN↔T
Computation of ( )fused, … ( )fused, … ( )fused
( )TIF ,i( )TIF ,1 ( )TIF ,N
( )TIF ,1 ( )TIF ,i ( )TIF ,N
Fig. 8-4: Hybrid method for biopsy site re-localisation: information from the EM tracking system helps
recover approximately the epipolar geometry formed by each pair of endoscopic images Ii ↔ T. This
returns a set of fundamental matrices ( ( )TIF ,i)EM which help constrain the matching process of the SIFT
features. Once the features have been matched, the estimation of the epipolar geometries is refined
which returns a set of more accurate fundamental matrices ( ( )TIF ,i)fused and the re-localisation method
can be applied.
160
I iI i T
pj(elj )EM
(F( ,T))EM
(Ii)
Ii
(Ii)
Fig. 8-5: Spatial constraint during SIFT feature matching: the search in the target image T for the
feature matching the feature ( )ijIp in Ii is constrained in a bounded region (blue dotted lines) centred on
the epipolar line ( ( )ijIel )EM derived from ( )i
jIp . The green dots are the features.
Secondly, the hybrid method for biopsy site re-localisation refines the estimation of
the fundamental matrices. The computer vision algorithms presented in Chapter 3 are applied
with the SIFT features matched using the spatial constraint from the EM tracking system for
the pairs of images Ii ↔ T (Fig. 8-4). This step returns a set of fundamental matrices
( ( )TIF ,i)fused. Given the biopsy site location ( ) ( ) ( )[ ] tr
1,, iiiyx pp IIIp = in the reference image Ii, the
re-localised biopsy site p is computed in T by minimising the sum of its squared
perpendicular distances to the N epipolar lines ( ( )iIel )fused = ( ( )TIF ,i)fused. ( )iIp = [( ( )i
xel I )fused,
( ( )iyel I )fused, ( ( )i
mel I )fused]tr as presented in Chapter 4:
( )( )( )( )( )
( )( ) ( )( )∑∑==
⋅⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
+==
N
iyx
N
ii
ii
i
i
elelCC
1
2
2
fused
2
fused
fusedtr
1
2fusedmin min,min
II
I
p
I
p
elppel
(8-3)
8.2.3 Computation of ( ( )TIF ,i)EM during the first step of the hybrid method
An EM tracking system consists of a sensor and an emitter. The endoscope camera positions
C and orientations are computed from the EM sensor positions S and orientations measured
in the emitter coordinate system whose reference position is termed O (Fig. 8-6). The
orientation of the sensor is given by the rotation matrix from the emitter coordinate system to
the sensor coordinate system referred as SOM . When the endoscope camera moves from
position C_Ii to position C_T, the sensor positions are S_Ii and S_T and the orientations are
reported by the matrices iISOM _ and TS
OM _ (Fig. 8-6). The computation of the fundamental
matrix ( ( )TIF ,i)EM requires the computation of the endoscope camera rotation ( )TIR ,i
and
translation ( )TIt ,i which are expressed in the coordinate system of the endoscope camera at
position C_T. Thus, the rotation matrix CSM from the sensor coordinate system to the
161
endoscope camera coordinate system has to be determined. By mechanical design, these two
coordinate systems are fixed relative to each other, and CSM is determined during a
calibration procedure by recording the EM sensor data and the endoscope camera images of a
calibration grid while the endoscope positions and orientations vary.
The translation ( )TIt ,i is computed from the EM sensor position S_Ii and S_T
measured in the emitter coordinate system. The translation has to be computed in the EM
sensor coordinate system at position S_T, first, using the rotation matrix TSOM _ . The resulting
vector is expressed in the camera coordinate system at position C_T, secondly, using the
rotation matrix CSM . Therefore, ( )TIt ,i
is computed as follows:
( ) ( ) ( ) ( )iiISTSMMt TS
OCSTI __.. 1_1
, −=−−
. (8-4)
The matrix ( )TIR ,i is the relative rotation from one endoscope camera position to
another position. It is given directly by the EM sensor and it can be decomposed as a product
of rotations from the EM sensor coordinate system to the EM emitter coordinate system SOM
at positions S_Ii and S_T:
( ) ( ) ( ) CS
ISO
TSO
CSTI MMMMR ... _1_1
,i
i
−−= . (8-5)
8.3 Experiments and results
This section describes a set of experiments to test how well the hybrid method performs to
derive the epipolar lines necessary for the re-localisation. The performance depends on:
• The accuracy of the EM tracking system to determine the position of the endoscope.
O
Emitter coordinate system
Sensor coordinate system
Camera coordinate system
CS
Endoscope
SOM
CSM
Fig. 8-6: Relations between the coordinate systems of the camera, of the EM sensor, and of the EM
emitter.
162
• The matched features between the reference image Ii and the target image T. As
discussed in Chapter 4 in section ‘4.3.2 Experiment: number of matches for the
computation of the fundamental matrix’ and as stated by Luong and Faugeras (1996)
and Hartley and Zisserman (2004), accurate epipolar geometry is recovered when
there are a lot of well matched features or inliers (Luong and Faugeras, 1996; Hartley
and Zisserman, 2004).
• The endoscope camera. If the resolution is high, the blood vessels are pictured with
more details in the acquired endoscopic images. The vessels may also appear with a
better contrast. Features may be better detected in these conditions.
• The illumination of the tissue. If the oesophagus structure is well illuminated, the
vessels are visible in the whole image and the detected features are spread well over
the endoscopic image.
The experiments consisted of first assessing the error with which the EM tracking
system measures the displacement of the EM sensor. Secondly, the experiments consisted of
studying the error of the positioning of the epipolar lines derived with the re-localisation
system. The last experiment consisted of testing the method on excised organs from pigs.
8.3.1 Experiment 1: error of an EM tracking system for the determination of the
displacement and of the orientation of the EM sensor
Fig. 8-7: Experimental setup: a) the EM sensor was mounted at the tip of the endoscope; b) the
endoscope and the EM sensor were clamped by a probe holder that could be moved in various
directions; c) the phantom was a carton box in which holes were drilled every centimetre.
163
An EM tracking system can be used to measure the endoscope camera movement.
Therefore, the error of the measurement of the orientation and of the relative distance
between two positions of the EM sensor had to be estimated.
8.3.1.1 Materials and method
Hummel et al. (2005) did an intensive study of the accuracy of various EM tracking systems.
They designed a phantom which was a plexiglass board with holes drilled at regular
distances. The EM sensor was inserted in each hole. The positions and orientations of the
sensor were measured. Hummel et al. (2005) compared the measured distances between two
neighbouring holes and the measured angles to the actual distance on the phantom and to the
actual change of angle. They also assessed the accuracy of the EM sensor in presence of
various metals which can distort the electromagnetic signal.
In this chapter, the experiment aimed to assess the accuracy of the EM tracking
system in the context of an endoscopy. The EM tracking system was the medSAFE system by
Ascension Technology Corporation and the EM sensor had 6 degrees of freedom. The
experiment was inspired from those presented in Hummel et al. (2005). An EM sensor was
mounted at the tip of an endoscope since this is the way the EM tracking system is used in the
hybrid method (Fig. 8-7 a)). The endoscope contains metallic parts that can distort the signal
measured by the EM sensor. The endoscope and the EM sensor were held by a probe holder
which helped move them manually with a controlled movement (Fig. 8-7 b)). The phantom
was similar to Hummel et al. (2005) phantom. Hundred holes were drilled every centimetre
(d0 = 1cm) in a carton board (Fig. 8-7 c)).
The experiment consisted of moving the endoscope and the EM sensor with the probe
holder from holes to holes without changing the orientation. The EM sensor position and
orientation were measured in the EM emitter coordinate system. For each hole, its distance di
from each of its 4 neighbours was computed and compared to d0. As suggested by Hummel et
al. (2005), the Root Mean Squared (RMS) error of the difference between the measured
distances di and the exact distance d0 was computed in order to assess the error resulting from
an estimation of the displacement of an endoscope camera with an EM tracking system:
( )∑=
−⋅−
=n
ii dd
1n 1
20error
1RMS . (8-6)
where n is the total number of distances compared. The maximum, minimum, and mean error
was computed. Furthermore, due to the nature of the movement of the probe holder, the
changes of orientations of the EM sensor should be null. The values of the azimuth ψ0,
elevation θ0, and roll Φ0 angles at the position of the hole scanned first with the EM sensor
164
were measured and used as the ground truth angles. The values of ψi, θi, and Φi angles for the
99 other positions were measured and compared to the ground truth. Therefore, the RMS
errors of (ψi - ψ0), of (θi - θ0), and of (Φi - Φ0) were computed as well as their maximum,
minimum, and mean errors. For this experiment, the board was positioned at a distance of
approximately 50cm from the EM emitter which is a reasonable distance during clinical
practice on the patient. Hundred holes were drilled because that is a significant number for
the estimation of the RMS error. Finally, the distance between two holes was 1cm such that
the distance along one line corresponded to the movement amplitude of the probe holder.
8.3.1.2 Results and discussion
The results are presented in Table 8-1. On average, the displacement and the orientation of
the EM sensor were measured with an error of approximately 1mm and less than 3 degrees.
As the EM sensor was placed in contact with the table supporting the phantom when it was
inserted into the hole, the difference of elevation from one hole to another was smaller than
for the other angles. Thus, the RMS error, maximum error, and mean error were smaller. The
insertion of the sensor into the hole could create small rotations of the sensor and the
resulting errors for the azimuth and roll angles were larger than for the elevation angle.
Finally, the variations of the measurements could be due to the noise generated by devices or
metal objects in the neighbourhood which use or carry large amounts of electrical current that
vary over time or to the internal noise of the EM tracking system (Hummel et al., 2005).
These sources of noise came from the endoscope and the metallic tables in the room where
the experiment was run.
The accuracy of the measurements from the EM sensor has an influence on the
accuracy of the fundamental matrix estimated from the EM tracking system. The next
experiments study the accuracy of the epipolar lines computed with the hybrid method and
used for the re-localisation.
Table 8-1: Results of the errors between the measured distances and the exact distance (in millimetres)
Displacement = 6mm Displacement = 8mm Displacement = 10mm Fig. 8-12: Decrease of the contrast of a vessel intersection in the endoscopic images with the
displacement of the endoscope camera: as long as the illumination was high, the intersection was
detected as a SIFT feature located at the centre of the green circle, representing the descriptor
(displacements from 0mm to 2mm). When the illumination decreased, the vessel intersections were not
detected anymore as features since their contrast with the background was getting too weak.
Displacement = 0mm Displacement = 4mm
Displacement = 8mm Displacement = 10mm
Fig. 8-13: Variations of the descriptor (green circle) and of its main orientation (green ray) with the
displacement of the endoscope camera: as the feature located in the centre of the green circle was
characterised by a smaller contrast and by a more blurred neighbourhood when the camera moved
backwards, the descriptor could be different in Ti from the descriptor of the corresponding feature in I.
This might make the feature matching more difficult.
176
The accuracy of the epipolar lines computed using data from the EM tracking system
depends not only on the accuracy of the EM tracking system but also on the accuracy of the
registration of the endoscope camera coordinate system with the EM sensor coordinate
system. Therefore, the impact of the accuracy of the EM tracking system on the epipolar lines
used for the re-localisation with the hybrid method involves complex propagations of errors.
It is, however, expected that the epipolar lines derived from the EM tracking system will be
less accurate when the EM tracking system accuracy gets worse. Because the epipolar lines
are less accurate, the width of the bounded region to match features becomes larger. The risk
of matching features incorrectly will then become higher and the epipolar lines computed for
the re-localisations will be less accurate. As demonstrated by the simulations described in
Chapter 5, the re-localised biopsy site is likely to be less accurate.
I Ti Fig. 8-14: Illustration of the SIFT feature matching with spatial constraint: a feature, green point, in the
reference image I of the phantom was matched to a feature in the bounded region in Ti (blue lines
surrounding the epipolar line).
a) b)
c) d)
Fig. 8-15: Illustration of the contribution of the spatial constraint from the EM tracking system: a) a
SIFT feature was detected in reference image I; b) the correct matching feature in the target image Ti
looks similar to c) another feature in Ti which is why the matching without the spatial constraint did
not return a match while d) a matching with spatial constraint within a bounded region (2 blue lines)
centred on the epipolar line (red line) returned the correct match (green point).
177
Tracheas
Bronchus
EMemitter
Endoscope holder
Endoscope
EM sensor
Fig. 8-16: Experimental setup: endoscopic images of 2 pigs’ trachea and bronchus were acquired with
an endoscope mounted with an EM sensor and controlled with a probe holder.
Biopsy siteBiopsy siteBiopsy site
Biopsy siteBiopsy siteBiopsy site
(a) Pig bronchus
(b) Pig trachea Fig. 8-17: Examples of endoscopic images acquired in the pig bronchus and trachea. The biopsy site
was a natural pigment spot of approximately 2mm diameter.
8.3.3 Experiment 3: test of the method on excised organs from pigs
The combination of the EM tracking system with the re-localisation method could not be
tested on real patients since EM tracking data could not be acquired. Therefore, this was
tested on excised pig organs.
8.3.3.1 Materials and method
The oesophagus was entirely collapsed; therefore, endoscopic data were acquired in the
trachea and the bronchus of 2 pigs. The experimental setup was the same as that presented in
section ‘8.3.2.1.1 Experimental setup’ for the experiments on the phantom representing an
oesophagus (Fig. 8-16).
178
The endoscope tip was moved randomly with the endoscope holder around a region
of interest to acquire a set of images in the first pig bronchus and in the second pig trachea
(Fig. 8-17). The image dimensions were 1600 pixels x 1200 pixels. For the 2 sets of images, a
target image T was selected. A pigment spot of 2mm diameter at the tissue surface was
selected as the biopsy site and its positions in the reference images Ii and in the target image
T were recorded. SIFT features were detected for each image Ii or T. As for the patient data,
a mask in the endoscopic images was used to remove the SIFT features that were detected
around the biopsy site.
The biopsy site was re-localised with 3 methods already presented in this chapter:
Method 1: Re-localisation using the EM tracking system only: For each pair of
images Ii ↔ T, the fundamental matrix ( ( )TIF ,i)EM was estimated from the EM tracking
system. Epipolar lines could be derived in T from the biopsy sites in the reference images Ii
using ( ( )TIF ,i)EM. The re-localised biopsy site could be computed in T from these epipolar
lines.
Method 2: Re-localisation without using the contribution of the EM tracking
system: For each image Ii and T, SIFT features were matched without the spatial constraint
from the EM tracking system that the matching features in T should be in a bounded region
around the epipolar line. Once the features were matched, the epipolar geometry formed by
each pair of images Ii and T was computed with the MAPSAC and the nonlinear
optimisation. Epipolar lines were derived in T from the biopsy site locations in the images Ii
to estimate the re-localised biopsy site.
Method 3: Re-localisation using the contribution of the EM tracking system:
This was the method of the whole re-localisation system presented in Chapter 8. For each pair
of images Ii ↔ T, the fundamental matrix ( ( )TIF ,i)EM was estimated from the EM tracking
system. For each image Ii and T, SIFT features were matched with the spatial constraint. A
training image was used to set the value of the width of the bounded region. The epipolar
lines were derived after refinement of the epipolar geometry with MAPSAC and a nonlinear
optimisation over the matched features. Epipolar lines were derived in T from the biopsy site
locations in the images Ii to estimate the re-localised biopsy site.
After the computation of the re-localised biopsy site, the analysis of the accuracy and
the conversion from pixels to millimetres were the same as those done for real patients in
Chapter 7. The biopsy site was re-localised with 6 epipolar lines in the pig bronchus and with
8 epipolar lines in the pig trachea. Also, for each pair of images Ii ↔ T, the number and the
percentage of inliers were counted manually.
179
a) b) c)
Without constraint:Accuracy = 30 pixels0.76mm
With constraint:Accuracy = 17 pixels0.43mm
Without constraint:Accuracy = 41 pixels1mm
With constraint:Accuracy = 12 pixels0.31mm
Fig. 8-18: Results of the experiment on excised pig bronchus and trachea: columns a) and b) re-
localisation results in bronchus and trachea: the blue epipolar lines were derived in T from the biopsy
site locations in the reference images Ii, and the green point is the re-localised biopsy site; column c)
illustration of erroneous matching (bottom) of SIFT features (top) that would be excluded by the
constraint provided by the EM tracker: a SIFT feature is represented in Ii (top row) at the centre of a
circle whose radius is proportional to the feature scale and whose drawn radius indicates the feature
orientation. The bottom row shows the matched feature in T.
8.3.3.2 Results For the ex vivo study, the spatial constraint from the EM tracking system could contribute to
more accurate re-localised biopsy sites (Fig. 8-18 a) and b)). For the sequence acquired in the
pig bronchus, one epipolar line derived from a set of matches counting 38% inliers without
spatial constraint i.e. without the contribution of the EM tracking system was less accurate
than the epipolar line derived from a set of matches counting 55% inliers with spatial
constraint (Fig. 8-18 a)). The biopsy site was re-localised with an accuracy of 1.6mm with the
EM tracking system only, 0.76mm without spatial constraint, and 0.43mm with spatial
constraint. For the sequence acquired in the trachea, one epipolar line was derived from 42
inliers without constraint while it was derived from 108 inliers with constraint (Fig. 8-18 b)).
The biopsy site was re-localised with an accuracy of 3mm with the EM tracking system only,
1mm without spatial constraint, and 0.31mm with spatial constraint. For these cases, the EM
tracking system helped avoid matching SIFT features whose neighbourhoods were similar
although they did not correspond to the same anatomical point (Fig. 8-18 c)).
180
8.4 Conclusion
This chapter aimed to demonstrate that accurate epipolar geometry can be estimated using the
contribution of an EM tracking system. The EM tracking system returns a crude
approximation of the epipolar geometry. It can help constrain spatially the matching of SIFT
features using information that the features must be next to their epipolar line estimated with
the EM tracking system.
A study on phantom showed that the EM tracking system has the potential to increase
the percentage and the number of inliers during the matching process of SIFT features. It is
particularly the case for 2 images acquired for camera positions which differ by more than
6mm and for images where the features have a poor contrast with the background. For
example, the epipolar geometry could be recovered from a set of matches containing 62%
inliers when information from the EM tracking system was used while it was recovered from
matches containing 48% inliers without the contribution of the EM tracking system. For this
example, information from the EM tracking system helped detect 86 matches containing 53
inliers while no information resulted in 69 matches containing 33 inliers. These results were
obtained on a static phantom with a texture whose patterns repeated a lot. The contribution of
the EM tracking system to the re-localisation was shown with experiments on excised organs
from pigs. The re-localised biopsy site was computed in a trachea and in a bronchus with an
accuracy better than 0.5mm thanks to the contribution of the EM tracking system while it was
computed with an accuracy between 0.5mm and 1mm without the contribution of the EM
tracking system.
As the endoscopes used during the endoscopy examinations in the experiments
described in this chapter did not have an integrated EM tracking system, these experiments
were limited to studies using a rigid static phantom and static excised tissues. The conditions
of these experiments were intended to reproduce those of an endoscopy examination of
Barrett’s Oesophagus. The assumptions of rigidity and of the absence of movement of the
oesophagus are justified since the oesophagus does not deform while the endoscope stays
near the biopsy site and the patient is usually calm and does not normally move during
endoscopy examinations. Furthermore, the oesophagus is located near the spine and is
physically constrained by the trachea and lungs. Therefore, it is not expected to move
significantly with the breathing motion. Nevertheless, further developments could take into
consideration small displacements of the organ with the patient breathing and movement to
further increase accuracy. Additional developments include the determination of the limits of
the accuracy of the EM tracking system in order to achieve a re-localisation accuracy that is
sufficiently high for a particular clinical application. This could be done by computer
181
simulations using a Monte Carlo approach. An initial scene could be created virtually in such
a way that epipolar lines all pass through the biopsy site and subtend an angle of 40°. Specific
standard deviations could be applied to the positions and orientations of a simulated EM
sensor. These standard deviations could be propagated as errors through the transformation to
the camera coordinate system. The epipolar lines would be recomputed with uncertainty and
the re-localised biopsy site would be computed. This computation would be repeated a great
number of times for the same standard deviations applied to the positions and orientations of
the simulated EM sensor in order to estimate the required accuracy of the re-localisation. The
standard deviations would be changed, afterwards, in order to determine a range of
inaccuracies of the EM tracking system that can return accurate re-localisations.
Chapters 5, 6, 7, and 8 presented methods to re-localise a biopsy site using epipolar
lines and to compute a confidence region around the re-localised biopsy site. The next chapter
consists of defining possible routes to develop a re-localisation system that can be used
clinically and potential applications of these methods.
182
Chapter 9 Conclusion and Future Work
9.1 Conclusion
Many diseases start at the superficial layers of tissue and are characterised by microscopic
lesions that evolve before the diseases reach a malignant stage. Optical biopsy techniques
have the potential to detect these lesions in vivo and in situ. They commonly consist of
miniprobes that are placed in contact with the tissue for interrogation of a 0.5mm x 0.5mm
extent and return information at the cellular level, the microscopic scale. Once these lesions
have been detected, it is necessary to re-localise these lesions in their macroscopic context in
order to go back to the same position at a later time. This thesis is based on the development
of a re-localisation method of lesions with an application in gastroenterology. This thesis
aimed to study the potential contribution of epipolar geometry to solve the problem of biopsy
site re-localisation in endoscopic images during a surveillance examination of Barrett’s
Oesophagus (BO).
The epipolar geometry formed by a pair of a reference endoscopic image and of a
target endoscopic image can be represented algebraically by the fundamental matrix. It is a
mapping between the two images. It transforms a point in the reference image as an epipolar
line in the target image which indicates the locus of the possible positions of the matched
point in the target image. As the fundamental matrix is a point-to-epipolar line mapping,
more pairs of endoscopic images are necessary to constrain the locus of possible positions of
the searched point. The pairs of images have to show the point under different viewpoints in
order to compute the re-localised biopsy site. Either 2 or N > 2 epipolar lines were used.
Chapter 5 discussed this method which was the first contribution of this thesis. Simulations
aimed to study the variations of the re-localised biopsy site accuracy and precision with the
noise in the images and, therefore, with the accuracy of the epipolar lines. When the epipolar
lines were determined less accurately, the resulting re-localised biopsy site tended to be less
accurate when it was computed either with 2 epipolar lines only or with N epipolar lines. This
chapter studied also the influence of the angle subtended by the epipolar lines on the accuracy
of the re-localised biopsy site. The larger the angle, the more accurate and precise the biopsy
site.
The angle subtended by the epipolar lines depends on the movement of the
endoscope camera (or, equivalently, endoscope tip). This movement helps generate multiple
different viewpoints. As the camera is commonly located around the rotation axis of the
183
endoscope, twisting the tip of the endoscope is sufficient to translate and rotate the camera.
With such motion, the epipoles follow a circular trajectory around the centre of the image as
discussed in Chapter 7. As the biopsy site is usually not far either from the centre of the target
image, the resulting epipolar lines can subtend a large angle up to 50°. Further experiments
should be done in order to determine the minimal angle with which the endoscope tip should
be rotated for the generation of sufficiently different epipolar lines. In clinical practice, the
epipolar lines could also be overlaid on the images in order to help the endoscopist decide
how to rotate the endoscope.
During an endoscopic examination, the display of a confidence region around the re-
localised biopsy site may be useful for instrument guidance. The confidence region of a point
is commonly drawn from the covariance matrix of this point. The covariance matrix can be
determined experimentally which consists of estimating the point several times adding
different noise samples in the images for each experiment. This method requires, however, a
lot of iterations. Chapter 6 presented an analytical derivation for the determination of this
covariance matrix without iterations which was the second contribution of this thesis. The
derivation was developed for the re-localisation method with N > 2 epipolar lines. The
resulting analytical covariance matrices were not in perfect correspondence with the
experimental covariance matrices. However, these derivations have the potential to display an
approximation of the confidence region around the re-localised biopsy site.
Chapter 5 and Chapter 6 focused on the use of epipolar lines to re-localise a biopsy
site and on analytical derivations to estimate the confidence region around the site in the
target image. Chapter 7 presented the results of the validation of the re-localisation method
and of the analytical uncertainty. The validation was done using data acquired on a phantom
and on real patients with various endoscopes and various endoscope camera motions. During
some examinations, air/water bubbles obstruct the field of view of the endoscope camera or
the camera may move too quickly which returns blurred images. These events may happen
between the time before which the endoscopist was interrogating the tissue with the
miniprobe and the time after which the biopsy site needed to be re-localised. The recovery of
the epipolar geometry between the reference image and the target image requires the
detection of common features which are commonly located on the vasculature. When events
like air/water bubbles appearance or quick camera movement does not occur, the Lucas
Kanade (LK) tracker may be used to detect features in the reference image and to track them
until the target image. This tracker has the advantage of constraining spatially the search for
the feature matches. However, in the case of such events, the LK tracker fails since features
cannot be tracked reliably. The Scale and Invariant Feature Transform (SIFT) may be used as
184
an alternative since SIFT does not track features. However, SIFT does not impose a spatial
constraint when matching the detected features. This may produce a high number of
mismatches or outliers which corrupt the computation of the epipolar geometry. Chapter 8
focused, therefore, on a method to reduce this number of outliers which was the third
contribution of this thesis. ElectroMagnetic (EM) tracking systems may be integrated into
endoscopes in order to track the movement of the endoscope camera in the 3D space. Thus,
the epipolar geometry formed by the pair of reference and target images can be recovered
from the EM tracking system. The search for feature matches can be constrained by this
estimate of the epipolar geometry. The resulting matches can be used again to refine the
computation of the epipolar geometry and to derive the epipolar lines used for the re-
localisation of the biopsy site.
Chapters 5, 6, 7, and 8 described and validated methods to re-localise biopsy sites
and determine their confidence region in critical or not critical cases. However, this thesis did
not explain how a whole re-localisation system could work in real-time with patients.
Moreover, the methods were developed only for re-localisation of a biopsy site in subsequent
images acquired during the same endoscopy examination. Finally, the thesis did not study the
potential fusion of information coming from various imaging modalities at different scales.
These three routes define possible future works to make use of or extend the re-localisation
method.
9.2 Re-localisation in real time
The results presented in this thesis were obtained from sequences processed after acquisition.
A major missing step is the automatic detection of the tip of the miniprobe. Also, the
algorithms for feature detection and matching and for recovery of the epipolar geometry were
implemented using Matlab (Matlab, R14, The Mathworks Ltd, Cambridge, UK). The
computations can be engineered to make them run faster.
Several authors suggested methods for the segmentation of the miniprobe in
endoscopic images (Speidel et al., 2006; Voros et al., 2006; Mountney et al., 2009).
Mountney et al. (2009) developed a method that segmented the miniprobe in the Hue-
Saturation-Value space of the endoscopic images; however, this method does not take into
account the presence of the metallic ferrule mounted at the tip of the miniprobe. Voros et al.
(2006) developed a method that detects the edges of the instruments using the Hough
transform and a prior knowledge about the position of the instrument in the endoscopic
image. This spatial information was derived from the hypothesis of rigidity of the endoscopes
and of the instruments. Thus, a more adapted method for miniprobe segmentation would be
185
that developed by Speidel et al. (2006) which is probabilistic segmentation using information
about the colour of the pixels to distinguish tissue from the instruments.
Implementations of the Lucas Kanade (LK) tracker and of the Scale Invariant Feature
Transform (SIFT) used in this thesis were not fast implementations. Graphic Processing Unit
(GPU) implementations of these algorithms have been developed and can be found at the