Page 1
Syste
m E
ng
ineeri
ng
an
d
Au
tom
ati
on
Dep
art
men
t
M. Belén Estebanez Campos
[email protected]
IROS 2009 : INTERNATIONAL CONFERENCE ON INTELLIGENT ROBOTS AND SYSTEMS
MINIMALLY INVASIVE SURGERY MANEUVER
RECOGNITION BASED ON SURGEON MODEL
B. Estebanez, G. Jimenez, V. Muñoz, I. Garcia-Morales, E. Bauzano and J.
Molina
Dpto. Ingeniería de Sistemas y Automática
http://www.isa.uma.es
Universidad de Málaga (Spain)
Page 2
Syste
m E
ng
ineeri
ng
an
d
Au
tom
ati
on
Dep
art
men
t
M. Belén Estebanez Campos
[email protected]
Minimally Invasive Surgery Maneuver Recognition Based On Surgeon Model
OUTLINE
I. Introduction
II. Intervention Model
III.Surgeon Behavior Model
IV.Implantation and Experiments
V. Conclusions and Future Works
Page 3
Syste
m E
ng
ineeri
ng
an
d
Au
tom
ati
on
Dep
art
men
t
M. Belén Estebanez Campos
[email protected]
Minimally Invasive Surgery Maneuver Recognition Based On Surgeon Model
I. INTRODUCTION
Page 4
Syste
m E
ng
ineeri
ng
an
d
Au
tom
ati
on
Dep
art
men
t
M. Belén Estebanez Campos
[email protected]
I. Introduction
Laparoscopic Surgery:
“Accessing to the abdominal
cavity with especial long
instruments and by means of an
endoscope with camera can be
seen the area of the operation.”
LAPAROSCOPIC SURGERY
Advantages
• Reduce recovery time
• Limit post-operative complications
• Lower scars
Constraints
• Movement limitations
•Techniques require high skill
• Loss of touch and 3D perception
• Hand-eye coordination problems
Page 5
Syste
m E
ng
ineeri
ng
an
d
Au
tom
ati
on
Dep
art
men
t
M. Belén Estebanez Campos
[email protected]
I. Introduction
COMMERCIAL SYSTEMS OF SURGICAL ROBOT
Zeus
Da Vinci
Page 6
Syste
m E
ng
ineeri
ng
an
d
Au
tom
ati
on
Dep
art
men
t
M. Belén Estebanez Campos
[email protected]
I. Introduction
INTERFACE OF THE ROBOTIC ASSISTANT OF TWO ARMS
Robotic Assistant of Two Arms
Surgeon and tools with marks
Computer with
Recognition System
Operation Room
Surgical
Tool
Surgeon
3D Tracker
Page 7
Syste
m E
ng
ineeri
ng
an
d
Au
tom
ati
on
Dep
art
men
t
M. Belén Estebanez Campos
[email protected]
Minimally Invasive Surgery Maneuver Recognition Based On Surgeon Model
II. INTERVENTION MODEL
Page 8
Syste
m E
ng
ineeri
ng
an
d
Au
tom
ati
on
Dep
art
men
t
M. Belén Estebanez Campos
[email protected]
THE WORKFLOW OF A SURGICAL PROCEDURE
II. Intervention Model
START
Dissecting tissue to expose cystic duct
Clipping cystic duct
Cutting cystic duct
Dissecting/Separating Gallbladder
Collecting Gallbladder
END
First Maneuver
Second Maneuver
Third Maneuver
Fourth Maneuver
Fifth Maneuver
Sixth Maneuver
Final Maneuver
Page 9
Syste
m E
ng
ineeri
ng
an
d
Au
tom
ati
on
Dep
art
men
t
M. Belén Estebanez Campos
[email protected]
SET OF BASIC ACTIONS OF A MANEUVER
II. Intervention Model
Four degrees of freedom
Maneuver: set of basic actions
that represent the instrument
movements: left-right, up-down, to
rotate on itself and insertion-
removal.
Others functions are clamp
opening and closing.
Each maneuver is identified as a
specific sequence of these basic
actions.
Page 10
Syste
m E
ng
ineeri
ng
an
d
Au
tom
ati
on
Dep
art
men
t
M. Belén Estebanez Campos
[email protected]
INTERVENTION MODEL
II. Intervention Model
UPPER LAYER INNER LAYER
Page 11
Syste
m E
ng
ineeri
ng
an
d
Au
tom
ati
on
Dep
art
men
t
M. Belén Estebanez Campos
[email protected]
Minimally Invasive Surgery Maneuver Recognition Based On Surgeon Model
III. SURGEON BEHAVIOR MODEL
Page 12
Syste
m E
ng
ineeri
ng
an
d
Au
tom
ati
on
Dep
art
men
t
M. Belén Estebanez Campos
[email protected]
LIBRARY OF MANEUVER MODELS
III. Surgeon Behavior Model
Off-line process
MODULE OF MANEUVER TRAINING
LIBRARY OF MANEUVER
MODELS
On-line process
MODULE OF MANEUVER
RECOGNITION
DATA ACQUISITION AND CODING
MODULE
Page 13
Syste
m E
ng
ineeri
ng
an
d
Au
tom
ati
on
Dep
art
men
t
M. Belén Estebanez Campos
[email protected]
DATA ACQUISITION AND CODING MODULE
III. Surgeon Behavior Model
CHARACTERISTIC EXTRACTION
FUZZY LABELING
Characteristic Vector
of a Movement
Signals of Movements of
surgical tools
Current
Movement
Label
Observable
Output
Page 14
Syste
m E
ng
ineeri
ng
an
d
Au
tom
ati
on
Dep
art
men
t
M. Belén Estebanez Campos
[email protected]
MODULE OF MANEUVER TRAINING
III. Surgeon Behavior Model
DATA ACQUISITION AND CODING
MODULE
MODULE OF MANEUVER TRAINING
LIBRARY OF MANEUVER
MODELS
Action
TypePosition and Orientation
of Tools
Outputs
SequenceParameters of a
Maneuver Model
Surgeon
Page 15
Syste
m E
ng
ineeri
ng
an
d
Au
tom
ati
on
Dep
art
men
t
M. Belén Estebanez Campos
[email protected]
MODULE OF MANEUVER RECOGNITION
III. Surgeon Behavior Model
Surgeon
DATA ACQUISITION AND CODING
MODULE
Change Condition
of Maneuver
Position and
Orientation
of Tools
MODULE OF MANEUVER
RECOGNITION
LIBRARY OF MANEUVER
MODELS
Outputs
Sequence
INTERVENTIONMODEL
Current
Maneuver
Parameters
of a Maneuver
Model
Page 16
Syste
m E
ng
ineeri
ng
an
d
Au
tom
ati
on
Dep
art
men
t
M. Belén Estebanez Campos
[email protected]
MODULE OF MANEUVER RECOGNITION
III. Surgeon Behavior Model
Change Condition
of Maneuver
Outputs
Sequence
INTERVENTIONMODEL
Current
Maneuver
Maneuver with
High
Probability
LIBRARY OF MANEUVER MODELS
Maneuver 1 (A1 , B1 matrixes)
Maneuver 2 (A2 , B2 matrixes)
Maneuver 3 (A3 , B3 matrixes)
Maneuver N (AN , BN matrixes)
ViITERBI ALGORITHM
Page 17
Syste
m E
ng
ineeri
ng
an
d
Au
tom
ati
on
Dep
art
men
t
M. Belén Estebanez Campos
[email protected]
Minimally Invasive Surgery Maneuver Recognition Based On Surgeon Model
IV. IMPLEMENTATION AND EXPERIMENTS
Page 18
Syste
m E
ng
ineeri
ng
an
d
Au
tom
ati
on
Dep
art
men
t
M. Belén Estebanez Campos
[email protected]
• Simplification of the reality.
• Recognition of trajectories of tool tip.
• The basic actions are lines with different angles
and directions.
• Library has geometrical figures like
circle, square and triangle.
• A protocol is a sequence of these maneuvers.
1800
1900
2000
2100
2200
2300
-250
-200
-150
-100
-50
-300
-200
-100
0
100
200
300
400
x (mm)y (mm)
z (
mm
)
EXPERIMENTAL SYSTEM SETUP
IV Implementation and Experiments
Tool Tip
Fulcrum
Hand
Page 19
Syste
m E
ng
ineeri
ng
an
d
Au
tom
ati
on
Dep
art
men
t
M. Belén Estebanez Campos
[email protected]
MANEUVER MODEL
IV Implementation and Experiments
RECOGNITION RESULTS
End
A 4
A 2A 1
A 3
Start
Table 1. Isolated Maneuvers
Trajectory % Real Correct
Answer (100 essays /
maneuver)
CIRCLE 98.35 %
SQUARE 100.00 %
TRIANGLE 95.87 %
Table 2. Protocols
Trajectory % Ideal Correct
Answer (50
essays /protocol)
% Real Correct
Answer (50
essays / protocol)
90.39 % 85.00 %
91.20 % 82.50 %
94.29 % 81.00 %
Page 20
Syste
m E
ng
ineeri
ng
an
d
Au
tom
ati
on
Dep
art
men
t
M. Belén Estebanez Campos
[email protected]
Minimally Invasive Surgery Maneuver Recognition Based On Surgeon Model
V. CONCLUSIONS AND FUTURE WORKS
Page 21
Syste
m E
ng
ineeri
ng
an
d
Au
tom
ati
on
Dep
art
men
t
M. Belén Estebanez Campos
[email protected]
Minimally Invasive Surgery Maneuver Recognition Based On Surgeon Model
CONCLUSIONS
A methodology has been established for modeling the surgical
behavior: acquisition and coding, off-line training and on-line
recognition .
The HMM are useful for surgeon maneuver recognition.
Limitations
The performance of the system can be affected by the shadows
during the instrument tracking.
Improvement of the maneuver ending condition detection.
The experiments are about fictitious maneuver and are dependent on
the figure orientation.
Page 22
Syste
m E
ng
ineeri
ng
an
d
Au
tom
ati
on
Dep
art
men
t
M. Belén Estebanez Campos
[email protected]
Minimally Invasive Surgery Maneuver Recognition Based On Surgeon Model
FUTURE WORKS
Acquisition of information inside an
operation room.
Inclusion of other useful information.
Signal reconstruction in instrument
shadows situation.
Design of new real maneuver
models, including the movements of two
instruments.
Integration of this recognition module
in the global system of two arms.
Page 23
Syste
m E
ng
ineeri
ng
an
d
Au
tom
ati
on
Dep
art
men
t
M. Belén Estebanez Campos
[email protected]
THANKS FOR YOUR ATTENTION
IROS 2009 : INTERNATIONAL CONFERENCE ON INTELLIGENT ROBOTS AND SYSTEMS