Image Processing, Image Recognition, Computer Vision, Image Understanding Takashi Matsuyama [email protected]Dept. of Intelligence Science and Technology Graduate School of Informatics, Kyoto University Sound Processing, Speech Recognition, Auditory Scene Understanding Mathematical Theory of Pattern Recognition
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
The introduction will be given in two weeks. Slides will be on http://vision.kuee.kyoto-u.ac.jp/lecture/dsp Questions should be asked to [email protected] All reports should be sent to [email protected] by May 2nd (Fri.)
Longman Dictionary: 1. the act of realizing and accepting that something is true or important 2. public respect and thanks for someone's work or achievements 3. the act of knowing someone or something because you have known or learned about them in the past 4. the act of officially accepting that an organization, government, person etc has legal or official authority
awareness perception cognition
understanding
Intelligent mental function
Outer
World Environments
Other systems
Intelligent System
Reasoning, Learning
Knowledge
Perception
(Sensory System)
Recognition Sensation
Action, Manipulation
(Motor System)
Architecture of Intelligent Systems
Thought
Interaction
Report 1
(a) Describe the meanings of and differences among 1. Sensation, 2. Perception, and 3. Recognition.
(b) Describe the differences between 1. Cognition and 2. Recognition.
Information at the (physical) signal level
VS
Information at the (mental) cognitive level
Discrimination between these levels is important!
>
>
<
<
Physical Quantity vs Psychophysical Quantity
We see what we want to see and
hear what we want to hear.
Information at the image (signal) level
a pair of intersecting line segments in an image
an imaged part
(b) Electric circuit world
Information at the cognitive level
(a) block world
an imaged part
3D world Semantic world
θ
Shape from Shading
Outer
World Environments
Other systems
Intelligent System
Reasoning, Learning
Knowledge
Perception
(Sensory System)
Recognition Sensation
Action, Manipulation
(Motor System)
Architecture of Intelligent Systems
Mental World (Informatics) Physical World
(Physics)
How to bridge two worlds
Thought Thought
Architecture of 21st Century
Cyber Society (mental world)
Physical World
Cyber Network Society
Physical Real World
Social Structure in the 21st Century
?
Physical Laws (obey)
Rules, Standards (comply)
Physical Model
Computation Model
Cyber-Physical Systems
13
Cyber-Physical Systems for Developing Smart Society
1. e-money in economy 2. e-Tag in transportation (ubiquitous
systems) 3. Digitizing Human Activities 4. Smart Energy Management
Cyber Network Society
Physical Real World
(1) e-Money in economy
Authentication Security Pricing
Credit Warranty
Cyber Network Society
Physical Real World
(2) e-Tag in transportation (ubiquitous systems)
ID type age origin grade
11 onion 1 kyoto A
12 beef 3 USA B
E-tag
E-tag
ID role name opinion
8 leader Jim Yes
10 chair John ?
Location Information GIS
(3) Digitizing Human Activities
Real-time Integration
Sensing and Recognition Presentation and Control
Cyber Network Society
Human
Sensor Networks Embedded in the Real World
Motion and Blood Pressure Sensor
Taken from Panasonic Homepage
Real-Time Sensing &
Control
Power, Frequency, Phase Sensing
Power, Frequency, Phase Control
Cyber Network Society
(4)Integrating Information and Electricity Networks
Physical Real World (Electricity Power Network)
Solar Cell
Fuel Cell
Solar Cell
Solar Cell
Solar Cell
Solar Cell
battery
EV
EV
battery battery
battery
・distributed ・personalization ・bi-directional
Fundamental Concepts
Symbol (in the cyber network society) segmented entity with a unique ID
(basic processing: entity identification)
Signal (in the physical real world) non-segmented numerical data with physical measures
In each application of the cyber-physical systems, explain how pattern recognition technologies can be used to realize the informationization. 1. e-money in economy 2. e-Tag in transportation (ubiquitous systems) 3. Digitizing Human Activities 4. Smart Energy Management
Pattern Recognition
Pattern Recognition in Informatics
1. What are patterns? ① Classes/categories/types of objects (class: a set of objects)
② Internal structures of objects(example: design patterns, fabric patterns, sound and image patterns, behavior patterns)
2. What is recognition? ① Decision about the membership of a set
Basic Scheme of Statistical Pattern Classification
Types of Pattern Recognition Methods
Types of
information
Method of
recognition
Classification
(Categorization)
Matching
(Identification)
Attributes Relations
Statistical
Pattern Classification
Syntactic
Pattern Classification
Pattern Matching Computer Vision
Image Understanding
Recognition by Matching
【Signal Matching】 【Symbol Matching】 ① Template Matching ① Word Matching ② Elastic Matching ② DNA Analysis ③ Model Matching ③ String Pattern Matching “at” matches with “hat”, “cat”, “bat”, … ④ Unification unify(f(x), f(g(a)) x=g(a)
2)()|(minarg tsignaltmodel −θθ
Correlation
*)()()(:TransformFourier )()()(
:)( and )(between Function n Correlatio
ωωω GFYdstsgsfty
tgtf
=−= ∫∞
∞−
Correlation Function between Signals
∫∫∫
∫
∫
∞
∞−
∞
∞−
∞
∞−
∞
∞−
∞
∞−
−+=
+−=
−
dttgtfdttgdttf
dttgtgtftf
dttgtf
)()(2)()(
))()()(2)((
))()((
:sDifference Squared of Sum
22
22
2
Correlation Function: ∫∞
∞−
−= dstsmsgtr )()()(
Input signal to be processed
Target signal to be matched
x
Correlation Function: ∫∞
∞−
−= dstsmsgtr )()()(
a ta ma
a
Normalized Correlation Function
∫∞
∞−−= dstsgsfty )()()(
:y)(SimilaritFunction n Correlatio
∫ ∫
∫∫∫
∞
∞−
∞
∞−
∞
∞−
∞
∞−
∞
∞−
==
−−−
−−−=
dttgtgtgdttf
tftf
dsgtsgdsfsf
dsgtsgfsfty
2
2
2
2
22
*
||)(|||)(|,||)(||
|)(|
)()(
)()()(
:Functionn Correlatio Normalized
Invariant against biasing and scaling
∫∫
∫∫∫∫
∫∫
−−−
−−+=
−−−=
S
SS
S
dxdyyxtyxf
dxdyyxtdxdyyxf
dxdyyxtyxfD
),(),(2
),(),(
),(),(),(
rity)(dissimila images obetween tw Difference
22
2
βα
βα
βαβα
∫∫∫∫
∫∫
∫∫
−−−−
−−−−=
−−=
SS
S
S
dxdytyxtdxdyfyxf
dxdytyxtfyxfS
dxdyyxtyxfS
22
*
),(),(
),(),(),(
:Fucntionn Correlatio Normalized
),(),(),(
:y)(SimilaritFunction n Correlatio
βα
βαβα
βαβα
Image Processing by Correlation
Template Matching
),( yxf ),( yxt
Image Light Source P Light Source
Camera
3D scene
P
P’
P’’
Stereo Image Analysis
Finding the best matching point
• Resultant displacement is in units of pixels.
),( yxf
),( yxt
49
Depth Measurement by Triangulation The 3D depth of a scene point can be computed from a pair of matching image points in left and right images.
Baseline b θ1 θ2
?d Image plane of camera 2 Image Plane
of camera 1
l1 cosθ1 + l2 cosθ2 = b l1 sinθ1 = l2 sinθ2 = d
Eliminate l1, l2
d = b/(tan-1 θ1 + tan-1 θ2 )
l2 l1
Motion Analysis by Correlation
Observed 2D motion images T=2 T=F T=1
template image
Best matching position
Motion vector: (i0-x0, j0-y0)
Elastic (DP) Matching
Model
Signal
P1 P2 P3 P4 ・・・・・・・・・・・・・・・・・・・・・・・・ Pn
Qm ・ ・ ・ ・ ・ ・ ・ ・ ・ Q4 Q3 Q2 Q1
Mode l
Signal
Q1 matches with P1 and P2
Principle of Optimality: An optimal policy has the property that whatever the initial state and initial decision are, the remaining decisions must constitute an optimal policy with regard to the state resulting from the first decision.
),,,()],1([
),,,,()1,1([
),,,,()]1,([min)],([:),( cost to minimum
1
11
1
><><+−
><><+−−
><><+−=
−
−−
−
jiji
jiji
jiji
QPQPcjiPEQPQPcjiPE
QPQPcjiPEjiPEjiP
The optimal paths to
these points have been computed
1−iP iP
jQ
1−jQ
Report 3
∫∫
∫∫
−−=
−−=
S
dxdyyxtyxfS
dxdyyxtyxfC
),(),(),(
:n Correlatio
),(),(),(
:nConvolutiorelation. their Discuss
similar.look very n Correlatio andn Convolutio
S
βαβα
βαβα
Computational Scheme of Statistical Pattern Classification
Types of Pattern Recognition Methods
Types of
information
Method of
recognition
Classification
(Categorization)
Matching
(Identification)
Attributes Relations
Statistical
Pattern Classification
Syntactic
Pattern Classification
Pattern Matching Computer Vision
Image Understanding
Natural
Pattern Feature
Measurement
Feature Selection
(Extraction) Classification
Learning
Sample
Pattern (Training Sample)
Picture
Data
Feature
Set 1
Feature
Set 2 x =
x x
x
1
2
m
....
y =
y y
y
....
1
2
n
Class
Name
Feature Vector : x =
x x
x
1
2
n
....
measurement 1 measurement 2
....
measurement n
α
Architecture of Statistical Pattern Classification Systems
g 1
g 2
g c
x
x
x
1
2
d
MAX
g c (×)
(×) g 2
g 1
(×)
FEATURE DISCRIMINANT VECTOR FUNCTIONS
MAXIMUM DECISION
SELECTOR
α
Architecture of Pattern Classifiers
[1] Nearest Neighbor Classification
Class1
Class3 x1
x2
Class2
decision boundaries
Feature Vectors
X =
X1 X2 ・ ・ ・ Xn
heightweight ・ ・ ・ age
Basic Scheme of Nearest Neighbor Classification
[Q1]What distance measures?
Measuring Unit Problem
height(cm)
weight (Kg)
Unit Change
height(cm)
weight (g)
Non-isotropic distance measure based on the shape of data distribution
=
nx
xx
X2
1
Distance between vectors and
=
ny
yy
Y2
1
1. Euclidean Distance :
2. Distance :
3. Similarity :
4. Mahalanobis Distance :
1L
2/12
1])([ i
n
ii yx −∑
=
||1
i
n
ii yx −∑
=
YXYX
⋅⋅
=θcos
2/11 )]()[( MXMX t −− ∑−
(M : Mean Vector, ∑ : Covariance Matrix )
•
θX
Y
1
2
3
Distance Measures between a pair of feature vectors
=
px
xx
2
1
x
=
pµ
µµ
2
1
u
=∑pppp
p
p
i
σσσ
σσσσσσ
21
22221
11211
Mean Covariance Matrix
Parameter Estimation from sample set Nxxx ,, 21
∑=
=N
lili x
N 1
1µ pi ,,2,1 =
))((1
11
kkljjl
N
ljk xx
Nµµσ −−
−= ∑
=pkpj
,,2,1,,2,1
==
Mean and Covariance matrix of data distribution
2µ
2x
1µ 1x
•
•
•
•• ••
•
•
•
•••
••
• ••
•
•••
•••
•
•
•
•
•
(b) SCATTER DIAGRAM ( a ) BIVARIATE NORMAL DENSITY
Two representations of a normal density.
n-dimensional Normal Distribution
[Q2]Distance between which entities?
Distance to distribution centers
Class1
Class3 x1
x2
Class2
Feature vector
X =
X1 X2 ・ ・ ・ Xn
Decision boundary
Distance to sample data
• Decision rule: – Find k nearest neighbor sample
data. – Find the most popular class by
voting from the k nearest neighbor sample data.
1x 2xjx
nx
...
2ω
1ωInput feature vector x
n dimensional feature space
X
Distance by voting: k-nearest neighbor classification
Report 4
Compare the performance between 1. the nearest neighbor classification with the Mahalanobis distance and 2. the k-nearest neighbor classification in the following case.
[2]Statistical Pattern Classification
. is nature of state when theaction for taking incurred loss the)|(diagnoses i.e. actions, possible ofset finite the,,
classesobject i.e. nature, of states ofset finite the,,
1
1
iiii
a
s
aAs
ωαωαλααωω
==Ω
x).|(x)|(x)|( x)|(x)|(x)|(
casecategory two
2221212
2121111
ωλωλαωλωλα
PPRPPR
+=+=
).()|()(
where
,)(
)()|()|(
1i
s
ji
iii
Ppp
pPpp
ωω
ωωω
∑=
=
=
xx
xxx
)|()|()|( 1
xx jj
s
jii pRrisklconditiona ωωαλα ∑
=
=
統計的パターン分類(BAYES DECISION THEORY)
A priori probability
a posteriori probability
Bayesian Decision Rule
Probability distribution
)|( 1ωxp
x x
)|( 2ωxp
.,,2,1 allfor
)|( )|(ifonly and if X Decide
mjxpxp ji
i
=
≥∈
ωωω
Maximum-Likelihood Classifier
(regard p(x|ωi) as a function of ωi)
12E 21E
)|x()( 11 ωω pP )|x()( 22 ωω pP
B1R 2R
dxxpPE
dxxpPE
EEP
R
R
E
)|()(
)|()(
where
1121
2212
2112
2
1
ωω
ωω
∫
∫=
=
+=
Minimal Error Rate Classifier
. and estaimate todifficult isIt
data. sample from ,,, as wellas , Estimate
),N()|(
),N()|(
:onsDistributi Normal of Mixture Assume
data. sample from ,,, Estimate),N()|(
),N()|(:onDistributi Normal Assume
21
2121
1222
1111
2121
22
111
2
1
NNba
bp
ap
pp
ji
N
jjjj
N
iiii
ΣΣ
Σ=
Σ=
ΣΣΣ=Σ=
∑
∑
=
=
µµ
µω
µω
µµµωµω
x
x
xx
2
【2】
【1】
Estimation of Probability Distribution Function
Report 5
.Statistics andTheory Probality between sdifference Discuss
y.Explain whtraining.-over fitting,-over todue
effectivenot is functionson distributiy probabilit as data sample from computed histograms Using
[3]Linear Discriminant Function
g 1
g 2
g c
x
x
x
1
2
d
MAX
g c (×)
(×) g 2
g 1
(×)
FEATURE DISCRIMINANT VECTOR FUNCTIONS
MAXIMUM DECISION
SELECTOR
α
Architecture of Pattern Classifiers
).( ofsign on the baseddecision make and ,)( asit denoteLet
0)()()()()(BoundaryDecision
)()(Functionsnt DiscriminaLinear
0
212121
222111
xxwxxwwxxx
xwxxwx
gwgwwggg
wgwg
t
tt
tt
+=
=−+−=⇒=
+=+=
:
,:
Class1
Class2
x
g>0
g<0 g = 0
Two Class Linear Discriminant Function
Geometric Representation
).( ofregion positive the towardheading is
.||)( |,|)( ,0)( sinceThen
.||
)()||
()(
have which wefrom ,||
:follow as represent can Then we .0)( onto
of projection orthogonal thedenote Let .0)(by defined
hyperplane the toorthogonal is that means This0)( Then,
boundary.decision on the points denote and Let
0
210201
21
xwwxwxx
wwwx
wwxwx
wwxx
xxxx
xw
xxwxwxwxx
g
grrgg
rgwrg
r
g
g
ww
p
t
ppt
p
p
ttt
①
===
+=++=
+=
=
=
=−⇒+=+
g>0
g<0 g = 0
______|)(| xg|||| w
xW
|| 0w___|||| w
px
!system coordinate same in thedrepresente are vector feature theand t vector coefficien that theNote