A Computational Theory of Writer Recognition
Catalin I. TomaiDepartment of Computer Science and EngineeringDepartment of Computer Science and Engineering
Dissertation Defense
Outline
• Problem Domain• A Computational Theory of Writer Recognition • Algorithms and Representations
– Pictorial Similarity Examination– Identification of Document Elements
• Complete Information• Partial Information
– Extraction of Document Elements• Classifier Combination • Dynamic Classifier Selection
– Determination of Discriminating Elements– Pattern Examination and Classification
• Writer Verification• Writer Identification
• Conclusions
Writer Recognition
• Biometrics:– Physiological: face, iris pattern, fingerprints– Behavioral: voice, handwriting
• Forensic Sciences: Court testimony: Daubert vs. Merrell Dow (1993 -Supreme Court) – forensic techniques need to be based on testing, error rates,
peer review and acceptability
• Practiced by Forensic Document Examiners (FDE’s)– Experts perform significantly better than non-professionals
[Kam et. al 1994,1997]
• Semi-automatic computer-based systems:– FISH (Germany 1970), SCRIPT (Holland, 1994)– Used by: Government agencies (IRS,INS,FBI)
Handwriting Analysis Taxonomy
Handwriting Analysis
Synthesis Recognition Personality identification(Graphology)
On-line Off-line Writer VerificationWriter Identification
Natural Writing Forgery Disguised Writing
Handwriting Recognition
WriterRecognition
Text Dependent
Text Independent
• Writer Recognition
• Handwriting Recognition
Problem Domain
Identification Model
Verification Model
Which writer?1,…,n
Yes, Same Writer No, Different Writer
Recognition Model started
Problem Domain
Individuality: no two writings by different persons are identicalVariability: no two writings by the same person are identical
Writer A
Writer B
. . .
. . .
. . .
. . .
Problem Domain – Previous WorkAuthors Writer
Identification (%)
Writer Verification
(%)
Writer Sets Features Classification
[Steinke 1981]
99 - 20 writers/40 documents each
- -
[Said
1999]
96 - 40 writers/1000 documents
Textural KNN
[Zhu
2001]
95.7 - 17 writers/chinese alphabet
Textural Euclidean Classifier
[Srihari 2001]
89 95 1000 writers/3000 documents
Character and Document
Neural Nets
[Bensefia 2002]
97.7 - 88 writers Stroke-Based KNN
[Kam 1997]
- 93.5-professionals
61.7-nonprofessionals
144 pairs of documents
- 100 document examiners
- partial solutions, no integrated framework
- features do not reflect the experience of human examiners
- small number of documents/writers
- document elements recognition/extraction overlooked
Computational Theory of Writer Recognition
EvaluationAnalysis
Query Documents
Comparison
Likelihood Ratio
Writer Identity
2. Representation
1. Theory: developed based on studies on how human experts discriminate between
handwritings of different people
Determination of Discriminating
Elements
Pictorial Similarity
Examination
Comparison
Identification of Document Elements
Pattern Examination
Classification
ComparisonAnalysis Analysis Evaluation
2. Algorithms: pattern recognition/machine learning/computer vision
Document elements (characters/words)
Discriminating Elements (Habits)
Textural/Statistical/Structural features
Inspired by the Computational Theory of Computer Vision [Marr,1980]
3. Hardware/Software
Computational Theory of Writer Recognition
Determination of Discriminating
Elements
Pictorial Similarity
Examination
Query Documents
Comparison
Identification of Document Elements
Pattern Examination
Classification
Likelihood Ratio
ComparisonAnalysis Analysis Evaluation
Writer Identity
Pictorial SimilarityTheory : eliminate dissimilar candidates Representation: handwritten documents = texturesAlgorithms:
– Wavelets - humans process images in a multi-scale way. – Gabor Filters – reasonably model most spatial aspects of the visual cortex
Pre-processing
…
…
Gabor filter bank
θ – [0,45,90,135]
f – [2,4,8,16,32]
and combinations
DCT
Daubechies
Haar
Antonini
…
Odagard
What is the most descriptive Wavelet Transform/Filter for a given handwriting?
GLCM
1 60
…
12
28
Feature Vector
Pictorial Similarity
ApproximationAlgorithm[GLCM]
TrainExemplars Database
Applyalgorithms on
each image
MeanFeaturesDatabase
Training Process
Query Process
Query Document
QueryFeatures Database
Decision Fusion/Find BestAlgorithm
Query Exemplars Database
TestExemplars Database
Return Most Similar Exemplars
Features Database
……
A2
A1
A28
A3
A15
A7
…
A1
A8
A3…
…
Ranked Algorithmsfor each Handwriting Exemplar
Classsifier/
Class
Results
Algorithm Set Retrieval Performance
Adaptive Scheme vs.
Best Algorithm
Train Set: 1927 images (167 writers)Test Set: 1985 images (173 writers)
Results
Document a15 Adaptive
Scheme
Best
Scheme
Document a15 Adaptive
Scheme
Best
Scheme
0001 0.341 0.330 0.375 0023 0.348 0.500 0.500
0003 0.288 0.205 0.288 0025 0.280 0.265 0.280
0005 0.144 0.091 0.265 0027 0.182 0.197 0.227
0007 0.402 0.470 0.500 0029 0.220 0.212 0.227
0011 0.265 0.333 0.333 0031 0.023 0.053 0.053
0015 0.189 0.220 0.220 0033 0.295 0.439 0.455
0017 0.144 0.182 0.220 … … … …
0019 0.424 0.424 0.455 Average 0.258 0.288 0.320
0
10
20
30
40
50
60
70
a5 a6 a7 a8 a9 a10 a11 a12 a13 a14 a15 a16 a17 a18 a19
rank1
rank2
rank3
rank4
rank5
rank6
rank7
rank8
rank9
rank10
rank11
Algorithm
Frequency
Conclusion
– Filtering properties: Allows us to eliminate approx 70% of the test set exemplars (the ones non-similar to the query image).
– Performance: outperforms the use of a unique filter for all query images.
– Extensibility: more algorithms/decision schemes can be added to the mix.
– Limited overhead: by using pre-computed features and mean feature vectors database.
Computational Theory of Writer Recognition
Determination of Discriminating
Elements
Pictorial Similarity
Examination
Query Documents
Comparison
Identification of Document Elements
Pattern Examination
Classification
Likelihood Ratio
ComparisonAnalysis Analysis Evaluation
Writer Identity
Complete Information(Transcript Mapping)
Partial Information(Heuristic and Bayesian)
16
Country name Postal Code structure
Country name Postal Code Structure
Country name Postal Code structure
Canada LDL DLD Ireland Czech Republic DDDDD
England L/LA/ADLL Brasil DDDDD-DD Latvia DDDD
Germany DDDDD New Zealand DDDD Singapore DDDDDD
… … … … … …
Document Elements Extraction
Theory: Extract document elements (characters/words/other)
From Nov 10 1999 Jim Elder ….
CompleteInformation
Partial Information
+
From
Nov...
No Information
+
Mexico
+ ? From
...
Transcript
17
Complete Information-Transcript Mapping
. . .
From Nov 10 1999 Jim Elder …
Word Recognition
Build Word-Hypothesis Set
Next Line/Refine Results
Dynamic Programming
From
Nov
1999
WordRecognition
18
Transcript Mapping
19
Transcript Mapping
20
Results
From 3000 document images
– almost ½ million word images extracted
– more than 2 million character images extracted
0
10
20
30
40
50
60
8 9 4 2 k j b B E F
90-99
80 - 89
70-79
60 - 69
50 - 59
40 - 49
30 - 39
20 - 29
10 to 19
0 - 9
0
2
4
68
10
12
14
16
8 9 4 2 k j b B E F
ErrorError
Rate
Error
Rate
21
Computational Theory of Writer Recognition
Determination of Discriminating
Elements
Pictorial Similarity
Examination
Query Documents
Comparison
Identification of Document Elements
Pattern Examination
Classification
Likelihood Ratio
ComparisonAnalysis Analysis Evaluation
Writer Identity
Complete Information(Transcript Mapping)
Partial Information(Heuristic and Bayesian)
22
Document Element Extraction-Partial Information
Theory: Partial information is available
• combine heterogeneous information• missing data/noisiness• interpretability
Script
Structure
Partial recognition results
Example: Foreign Mail Recognition (FMR) - extract character/word images from mail pieces sent to foreign addresses
Mail Stream
23
Foreign Mail Recognition
Domestic Mail Recognition
(DMR)
Foreign Mail Recognition (FMR)
Lexicon Large Large
Postal Code Structure Small variation Large variation
Postal Code Length Usually Fixed Variable
Script Small set Large set
Address Elements Positioning
Small configuration set
Large configuration set
24
Partial Information-Foreign Mail-Samples
25
Partial Information – Heuristic Solution
PC candidate Position PC- Format
Country Confidence PC FormatFrance 3.04 dddddLatvia 4.58 dddd
75014 France
2:1:4 ddd
2:1:5 invalid
2:2:5 dd ddddd
FKANCE
REORDER/ELIMINATE
WMR
HNNCR
2:1:2 ddddd
Country Confidence PC-FormatFrance 2.04 dddddGreece 5.62 ddddd
Greece 5.62 ddddd …. ….. ...
... ... ...
Id Country
Configuration Frequency
Country
City State
PC
1 France 1,1,1 2,2,2
0,0,0
2,1,2 0.40
2 France 1,1,1 3,2,2
0,0,0
3,1,2 0.11
3 France 1,1,1 3,1,1
0,0,0
2,1,1 0.39
4 France 2,1,1 1,1,2
0,0,0
1,2,2 0.05
5 Greece 1,1,1 2,1,2
0,0,0
2,2,2 0.5
6 Greece 2,1,1 0,0,0
0,0,0
1,1,1 0.3
… … … … … … …
Country Candidates
26
Partial Information – Bayesian SolutionStructural Features:
2 No Of Lines3,4,5 No Of Strokes (Lines
1,2,3)8,9,10,11 Line Length (Lines
1,2,3)14 IsGap – gap on the last
line
Script Differentiation Features
12 IsLatin(0/1) 13 IsCKJ(0/1)
Address Block Component Features
15 PostalCodeLength16 PostalCodeLine – line on which the PostCode is
located18 PostalCodeFormat (e.g. ####,####,#a#a#a, etc)19 PostalCodeHasDash (e.g. ####-###)17 PostalCodeIsMixed(0/1) – is the Postcode of
digits only or not20 PostalCodeCountryOrder (0/1) – is the
PostCode located before/after the country name
1
2
3
4
Recognition Features
6 CountryNameFirstLetter (from character recognizers)
7 CountryNameLastLetter (from character recognizers)
21 Continent22 CountryName
27
Results
Method/Performance
HM-T
HM-NT
BNM-R
BNM-NR
Acceptance Rate
0.376
0.576 0.552 0.408
Rejection Rate 0.607
0.000 0.000 0.000
Error rate 0.017
0.424 0.448 0.592
• 9270 mail piece images• 22 country destinations (> 30 train set samples)
– almost 3090 word images extracted
28
Computational Theory of Writer Recognition
Determination of Discriminating
Elements
Pictorial Similarity
Examination
Query Documents
Comparison
Identification of Document Elements
Pattern Examination
Classification
Likelihood Ratio
ComparisonAnalysis Analysis Evaluation
Writer Identity
Complete Information(Transcript Mapping)
Partial Information(Foreign Mail)
Classifier Combination Classifier Selection
29
Classifier Combination – Decision Level
Input
C1
C2
…
CN
A7 A8 A3…AN
A8 A4
A7 A8 A3…AN
s7 s8 s3… sN
“Ensemble”Information
“Local” information
e1
e2
…eN
“Global” information
-heterogeneity-uncertainty
Global - [Xu et. al, 1998], [Zhang et. al 2003]Local: [Rogova et. al, 1997]Ensemble: BKS [Huang et. al, 1995]Feature and Classifier: FIM [Tahani et. al, 1990]
…
Dempster-Shafer-based unified framework for heterogeneous ensembles of classifiers
30
Classifier Combination
Motivation:– Current DST-based combination methods use only
global/local/ensemble information or combined local + global– Current solutions not always suitable for combining Type-III
classifiers (assume we have scores attached to each class)
Goals:– Adapt classic DST mass-computation methods for Type-III classifiers– Integrate “ensemble” information into the DS Theory of Evidence– Combine “local”, “global” and “ensemble” classifier information
into one unique framework– Estimate impact of affirming uncertainty regarding the top choice
(double hypotheses)
31
Classifier Combination-Adaptation to Type-III Classifiers Computation of Evidences
Classifier Level Class Level
Method 1 Method 2 Method 3
Use recognition/substitution rates for each top class
For each classifier ek the sum of masses for all classes add up to 1
Use recognition/substitution rates over all classes
Distance to a mean vector
Membership
function
Use uncertainty in the combination
32
Classifier Combination-Integrate BKS in DST
BKS
(Behavior
Knowledge
Space
- a unit of the K-dimensional knowledge space
- no. of training patterns for which T = Cm in
- no. of training patterns for the configuration
RKS
Problem: - as the number of classifiers increases, BKS becomes sparseSolution: - RKS (Reduced Knowledge Space)
- addresses sparseness - models joint behavior of recognizers, irrespective of the class
- set of groups of classifiers that agree on the top choice
- no of train patterns whose output configurations belong to L
- the classifiers set
- no of training patterns whose output configurations belong to L for which
- no of training patterns whose output configurations belong to L for which
33
Classifier Combination – Unified FrameworkDecision
Component Classifier C1
Global
information
Local
information
Component Classifier C2
Global
information
Local
information
Component Classifier CK
Global
information
Local
information
K classifiers:
Output:
“Global” performance:
“Local” performance:
Frame of Discernment:
)(, xeyRy kkM
k
…
Dempster-ShaferCombination Algorithm
Ensemble
information
Component ClassifiersC1 C2 … CK
&
34
Classifier Combination – Methods used• s1 - first classifier
• s2 - second classifier
• s3 - third classifier
• X – original method in [Xu et al 1998]• P – original method in [Parikh et al. 2001]
• R1 – original method in [Rogova el al. 1997], cosine measure
• R2 – original method in [Rogova el al. 1997], Euclidean distance based measure
• FIM – Fuzzy Integral Method [Tahani et. al 1990]• BKS – Behavior Knowledge Space [Huang et. al, 1995]
• M1 – “global” – recognition, substitution rates for each class
• M2 – “local” – sum of masses for all classes add up to 1
• M3 – “local” – use membership functions instead of distances to mean vectors
• RKS – Reduced Knowledge Space
• X+M2
• M2 +DH – M2+double hypotheses
• X+M2+DH – X+ M2 + double hypotheses
• M3+DH – M3+double hypotheses
• M2+BKS – M2+ ensemble BPA obtained from BKS
• M2+RKS – M2+ ensemble BPA obtained from RKS
• X+M2+BKS – X + M2 + ensemble BPA obtained from BKS
• X+M2+RKS – X + M2 + ensemble BPA obtained from RKS
35
Results
Local, Global,
Local + Global Information
75
80
85
90
95
100
s1 s2 s3
Recognizers
Re
co
gn
itio
n R
ate
96
96.5
97
97.5
98
98.5
X P R1 R2 M1 M2 M3 FIMX+M2
Methods
Re
co
gn
itio
n R
ate
e1+e2+e3
e1+e3
Original
Recognizer
Performance
36
Results
Method
e1+e2+e3 e1+e3
Recog. Rate(%)
Error Rate(%)
Reject Rate(%)
Recog. Rate(%)
Error Rate(%)
Reject Rate(%)
e1 94.47 4.73 0.08 94.47 4.73 0.08
e3 97.66 2.34 0.00 97.66 2.34 0.00
e2 83.96 16.04 0.00
BKS 97.86 1.49 0.65 97.81 2.16 0.03
RKS 97.21 2.79 0.00 97.66 2.34 0.00
Ensemble
Double Hypotheses
Method e1+e2+e3 e1+e3
Recog. Rate(%)
Error Rate(%)
Reject Rate(%)
Recog. Rate(%)
Error Rate(%)
Reject Rate(%)
e1 94.47 4.73 0.08 94.47 4.73 0.08
e3 97.66 2.34 0.00 97.66 2.34 0.00
e2 83.96 16.04 0.00
M2+DH 98.14 1.86 0.00 97.96 2.04 0.00
X+M2+DH
98.05 1.95 0.00 97.73 2.27 0.00
37
Results
Local + Global + Ensemble Information
97.697.797.897.9
9898.198.298.3
e1+e2+e3
e1+e3
38
Computational Theory of Writer Recognition
Determination of Discriminating
Elements
Pictorial Similarity
Examination
Query Documents
Comparison
Identification of Document Elements
Pattern Examination
Classification
Likelihood Ratio
ComparisonAnalysis Analysis Evaluation
Writer Identity
Complete Information(Transcript Mapping)
Partial Information(Foreign Mail)
Classifier Combination Classifier Selection
39
Classifier Selection- When to use which Classifier?
Dynamic ClassifierSelection
Input A1,A2,…,AK
C1,C2,…,Cn
Class Set
Cj
[Woods et. al, 1997]
[Kanai et. al, 1997]…
[Xue et. al, 2002]…
Classifiers “Best” Classifier
Decision
Goal: Choose the classifier based on the class set (alphabet) size and composition
40
Classifier Selection
• How to measure the confusability of an alphabet?– recognizer confusion matrices [Kanai et al, 1994] , deformation
energy of elastic nets [Revow et al, 1996] , image-to-image distances [Huttenlocher et al, 2002], degradation model [Baird et. al 1993] –no “perfect” handwriting sample
• Drawbacks: classifier dependency and vulnerability to outliers• Approach: eliminate outliers and variances of shape by looking
at the character skeletons
IranIraqZair
IranOran≠Word Lexicons:
Alphabets o O, PD,t≠
size,“confusability”
ac
vwt
edit-distance
?
41
Classifier Selection
• Structural Features [Xue et. al, 2002]– Loop, Cusp, Arc, Circle, Cross, Gap - with different
attributes: Position, Orientation, Angle
Extract structural features from each character and build a profile HMM model for each character (of different sizes)
Match Match
Insert Insert
EndBegin
Delete
Upward Arc Upward CuspLoop
Downward Arc
42
Classifier Selection
HMM model for Digit ’2’ Emission Probabilities for State 3
43
Classifier Selection
Characters Confusability (Similarity) : distance between their corresponding HMM models– measure the probability that two models generate the same sequence
Match Match
Insert Insert
EndBegin
Delete
Match Match
Insert Insert
EndBegin
Delete
M1 M2
Co-emission probability
Alphabet confusability
c1 c2 s(c1c2)0 O 0.9862
71 I 0.9840
3U V 0.9103
3… … …
ac
vwt
Alphabet APairs of Characters Confusability ranking
+
44
Performance
Results obtained using the R1-based confusability measure
N CI R1
(%)
R2
(%)
R3
(%)
DCS
(%)
N CI R1
(%)
R2
(%)
R3
(%)
DCS
(%)
3 1 97.29 97.77 95.99 97.29 7 1 94.78 95.51 92.03 95.51
3 2 97.72 96.96 96.72 96.96 7 2 93.97 94.50 91.79 94.50
3 3 95.64 96.36 96.04 96.36 7 3 93.93 93.97 92.19 93.97
3 4 93.02 94.11 93.31 94.11 7 4 93.42 94.01 91.87 94.01
5 1 96.40 96.03 93.77 96.03 Average performance
5 2 95.59 95.55 93.89 95.55 95.00 95.38 93.68 95.33
5 3 94.82 94.78 93.85 94.78
5 4 94.42 94.94 92.72 94.94
94
95
96
97
98
99
100
1 2 3 4
R1
R2
R392
94
96
98
100
1 2 3 4
R1
R2
R3
Alphabet Size 5 Alphabet Size 7
Confusability Interval Confusability Interval
Train
Test
45
Performance
N CI R1
(%)
R2
(%)
R3
(%)
DCS
(%)
N CI R1
(%)
R2
(%)
R3
(%)
DCS
(%)
3 1 97.05 96.97 96.89 96.97 7 1 94.10 94.26 91.92 94.26
3 2 97.41 97.53 96.08 97.41 7 2 93.94 94.50 91.51 94.50
3 3 96.20 96.40 94.99 96.40 7 3 94.26 94.30 91.72 94.30
3 4 97.01 97.01 95.47 97.01 7 4 94.30 94.18 91.96 94.30
5 1 95.39 95.63 94.18 95.63 Average performance
5 2 94.83 95.43 93.29 95.43 95.43 95.57 93.77 95.61
5 3 95.15 95.51 93.45 95.51
5 4 95.51 95.15 93.81 95.51
Results obtained using the HMM-based confusability measure
94
95
96
97
98
99
1 2 3 4
R1
R2
R3
92
94
96
98
100
1 2 3 4
R1
R2
R3
Confusability IntervalConfusability Interval
Alphabet Size 5 Alphabet Size 7
Train
Test
46
Computational Theory of Writer Recognition
Determination of Discriminating
Elements
Pictorial Similarity
Examination
Query Documents
Comparison
Identification of Document Elements
Pattern Examination
Classification
Likelihood Ratio
ComparisonAnalysis Analysis Evaluation
Writer Identity
Document ElementsWord and Document
Features
47
Character Discriminability
Di
ww,
Di
wx,
• Theory: Individuality is exhibited by writers in the execution of more complex forms
• Characters/Words differ in their discriminability power
For each character:
Distance Space
w1-w1
w2-w2
w3-w3
…wn-wn
w1-w2
w2-w3
...wi-wj
(μ,σ)
48
Character Discriminability
di
B
Di
ww,
Discriminability:– Bhattacharya distance– Area below Receiver Operating Curve (ROC)
w1-w1
w2-w2
w3-w3
…wn-wn
w1-w2
w2-w3
...wi-wj
49
Character Discriminabilityrank(c
)Character
c
1 G
2 B
3 N
4 I
5 K
6 J
5 W
6 D
7 h
8 F
9 r
10 H
11 B
…
62 1Discriminabilityranking of characters by their Bhattacharya distance/ROC area between the SW and DW distance distributions
Writer Verification
50
Word DiscriminabilityWord w d(w)
Queensberry
0.560
Allentown 0.356
Virginia 0.302
Grant 0.287
Parkway 0.282
Street 0.237
From 0.205
Elder 0.203
West 0.171
… …
10 0.002
Discriminability ranking of thefirst 25 words of the CEDAR letter
Queensberry Allentown
Virginia Grant Parkway
- length of word
Word Discriminability
Queensber r y
012345
WM R GSCW SC SCON
Al lentown
012345
WMR GSCW SC SCON
Virginia
01
23
45
WMR GSCW SC SCON
Grant
0
1
2
3
4
5
WMR GSCW SC SCON
51
Group-dependent Character Discriminability
Handwriting: Influenced by Age/Handedness/Gender?
In some cases partial information about the writer of the query document is available from other sources (e.g. gender, age group, etc)
what is the discriminability of characters written by writers of a certain group?
Males Females Bachelor High School LH RH Under 24
Above 45
C s C s C s C s C s C s C s C s
G 0.58 G 0.55 D 0.55 N 0.59 J 0.55 G 0.48 G 0.60
N 0.53
I 0.60 W 0.57 I 0.56 G 0.60 D 0.56 h 0.50 d 0.61
D 0.56
h 0.60 I 0.58 N 0.60 I 0.61 h 0.56 I 0.53 J 0.62
G 0.56
b 0.60 D 0.59 b 0.60 F 0.63 S 0.57 A 0.53 I 0.63
M 0.57
J 0.61 w 0.59 G 0.61 W 0.63 b 0.60 b 0.55 W 0.63
W 0.57
y 0.62 N 0.60 W 0.61 D 0.64 f 0.60 H 0.56 b 0.63
F 0.57
Group-dependent discriminability ranking of characters
52
Group-dependent Character Discriminability
G
01234567
N
01234567
I
0123456
F
0123456
W
0123456
b
0123456
J
0123456
53
Accumulated Writer Verification Performance
Accumulated Writer Verification
Performance for different groups
Base Case - letters are considered in alphabetical orderOther cases - letters are considered in decreasing order of discriminability
54
Computational Theory of Writer Recognition
Determination of Discriminating
Elements
Pictorial Similarity
Examination
Query Documents
Comparison
Identification of Document Elements
Pattern Examination
Classification
Likelihood Ratio
ComparisonAnalysis Analysis Evaluation
Writer Identity
Document ElementsWord and Document
Features
55
Word and Document Features
• Theory: Use interpretable features • Current features:
– holistic view, without considering the relationships between document elements
– mostly content-dependent
• Proposed Features– Word Features– Document features:
• Lexeme (writing style)• Lexeme (writing style) context• Relative character proportions – height and slant• Handwriting legibility +inter-character distance
56
Curvature-based Word Feature
Original Image
ExteriorContour
Extraction
Curvature distribution
Curvature Computation
… …
… …‘o’
DTW
…3.45 …
Curvature: how much the curve "bends" at each point
- information preserving feature - rotation invariant.
/,
)(
22/322
22
rrrr
rrrr
57
Results – Word Features
Writer verification
58
Lexeme Distribution
Theory: writers use one or more forms for each character
Soft ClusteringHard Clustering
…
ratkowskyscottmarriottballtrcovwtracewfriedmanrubinssilikelihoodcalinskidbcindexhartigan
Clustering
Validity Indices
+
Lexemes
59
Lexeme Distribution
…
‘1’ ‘4’
1 2 3
4
‘0’
…01a 0
2a 11a 1
2a 41a 4
2a 43a 4
4a
Lexemes
;0)4(1 n ;0)4(2 n ;0)4(3 n ;3)4(4 n
Writer W2
1 2 3
4
Writer W1
… …01a 0
2a 11a 1
2a 41a 4
2a 43a4
4a
‘1’ ‘4’‘0’
)(
)()(
i
ilil cn
cnca
;0)4(1 n ;2)4(2 n ;1)4(3 n ;0)4(4 n ;3)4( n
22
21
221
21 ||),(||||),(||
||),(),(||),(
cDacDa
cDacDaDDd
cc
)4,,( 12 DWa
)4,,( 11 DWa
Lexemes
60
Lexeme Context Distributions
223322
3333
22
11
22
1133
44
…
11 22 33…… 6/106/10
dd
…
00 11 22 33
…
ZZ
… …
ee ll oo…nn
… …
2/102/10 …
11 22 33 44
44
Theory: neighboring allograph shapes influence a given allograph
)),(),((()),(),(((),( '2
)(
0
'12
)(
1121
'
'
' cDbcDbcDacDaDDd j
cNL
jj
ccl
cNL
ll
cc
aa
…
a
b
61
9.1 8.8 8.1 8.2 8.5 8.9 9.1 7.9
8.8 9.1 8.7 8.6 8.2 8.5 8.1 8.3
8.3 7.9 8.8 8.8
Avg. score: 8.6
Character/Word Legibility
…
Writer A
……
……
10.3 11.312.5 13.7
Writer B
8.9 7.7 7.9 8.5 8.6 5.7 7.8 7.9 8.1
7.6 7.9 7.0 7.1 8.0 6.8 7.9
7. 7 7.7 7.7
Average score : 7.2
…
…
……
9.5 8.210.1 12.5
Avg. Character LegibilitiesAvg. Character Legibilities
|)()(
|),(2
)(
1
1
)(
121
21
Dn
o
Dn
oDDd
i
Dn
j
ij
ij
i
Dn
j
ij
ij
ww
ii
62
Inter-Character Distance
Word W1
123
1ca
Word W2
123
1cb
13
12
)1(3
11 )()1(
1
)()1(
1)2,1(
N
c
ic
i
N
c
ic
i
bWareaN
aWareaN
WWd
63
Relative Slant and Height• Theory: Individuality is given mostly by relative not absolute measures like size
and slant
• ratios of letters are maintained despite changes in size, speed or intent of writing (normal or disguised)
a
d
ascenders: l,d,h
descenders: g y
n none: x a n
…
…
aa ad UU
hpp
1 2 9
p positive angle
n negative angle
ax-pp
xx-pn
hpn hnp hnn hpp hpn hnp hnn
Example:
hpp hpn hnp hnn
Compute RelativeSlant/Height
U Uppercase: A,B
NiDn
rh
i
ji
ij
i ,1;)(
,
64
Experimental Settings
• Data Set: 3000 documents written by 1000 Writers, representative of the US population
• Writer Verification:– Train/Test set: 1,500 same-writer document pairs, 1,500
different-writer document pairs.
• Writer Identification:– Test Set: 1000 documents (1000 writers)– Train Set: 2000 documents(1000 writers)
Same Content Different ContentCEDAR Letter
65
Results – Individual features
020406080100
Same Content
Different Content
05
101520
Same Content
Different Content
Writer
verification
Writer
identification
66
Results –Accumulated Features
Features Writer Verification Writer Identification
Same Content(%)
Different Content(%)
Same Content(%)
Different Content(%)
Original(12) 93.07 90.40 64.18 11.34
Proposed(6) 80.50 80.30 35.62 16.04
Features Writer Verification
Same Content(%)
Different Content(%)
Original+Proposed 93.77 90.60
Original+Characters 93.77 90.23
Proposed+Characters 91.47 84.23
Original+Proposed+Characters
93.90 94.30
67
Results – Writer Identification - Accumulated Features
CMC curves for original+proposed+character features
Computational Theory of Writer Recognition
Determination of Discriminating
Elements
Pictorial Similarity
Examination
Query Documents
Comparison
Identification of Document Elements
Pattern Examination
Classification
Likelihood Ratio
ComparisonAnalysis Analysis Evaluation
Writer Identity
P: GNB
NP: KNN
Verification Identification
-
NP: KNN P:GNB, BNWriter independent
Writer dependent
Classification Models
• Theory: “The Bayesian approach is the best measure we have for assessing the value of evidence” [Aitken 1986] -> likelihood must be returned !
• Challenges:– variable number of features – model interpretation LR > 1 same writer
LR < 1 different writer
K
k
M
m
M
n
jkn
ikm
jkn
ikm
i j
DWssdpSWssdpLLR1 1 1
,,,, )))|),((ln()|),(((ln(
),(),|( ,,,,,, tkrtkrtrkt NCdp
),|()|()|( ,,1
,, trktt
R
rtrkt CdpCPCdp
k
Classification Models
Proposed approach–Verification:
– Non-Parametric: KNN - character/document features– Parametric:
–Gaussian Naïve Bayes – character features–Bayesian Networks – document features
–Identification – Non-Parametric: KNN - character features
Writer Verification Model - Train Process
di
B
Di
ww,
Di
wx,
...
Any Character
WriterWk
...
Writer-Independent
DWSW
...
...
Writer-Dependent
DWSW
TrainWriters
Set
Any Character
di
B
wk-wk
wk-w2
wk-w3
...wk-wj
Writer Verification Model-Test Process
...
‘0’
d(Wk ,Wx )
...
...
...….
‘Z’
N
iiSW cSWpP
1
)|(
‘SW’ ‘DW’
…
‘SW’ ‘DW’
…
……
‘0’ ‘1’ ‘Z’
N
iiDW cDWpP
1
)|(
LikelihoodRatio
WriterWk
Writer Verification-Document features
Id Feature Name Node Type
Id Feature Name Node Type
1 SW/DW discrete 7 Number of vertical slope components
continuous
2 Entropy continuous
8 Number of horizontal slope components
continuous
3 Gray-level threshold continuous
9 Number of negative slope components
continuous
4 Number of black pixels continuous
10
Number of positive slope components
continuous
5 Number of interior contours
continuous
11
Slant continuous
6 Number of exterior contours
continuous
12
Height continuous
13
Word gap continuous
Results-Writer Verification
Writer Independent Writer Dependent
GNB KNN GNB KNN
No. of Mixture
s
Accuracy
No. of Neighbor
s
Accuracy
No. of Mixture
s
Accuracy
No. of Neighbor
s
Accuracy
1 93.76 3 95.30 1 79.39 3 84.68
2 94.26 5 95.93 2 80.37 5 85.50
3 94.33 7 95.80 3 80.78 7 85.89
4 94.40
variable 94.30 variable 81.09
Character features
Writer independent
GNB BN KNN
93.96 93.93 90.60
Document features
Conclusion
– A framework for Writer Recognition in the form of a Computational Theory
– Adaptive handwriting retrieval system for Pictorial Similarity Examination
– Algorithms for extracting characters/words from documents for complete and partial information scenarios
– Framework for combining global, local and ensemble of classifiers information using the DS Theory of Evidence
– Classifier selection scheme based on an alphabet confusability measure based on distance between structure-based HMM models of characters
– Character and word ranking by their authorship discriminability– Parametric and Non-parametric models for writer recognition
Publications
Writer Recognition• Catalin I. Tomai, Devika Ksirhagar and Sargur N. Srihari "Group
Discriminatory power of handwritten characters" in Proceedings of SPIE, Document Recognition and Retrieval XI,San Jose, California, USA, Jan 18-22 2003
• Catalin I. Tomai , Bin Zhang and Sargur N. Srihari, "Discriminatory power of handwritten words for Writer Recognition“ in Proceedings of ICPR'04
• Sargur N. Srihari , Catalin I. Tomai, Bin Zhang and Sanjik Lee, Individuality of Numerals in Proceedings of ICDAR'03, Edinburgh, Scotland, August 2002
• Sargur N. Srihari, Bin Zhang Catalin I. Tomai, Sanjik Lee, Zhixin Shi and Yong-Chul Shin "A System for Handwriting Matching and Recognition" in Proceedings of the 2003 Symposium on Document Image Understanding Technology (SDIUT'03),Greenbelt, Maryland, April 2003
• Sargur N. Srihari, S. Lee, Bin Zhang and Catalin I. Tomai, “Recognition-based System for Handwriting Verification and Identification” , in Proceedings of the Intl. Graphonomics Conference (IGS'03) Scottsdale, Arizona, 2-5 November 2003
• Sargur N. Srihari, Catalin I. Tomai and Sanjik Lee, "Quantitative Assesment of Handwriting Individuality" in Proceedings of the Fifth International Conference On Advances In Pattern Recognition (ICAPR-2003),Calcutta, India, December 10-13, 2003 invited paper
• Sargur N. Srihari, Anantharaman Ganesh, Catalin I. Tomai and Yong-Chul-Shin "Information Retrieval System for Handwritten Documents" in Proceedings of DAS'04
Publications
Handwriting recognition• Catalin I. Tomai, Kristin Allen and Sargur N. Srihari, Foreign Mail
Recognition, in Proceedings of ICDAR’01, Seattle, WA, September 2001
Classifier Combination• Catalin I. Tomai and Sargur N. Srihari, Combination of Type-III classifiers
using DS Theory of Evidence, Proceedings of ICDAR'03, Edinburgh, Scotland, August 2002
Digital Libraries• Catalin I. Tomai, Bin Zhang and Venu Govindaraju, Transcript Mapping for
Historical Documents, in Proceedings of IWFHR’02, Niagara On The Lake, 2002
• Bin Zhang, Catalin I. Tomai, Venu Govindaraju and Sargur N. Srihari, Construction of Handwriting Databases Using Transcript-based Mapping" in Proceedings of the Intl. Workshop on Document Image Analysis for Libraries 2004 (DIAL'04),January 23-24, 2004, PARC, Palo Alto, CA, USA
Image Retrieval• Bin Zhang, Catalin I. Tomai and Aidong Zhang, Adaptive Texture Image
Retrieval in Transform Domain , in Proceedings of ICME’02, Lausanne, Switzerland, September 2002
• Bin Zhang, Catalin I. Tomai and Aidong Zhang, An Adaptive Texture-Information Retrieval System using Wavelets, in Proceedings of the Seventh International Conference on Control, Automation, Robotics and Vision (ICARCV 2002),Singapore, December 2002
Thank You
Complete Information-Transcript Mapping
From (0.73)
10 (0.00)
Nov (0.36)
1999 (0.77). . .
Nov (0.88)
Jim (0.37)
1999 (0.77)
From (0.73)
From Nov 10 1999 Jim Elder …
Word Recognition
Build Word-Hypothesis Set
Next Line/Refine Results
Dynamic Programming
10 (0.26)
Nov (0.42)
From
Nov
1999
WordRecognition
Results
Query Image Image 1 Image 2 Image3
0005b-0
005a-2 0339b-2
0021a-0
0013a-0
0013a-1
0313c-2
0015b-0
0059b-0
0145c-3
0237c-3
0145c-1
Computational Theory of Writer Recognition
• Inspired by the Computational Theory of Computer Vision [Marr,1980]
• Three Levels:– Theory - developed based on studies on how human
experts discriminate between handwritings of different people
• Analysis, Comparison, Evaluation (Law of the ACE’s)– Algorithms and Representation
• pattern recognition/machine learning/computer vision• Document elements: character/words• Discriminating elements: elements of handwriting that vary
measurably with its author– Hardware/Software
Discussion
– Proposed a DST-based framework for combining global, local and ensemble of classifiers information which outperforms the FIM method
– Adapted classic BPA-computation methods to combine Type-III recognizers
– Integrated “ensemble of classifiers” information into the DST Theory of evidence
– Investigated the use of double hypotheses when combining recognizers