Recognition of Cursive Roman Handwriting – Past, Present and Future H. Bunke [email protected]Department of Computer Science, University of Bern Neubr ¨ uckstrasse 10, CH-3012 Bern, Switzerland Acknowledgments: - S. G ¨ unter, T. Varga, M. Zimmermann - Swiss National Science Foundation (20-5287.97 and IM2) Recognition of Cursive Roman Handwriting – Past, Present and Future – p.1/61
61
Embed
Recognition of Cursive Roman Handwriting – Past, Present ... · optical character recognition (OCR) HH HH HH Oriental Script Roman Script HH HHH machine printed text handwritten
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Recognition of Cursive Roman Handwriting– Past, Present and Future
• cursively handwritten text− Senior/Robinson, PAMI 1998− Elliman/Sherkat, ICDAR 2001− IAM, collection in progress (since about 1997)
Recognition of Cursive Roman Handwriting – Past, Present and Future – p.31/61
Some Details of the IAM Database
• more than 1,500 scanned pages of handwritten text
• material from over 600 individual writers− 95,000 correctly segmented words− over 13,000 lines of text− over 5,000 complete sentences
• covering a vocabulary of over 12,000 words
• ground truth and lexical tags available (LOB corpus)
Recognition of Cursive Roman Handwriting – Past, Present and Future – p.32/61
Some Details of the IAM Database (2)
Recognition of Cursive Roman Handwriting – Past, Present and Future – p.33/61
Some Details of the IAM Database (3)
Recognition of Cursive Roman Handwriting – Past, Present and Future – p.34/61
Multiple Classifier Systems
• motivation: use a group of experts rather than a single expert
• many approaches to handwriting recognition have been proposed usingmcs’s
• often the basic classifiers are constructed ’by hand’
• recently so-called ensemble methods have been proposed:− they require only a single classifier to be constructed by hand− the classifier ensemble is generated automatically
Recognition of Cursive Roman Handwriting – Past, Present and Future – p.35/61
Multiple Classifier Systems (2)
"classical" approach
input resultcombiner
nc
1c
Recognition of Cursive Roman Handwriting – Past, Present and Future – p.36/61
Multiple Classifier Systems (3)
c1
cn
combiner resultinput
ensemble method
generateautomatically
base classifier
c
Recognition of Cursive Roman Handwriting – Past, Present and Future – p.37/61
Recognition of Cursive Roman Handwriting – Past, Present and Future – p.38/61
Some Results
recognition rates achieved by various ensemble generation methods
algorithm recognition rate
Bagging 68.11%
AdaBoost 68.67%
random subspace 67.35%
feature selection 71.58%
original classifier 66.23%
Recognition of Cursive Roman Handwriting – Past, Present and Future – p.39/61
Synthetic Generation of Training Data
• all recognizers need to be trained
• the larger the training set, the better the performance("you never have enough training data")
• but collection of training data is expensive
• previous work on generation of synthetic training data:− machine printed OCR [Baird et al.]− Arabic and Chinese OCR− isolated characters− (synthetic handwriting for other purposes [Guyon, Plamondon])
Recognition of Cursive Roman Handwriting – Past, Present and Future – p.40/61
Synthetic Generation of Training Data
• no work on synthetic training data generation for cursive Romanhandwriting recognition
• two approaches:− using templates− applying geometric distortions to existing handwritten text
Recognition of Cursive Roman Handwriting – Past, Present and Future – p.41/61
Synthetic Handwriting from Templates
• templates extracted from forms
• templates extracted from running text, using HMM in forced alignmentmode
Recognition of Cursive Roman Handwriting – Past, Present and Future – p.42/61
Synthetic Handwriting from Templates (2)
Recognition of Cursive Roman Handwriting – Past, Present and Future – p.43/61
Synthetic Handwriting from Templates (3)
• disadvantages:− all instances of a character are identical− no ligatures
Recognition of Cursive Roman Handwriting – Past, Present and Future – p.44/61
Synthetic Handwriting from N-Grams
• compile a list of frequent 3- and 2-tuples from an electronic corpus
• extract templates of these tuples from a handwritten text, using forcedalignment
• split the given text into available tuples and generate the synthetichandwriting
Recognition of Cursive Roman Handwriting – Past, Present and Future – p.45/61
Synthetic Handwriting from N-Grams (2)
Recognition of Cursive Roman Handwriting – Past, Present and Future – p.46/61
Some Results
0 1 2 3 4 560
62
64
66
68
70
72
74
training set
reco
gniti
on r
ate
[%]
• 1193 word instances; 16 writers; 357 word vocabulary
• 1 = natural training data2 = synthetic training data3 = synthetic training data4 = synthetic training data
• test data: always natural
• except for the training data (natural/synthetic) identical conditions for allexperiments (same training/test words; same size of training/test set etc.)
Recognition of Cursive Roman Handwriting – Past, Present and Future – p.47/61
Future Perspectives
• some random comments:
− MCS’s− synthetic training data− enhanced HMMs (for example, 2D)− enhanced language models− etc.
Recognition of Cursive Roman Handwriting – Past, Present and Future – p.48/61
Future Perspectives
• to reach a new quality of recognition we need to go from text transcriptionto text understanding:
− include syntactic and semantic text analysis− include task specific knowledge (in addition to statistical parameter
estimation)
Recognition of Cursive Roman Handwriting – Past, Present and Future – p.49/61
Who can read this?
Recognition of Cursive Roman Handwriting – Past, Present and Future – p.50/61
Who can read this?
When I was in high school, my physics teacher - whose namewas Mr. Bader - called me down one day after physics classand said, "You look bored; I want to tell you something inte-resting." Then he told me something which I found fascina-ting, and have, since then, always found fascinating....The subject # is this - the principle of least action.Richard P. Feynman: The Feynman Lectures, Volume II.
Recognition of Cursive Roman Handwriting – Past, Present and Future – p.51/61
Who can read this?
Recognition of Cursive Roman Handwriting – Past, Present and Future – p.52/61
Who can read this?
Középiskolás koromban, egy nap a fizikatanárom - Bader úrnakívták - magához hívott fizikaóra után és azt mondta: "Unott-nak látszol; szeretnék mondani neked valami érdekeset." Majdelmondott valamit, amit elbûvölõnek találtam, és az-óta is mindig elbûvölõnek találom ... A legkisebb hatáselvérõl van szó.Richard P. Feynman: The Feynman Lectures, Volume II.
Recognition of Cursive Roman Handwriting – Past, Present and Future – p.53/61
Integration of Grammatical Knowledge
• prerequisites:
− a word sequence recognizer that produces an n-best list (see before)− a stochastic context free grammar− a parser to compute the probability of a sentence or the most
probable parse tree
• procedure:
− reorder the n-best list from the recognizer taking parse probabilitiesinto account
final score = recognition score + γ f(parse probability)
where γ is a normalization factor and f(.) is a normalization function
Recognition of Cursive Roman Handwriting – Past, Present and Future – p.54/61
Example of Grammatical Knowledge Integration
Rank Recognition Score Candidate Sentence
1 23923.6 She has put up the value other money .
2 23921.8 She has put up the value of her money .
3 23890.3 She had put up the value other money .
4 23888.4 She had put up the value of her money .
5 23854.3 She has put up the value at her money .
Rank Parse Prob. Candidate Sentence
1 1.58352e-19 She had put up the value of her money .
2 4.62861e-20 She has put up the value of her money .
3 1.12458e-21 She has put up the value at her money .
4 2.63105e-22 She had put up the value other money .
5 7.69052e-23 She has put up the value other money .
Recognition of Cursive Roman Handwriting – Past, Present and Future – p.55/61
Example of Grammatical Knowledge Integration
Rank Recognition Score Candidate Sentence
1 23923.6 She has put up the value other money .
2 23921.8 She has put up the value of her money .
3 23890.3 She had put up the value other money .
4 23888.4 She had put up the value of her money .
5 23854.3 She has put up the value at her money .
Rank Parse Prob. Candidate Sentence
1 1.58352e-19 She had put up the value of her money .
2 4.62861e-20 She has put up the value of her money .
3 1.12458e-21 She has put up the value at her money .
4 2.63105e-22 She had put up the value other money .
5 7.69052e-23 She has put up the value other money .
Recognition of Cursive Roman Handwriting – Past, Present and Future – p.56/61
Some Experimental Results
6
8
10
12
14
16
18
20
22
24
26
28
30
32
34
0 10 20 30 40 50 60 70 80 90 100
Sen
tenc
e R
ecog
nitio
n R
ate
[%]
Rank [n]
Reordered 100-Best ListBaseline System
Recognition of Cursive Roman Handwriting – Past, Present and Future – p.57/61
Future Challenge
• to deal with human factors (i.e. errors and abnormalities introduced byhumans)
− statistical modeling has proven very useful− however we also need to incorporate task specific knowledge
provided by human experts
Recognition of Cursive Roman Handwriting – Past, Present and Future – p.58/61
Sample Check Images
Recognition of Cursive Roman Handwriting – Past, Present and Future – p.59/61
Sample Check Images (2)
Recognition of Cursive Roman Handwriting – Past, Present and Future – p.60/61
Conclusions
• the recognition of cursive Roman handwriting has been a subject ofresearch for several decades
• for specific tasks some level of maturity has been reached andcommercial systems have become available
• some other tasks, particularly the recognition of unconstrained generaltext, need much more research
• these tasks are interesting for practical applications
• there do exist promising directions to further develop the field
Recognition of Cursive Roman Handwriting – Past, Present and Future – p.61/61