Combining Committee-Based Semi-supervised and Active Learning and Its Application to Handwritten Digits Recognition

Overview Semi-Supervised Learning(SSL) Single-View CoBC Experimental Results Conclusion Future Work

Combining Committee-basedSemi-supervised and Active Learning and

Its Application to Handwritten DigitsRecognition

Mohamed Farouk Abdel Hady, Friedhelm Schwenker

Institute of Neural Information ProcessingUniversity of Ulm, Germany

{mohamed.abdel-hady|friedhelm.schwenker}@uni-ulm.de

April 8, 2010

1 / 24


Overview

2 / 24


Semi-Supervised Learning

In many domains, the amount of training examples is largebut unlabeled.Data labeling process is often tedious, expensive andtime consuming because it requires the effort of humanexperts.Research directions of SSL

Semi-Supervised ClusteringSemi-Supervised ClassificationSemi-Supervised RegressionSemi-Supervised Dimensionality Reduction

3 / 24


Semi-Supervised Learning

Description SSL algorithmSingle-view, Single-learner EM (Nigam and Ghani, 2000)

Single-classifier Self-Training (Nigam and Ghani, 2000)Multi-view, Single-learner Co-EM (Nigam and Ghani, 2000)

Multiple classifiers Co-Training (Blum and Mitchell, COLT’98)Single-view, Multi-learner Statistical Co-Learning (Goldman et al., 2000)

Multiple classifiers Democratic Co-Learning (Y. Zhou et al., 2004)Single-view, Single-learner Tri-Training (Z.-H. Zhou, TKDE’05)

Multiple classifiers Co-Forest (Li and Z.-H. Zhou, TSMC’07)Co-Training by Committee

Z.-H. Zhou and M. Li, Semi-supervised learning by disagreement, Knowledge and

Information Systems, in press.

4 / 24


How can unlabeled data be helpful?

5 / 24


How can unlabeled data be helpful?

6 / 24


Self-Training

But the most confident examples often lie away from the targetdecision boundary (non informative examples). Therefore, inmany cases this process does not create representativetraining sets as it selects non informative examples.

7 / 24


Multi-View Co-Training

Blum and Mitchell (1998)As any multi-view learning algorithm, it requires that eachtraining example is represented by multiple sufficient andredundant views,i.e. two or more sets of features that are conditionallyindependent given the class label and each is sufficient forlearning.For web page classification: 1) the text appearing on thepage itself, and 2) the text attached to hyperlinks pointingto this page, from other pages.

8 / 24


Multi-View Co-Training

9 / 24


Single-View Co-Training by Committee

ContributionA single-view variant of Co-Training for applicationdomains in which there are not redundant and independentviews is proposed.Two learning frameworks for combining the merits of activelearning with semi-supervised learning.

MotivationFor many real-world applications, the requirement for twosufficient and independent views can not be fulfilled.Co-Training does not work well without an appropriatefeature splitting (Nigam and Ghani, 2000)Measuring the labeling confidence is not a straightforwardtask.

10 / 24


Single-View Co-Training By Committee

11 / 24


How to measure confidence

Inaccurate confidence estimation→ selecting and adding mislabeled examples to the training set→ degrade the classification accuracyEstimating Class Probabilities (CPE) provided by companioncommittee.

Confidence(xu,H(t−1)i ) = max

1≤c≤CH(t−1)

i (xu, ωc)

Unfortunately, in many cases the classifier does not provide anaccurate CPE. For instance, a decision tree provides piecewiseconstant probability estimates. That is, all unlabeled examplesxu which lie into a particular leaf, will have the same CPEsbecause the exact value of xu is not used in determining itsCPE.

12 / 24


Improving CPE of Decision Trees

Laplace Correction, Probability Estimation Tree (PET),(Provost, Machine Learning 2003)

P(ωc |xu) =nc + 1N + C

Bagging of PETRetrofitting Decision Tree Classifiers Using Kernel DensityEstimation (Fayyad, ICML’95)Improve Decision Trees for Probability-Based Ranking byLazy Learners (Liang, ICTAI’06)

13 / 24


Estimating local competence

The local competency of an unlabeled example xu givenH(t−1)

i is defined as follows:

Comp(xu,H(t−1)i ) =

∑xn∈N(xu),xn∈ωpred

H(t−1)i (xn, ωpred)

||xn − xu||2 + ε

where ωpred is the class label assigned to xu by H(t−1)i ;

H(t−1)j (xn, ωpred) is the probability given by H(t−1)

j thatneighbor xn belongs to class ωpred ; ε is a constant added toavoid zero denominator.It is inspired by decision-dependent distance-based k-nnestimate of the competence that was proposed for dynamicclassifier selection. (Woods, PAMI’97)

14 / 24


Estimating local competence

estimating local competence of an unlabeled examplegiven companion committee

15 / 24


Handwritten Digits Recognition

The Handwritten Digits that are described by four sets offeatures and are publicly available at UCI Repository. The digitswere extracted from a collection of Dutch utility maps. A total of2,000 patterns (200 patterns per class) have been digitized inbinary images.

Name Descriptionmfeat-pix 240 pixel averages in 2 x 3 windowsmfeat-kar 64 Karhunen-Love coefficientsmfeat-fac 216 profile correlationsmfeat-fou 76 Fourier coefficients of the character shapes

16 / 24


Experimental Setup

WEKA4 runs of 10-fold cross-validationFor SSL, 10% of the training examples (180 patterns) arerandomly selected as the initial labeled data set L while theremaining are used as unlabeled data set U.The Random Subspace Method constructs an ensemble often C4.5 pruned decision trees (with Laplace Correction)where each tree uses only 50% of the features.We set the pool size u = 100, the sample size n = one andthe number of nearest neighbors used to estimate localcompetence k is 10.

17 / 24


Experimental Results

Comparison between forests and individual trees.Comparison between CoBC and Self-Training.Comparison between CPE and local competenceconfidence measures.Comparison between CoBC and Co-Forest.

18 / 24



•: corrected paired t-test implemented in WEKA at 0.05 significance level.

19 / 24


Combining QBC and CoBC

Both semi-supervised learning and active learning tackle thesame problem but from different directions.

QBC-then-CoBC: QBC provides CoBC with a betterstarting point instead of randomly selecting labeledexamples.QBC-with-CoBC: In QBC-then-CoBC, QBC does notbenefit from CoBC. On the other hand, in QBC-with-CoBC,both algorithms are benefiting from each other.

20 / 24



•: corrected paired t-test implemented in WEKA at 0.05 significance level.

21 / 24


Conclusion

A new single-view committe-based semi-supervisedlearning framework is proposed.An ensemble of diverse and accurate classifiers caneffectively exploit the unlabeled data to improve therecognition accuracy.The random subspace method not only enforces thediversity but also reduces the dimensionality which isdesirable in case of small training set size.CoBC outperforms Self-Training.The local competence estimates is an effective confidencemeasure that outperforms the class probability estimatesfor sample selection.

22 / 24


Future Work

Influence of ensemble size, random subspace sizeDifferent ensemble learners, base learners such as SVMor kNNCoBC depends only on the companion committee H(t−1)

jconstructed at the previous iteration to measureconfidence. We will study the influence of depending on allthe previous versions (H(t ′)

j , t ′ = t − 1, t − 2, . . . ,0).

23 / 24


Thanks for your attention

Questions ??

24 / 24

Combining Committee-Based Semi-supervised and Active Learning and Its Application to Handwritten Digits Recognition

Technology

colt98 singleview

singlelearner tritraining

multiview learning algorithm

training examples

tsmc07 cotraining

singlelearner coem nigam

singlelearner em nigam

machine learning