ICML2004, Banff, Alberta, Canada ICML2004, Banff, Alberta, Canada Learning Larger Margin Learning Larger Margin Machine Locally and Machine Locally and Globally Globally Kaizhu Huang ( Kaizhu Huang ( [email protected][email protected]) Haiqin Yang, Irwin King, Michael R. Lyu Haiqin Yang, Irwin King, Michael R. Lyu Dept. of Computer Science and Engineering Dept. of Computer Science and Engineering The Chinese University of Hong Kong The Chinese University of Hong Kong July 5, 2004 July 5, 2004 The Chinese University of Hong The Chinese University of Hong Kong Kong
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Learning Larger Margin Learning Larger Margin Machine Locally and Machine Locally and GloballyGlobally
Kaizhu Huang (Kaizhu Huang ([email protected]@cse.cuhk.edu.hk))Haiqin Yang, Irwin King, Michael R. LyuHaiqin Yang, Irwin King, Michael R. LyuDept. of Computer Science and EngineeringDept. of Computer Science and EngineeringThe Chinese University of Hong KongThe Chinese University of Hong Kong
July 5, 2004July 5, 2004
The Chinese University of Hong KongThe Chinese University of Hong Kong
The Chinese University of Hong KongThe Chinese University of Hong Kong
Learning Larger Margin Learning Larger Margin Machine Locally and GloballyMachine Locally and Globally
ContributionsContributionsBackground:Background:– Linear Binary ClassificationLinear Binary Classification– MotivationMotivation
Maxi-Min Margin Machine(MMaxi-Min Margin Machine(M44))– Model DefinitionModel Definition– Geometrical InterpretationGeometrical Interpretation– Solving MethodsSolving Methods– Connections With Other ModelsConnections With Other Models– Nonseparable caseNonseparable case– KernelizationsKernelizations
The Chinese University of Hong KongThe Chinese University of Hong Kong
Theory:Theory: A unified model of Support Vector Machi A unified model of Support Vector Machine (SVM), Minimax Probability Machine (MPM), anne (SVM), Minimax Probability Machine (MPM), and Linear Discriminant Analysis (LDA).d Linear Discriminant Analysis (LDA).
Practice:Practice: A sequential Conic Programming Proble A sequential Conic Programming Problem.m.
Learning Locally and Learning Locally and GloballyGlobally
The Chinese University of Hong KongThe Chinese University of Hong Kong
wT z + b=0y
x
Along the dashed axis, y data have a larger data trend than x data. Therefore, a more reasonable hyerplane may lie closer than x data rather than locating itself in the middle of two classes as in SVM.
The Chinese University of Hong KongThe Chinese University of Hong Kong
MM44: Links with MPM (Cont’): Links with MPM (Cont’)
MPM
M4
Remarks: The procedure is not reversible: MPM is a special case of M4
MPM focuses on building decision boundary GLOBALLY, i.e., it exclusively depends on the means and covariances. However, means and covariances may not be accurately estimated.
In linear cases, M4 outperforms SVM and MPM In Gaussian cases, M4 is slightly better or comparable than SVM (1). Sparsity in the feature space results in inaccurate estimation of covariance matrices (2) Kernelization may not keep data topology of the original data.—Maximizing Margin in the feature space does not necessarily maximize margin in the original space
The Chinese University of Hong KongThe Chinese University of Hong Kong
Future WorkFuture Work
Speeding up MSpeeding up M44
Contain support vectors—can we employ its sparsity aContain support vectors—can we employ its sparsity as has been done in SVM?s has been done in SVM?
Can we reduce redundant points??Can we reduce redundant points??
How to impose constrains on the kernelization fHow to impose constrains on the kernelization for keeping the topology of data?or keeping the topology of data?
Generalization error bound?Generalization error bound? SVM and MPM have both error bounds.SVM and MPM have both error bounds.
How to extend to multi-category classifications?How to extend to multi-category classifications?
The Chinese University of Hong KongThe Chinese University of Hong Kong
ConclusionConclusion
Proposed a new large margin classifier MProposed a new large margin classifier M44 which learns the decision boundary both which learns the decision boundary both locally and globallylocally and globally
Built theoretical connections with other Built theoretical connections with other models: A unified model of SVM, MPM and LDAmodels: A unified model of SVM, MPM and LDA
Developed sequential Second Order Cone Developed sequential Second Order Cone Programming algorithm for MProgramming algorithm for M44
Experimental results demonstrated the Experimental results demonstrated the advantages of our new modeladvantages of our new model