Total Variation and Euler's Elastica for Supervised Learning Tong Lin, Hanlin Xue, Ling Wang, Hongbin Zha Contact: tonglin123@gmail.com Peking University,
Post on 31-Dec-2015
217 Views
Preview:
Transcript
Total Variation and Euler's Elastica for Supervised Learning
Tong Lin, Hanlin Xue, Ling Wang, Hongbin Zha
Contact: tonglin123@gmail.com
Peking University, China
2012-6-29
1Key Lab. Of Machine Perception, School of EECS,
Peking University, China
Background• Supervised Learning:
• Definition: Predict u: x -> y, with training data (x1, y1), …, (xN, yN)
• Two tasks: Classification and Regression
• Prior Work:• SVM:
• RLS: Regularized Least Squares, Rifkin, 2002
2
Hinge loss:
Squared loss:
3
Background• Prior Work (Cont.):
• Laplacian Energy: “Manifold Regularization: A Geometric Framework for Learning from Labeled and Unlabeled Examples,” Belkin et al., JMLR 7:2399-2434, 2006
• Hessian Energy: “Semi-supervised Regression using Hessian Energy with an Application to Semi-supervised Dimensionality Reduction,” K.I. Kim, F. Steinke, M. Hein, NIPS 2009
• GLS: “Classification using geometric level sets,” Varshney & Willsky, JMLR 11:491-516, 2010
4
Motivation
SVM Our Proposed Method
5
Large margin should not be the sole criterion; we argue sharper edges and smoother boundaries can play significant roles.
3D display of the output classification function u(x) by the proposed EE model
6
• General:
• Laplacian Regularization (LR):
• Total Variation (TV):
• Euler’s Elastica (EE):
1min ( ( ), ) ( )
n
i iiuL u x y S u
2 2min ( ) | |u
u y dx u dx
2 2min ( ) ( ) | |u
u y dx a b u dx
2min ( ) | |u
u y dx u dx
| |
u
u
Models
7
TV&EE in Image Processing• TV: a measure of total quantity of the value change• Image denoising (Rudin, Osher, Fatemi, 1992)
• Elastica was introduced by Euler in 1744 on modeling torsion-free elastic rods
• Image inpainting (Chan et al., 2002)
8
• TV can preserve sharp edges, while EE can produce smooth boundaries
• For details, see T. Chan & J. Shen’s textbook: Image Processing and Analysis: Variational, PDE, Wavelet, and Stochastic Methods, SIAM, 2005
9
Decision boundary
The mean curvature k in high dimensional space can have same expression except the constant 1/(d-1).
Framework
10
11
• The calculus of variations → Euler-Lagrange PDE
3
1 1( ) ( '( ) | |) ( ( '( ) | |))
| | | |V n u u u u
u u
2min [ ] ( ) ( )J u u y dx S u
2( ) | |LRS u u dx
( ) | |TVS u u dx
2( ) ( ) | |EES u a b u dx
2( ) 0 (#)u u y 2( ) 0| |
uu y
u
2( ) 0V u y
2( ) a b
Energy Functional Minimization
12
Solutions
a. Laplacian Regularization (LR)
Radial Basis Function Approximation
b. TV & EE: We develop two solutions• Gradient descent time marching (GD)• Lagged linear equation iteration (LagLE)
13
Experiments: Two-Moon Data
SVM
EE
Both methods can achieve 100% accuracies with different parameter combinations
14
Experiments: Binary Classification
15
Experiments: Multi-class Classification
16
Experiments: Multi-class Classification
Note: Results of TV and EE are computed by the LagLE method.
17
Experiments: Regression
18
Conclusions• Contributions:
• Introduce TV&EE to the ML community
• Demonstrate the significance of curvature and gradient empirically
• Achieve superior performance for classification and regression
• Future Work:• Hinge loss
• Other basis functions
• Extension to semi-supervised setting
• Existence and uniqueness of the PDE solutions
• Fast algorithm to reduce the running time
End, thank you!
top related