Label Embedding Trees for Large Multi-class Tasks

Label Embedding Trees for Large Multi-class Tasks

Samy Bengio Jason WestonDavid Grangier

Presented by Zhengming Xing

Outline

• Introduction• Label Trees• Label Embeddings• Experiment result

Introduction

Main idea: propose a fast and memory saving multi-class classifier for large dataset based on trees structure method

Large scale problem:

the number of example Feature dimensionNumber of class

Introduction Label Tree:

Label Predictors:

Indexed nodes:

Edges:

Label sets:

The root contain all classes, and each child label set is a subset of its parent

K is the number of classes

Disjoint tree: any two nodes at the same depth cannot share any labels.

IntroductionClassifying an example:

Label TreesTree loss

I is the indicator function

is the depth in the tree of the final prediction for x

Label treeLearning with fixed label tree: N,E,L chosen in advanceGoal: minimize the tree loss over the variables F

Given training data

Relaxation 1

Relaxation 2

Replace indicator function with hinge loss and

Label treeLearning label tree structure for disjoint tree

Treat A as the affinity matrix and apply the steps similar to spectral clustering

Basic idea: group together labels into the same label set that are likely to be confused at test time.

Label embeddingsis a k-dimensional vector with a 1 in the yth position and 0 otherwise

Problem : how to learn W, V

define

solve

Label embeddingsMethod 1:

The same two steps of algorithm 2Learn V

Learn W

minimize

minimize

Label embedding

Combine all the methods discussed above

Method 2: join learn W and V

minimize

minimize

ExperimentDataset

Experiment

Experiment