1 Using Bayesian Network for combining classifiers Leonardo Nogueira Matos Departamento de Computação Universidade Federal de Sergipe.

1

Using Bayesian Network for combining classifiers

Leonardo Nogueira MatosDepartamento de Computação

Universidade Federal de Sergipe

2

Agenda

Why combining classifiers?

Bayesian network principles

Bayesian network as an ensemble of classifiers

Experimental results

Future works and conclusions

3

Why combining classifiers?

Classifiers can colabore with each other

Minimizes computational effort for training

Maximizes global recognition rate

4

Why not to do so?

Because combining individual preditions can be so difficult as divising a robust single classifier

5

Why not to do so?

Because combining individual preditions can be so difficult as divising a robust single classifier

Decision

Classifiers

Combiner

6

Approaches for combining classifiers

L1. Data Level L3. Decision Level L2. Feature Level

Fixed rules Trainable rules

7

Approaches for combining classifiers

L1. Data Level L3. Decision Level L2. Feature Level

Fixed rules Trainable rules

8

Why not to do so?

Because combining individual preditions can be so difficult as divising a robust single classifier

Decision

Classifiers

Combiner

p(w|x)

p(w|x)

p(w|x)

9

Approaches for combining classifiers

L1. Data Level L3. Decision Level L2. Feature Level

Fixed rules Trainable rules

10

Existent scenarios

Pattern space

Pattern21

classifiers

classifiers

11

Our scenery

Pattern space

classifiers

12

A closed look

13

A closed look – discriminant function

14

A closed look – using multiple classifiers

15

A closed look – using multiple classifiers

The challegers:

How can we combine classifier's output?How can we identify regions in pattern space?

16

Agenda

Why combining classifiers?

Bayesian network principles

Bayesian network as an ensemble of classifiers

Experimental results

Future works and conclusions

17

Bayesian network principles

A

B C

Those circles represent binary random variables

18

Bayesian network principles

A

B C

Those circles represent binary random variables

a0a1

b0b1

c0c1

19

Bayesian network principles

A

B C

Those circles represent binary random variables

a0a1

b0b1

c0c1

a0 b0 c0

a1 b1 c1

⋮ ⋮ ⋮aN bN cN

dataset

20

Bayesian network principles

A

B C

Those circles represent binary random variables

a0a1

b0b1

c0c1

a0 b0 c0

a1 b1 c1

⋮ ⋮ ⋮aN bN cNinstance

21

Bayesian network principles

A

B C

Jointly probability inference is a combinatorial problem

P abc = P a P b∣a P c∣ab

2 possibilities

4 possibilities

22

Bayesian network principles

A

B C

Jointly probability inference is a combinatorial problem

P abc = P a P b∣a P c∣ab

P abc = P a P b∣a P c

Independence makes computation alittle more simple

23

Bayesian network principles

A

B C

Arest – indicates statistical dependence between variables

24

Bayesian network principles

A

B C

Arc – represents causality

25

Bayesian network principles

A

B C

A Bayesian network is a DAG (DirectAciclic Graph) where nodes representrandom variables and arcs representcausality relatioship

26

Bayesian network principles

A

B C

There are polinomial time algorithmsto compute inference in BN

27

Bayesian network principles

A

B C

There are polinomial time algorithmsto compute inference in BN

Evidence

28

Bayesian network principles

A

B C

There are polinomial time algorithmsto compute inference in BN

Evidence messages

29

Bayesian network principles

A

B C

There are polinomial time algorithmsto compute inference in BN

Evidence

[P a0∣bP a1∣b]

[P c0∣b P c1∣b ]

30

Agenda

Why combining classifiers?

Bayesian network principles

Bayesian network as an ensemble of classifiers

Experimental results

Future works and conclusions

31

A Fundamental Goal

32

Another insight

From a statistical point-of-view a Bayesian network is also a graphicalmodel to represents a complex and factored probability distribution function

33

Another insight

From a statistical point-of-view a Bayesian network is also a graphicalmodel to represents a complex and factored probability distribution function

34

Another insight

From a statistical point-of-view a Bayesian network is also a graphicalmodel to represents a complex and factored probability distribution function

The challegers:

How can we combine classifier's output?How can we identify regions in pattern space?

35

How can we combine classifier's output?

We use a BN as a graphical model of the pdf P(w|x)

We assume that classifier participate in computing that function

Each classifier must be a statistical classifier

36

How can we identify regions in pattern space?

37

Splitting pattern space

38

Defining a region

39

Patterns in a region

40

Algorithm

41

Bayesian Network Structure

42

Bayesian networks for combining classifiers

43

Agenda

Why combining classifiers?

Bayesian network principles

Bayesian network as an ensemble of classifiers

Experimental results

Future works and conclusions

44

Results with UCI databases

45

Results with NIST database

46

System I classifiers

47

Preliminaries

48

Results with the complete dataset

49

Agenda

Why combining classifiers?

Bayesian network principles

Bayesian network as an ensemble of classifiers

Experimental results

Future works and conclusions

50

Future works

51

Future works

52

Future works

53

Future works

54

Future works

Pattern space

Pattern21

classifiers

classifiers

55

ConclusionsWe have developed a method for combining classifiers using a Bayesian network

A BN act as trainable ensemble of statistical classifiers

The method is not suitable for small size dataset

Experimental results reveal a good performance with a large dataset

As a future work we intend to use a similar approach for splitting the feature vector and combine classifiers specialized on each piece of it.

56

Thank you!

lnmatos@ufs.br