Motivation MSUMO: A Meta Learning Algorithm for Architecture Selection Experiments Conclusions & Future Work An Incremental Model Selection Algorithm Based on Cross-Validation for Finding the Architecture of a Hidden Markov Model on Hand Gesture Data Sets Aydın Ula¸ s 1 Olcay Taner Yıldız 2 1 Department of Computer Engineering Bo˘ gaziçi University Istanbul, Turkey 2 Department of Computer Engineering I¸ sık University Istanbul, Turkey ICMLA 2009 Aydın Ula¸ s, Olcay Taner Yıldız MSUMO: An Incremental Model Selection Algorithm for HMM
28
Embed
An Incremental Model Selection Algorithm Based on Cross ...€¦ · Model Selection in HMM Model Selection in HMM Hard to estimate number of hidden states & models in each state Need
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
MotivationMSUMO: A Meta Learning Algorithm for Architecture Selection
ExperimentsConclusions & Future Work
An Incremental Model Selection AlgorithmBased on Cross-Validation for Finding theArchitecture of a Hidden Markov Model on
Hand Gesture Data Sets
Aydın Ulas1 Olcay Taner Yıldız2
1Department of Computer EngineeringBogaziçi University
Istanbul, Turkey
2Department of Computer EngineeringIsık University
Istanbul, Turkey
ICMLA 2009Aydın Ulas, Olcay Taner Yıldız MSUMO: An Incremental Model Selection Algorithm for HMM
MotivationMSUMO: A Meta Learning Algorithm for Architecture Selection
ExperimentsConclusions & Future Work
Outline
1 MotivationHidden Markov ModelsModel Selection in HMM
2 MSUMO: A Meta Learning Algorithm for Architecture SelectionStructure Learning as State Space SearchMSUMO
3 ExperimentsExperimental SetupComparison of ArchitecturesResults
4 Conclusions & Future Work
Aydın Ulas, Olcay Taner Yıldız MSUMO: An Incremental Model Selection Algorithm for HMM
MotivationMSUMO: A Meta Learning Algorithm for Architecture Selection
ExperimentsConclusions & Future Work
Hidden Markov ModelsModel Selection in HMM
Hidden Markov ModelsGraphical network
N hidden states
M mixture components in these hidden states
Connectivity between hidden states
Aydın Ulas, Olcay Taner Yıldız MSUMO: An Incremental Model Selection Algorithm for HMM
MotivationMSUMO: A Meta Learning Algorithm for Architecture Selection
ExperimentsConclusions & Future Work
Hidden Markov ModelsModel Selection in HMM
Hidden Markov ModelsProbability model
Initial state probabilities
Observation probabilities
State transition probabilities
Baum-Welch algorithm using Expectation-Maximization
Aydın Ulas, Olcay Taner Yıldız MSUMO: An Incremental Model Selection Algorithm for HMM
MotivationMSUMO: A Meta Learning Algorithm for Architecture Selection
ExperimentsConclusions & Future Work
Hidden Markov ModelsModel Selection in HMM
Hidden Markov ModelsModels used in this paper
Left-right model
Left-right-loop model
Full model
Aydın Ulas, Olcay Taner Yıldız MSUMO: An Incremental Model Selection Algorithm for HMM
MotivationMSUMO: A Meta Learning Algorithm for Architecture Selection
ExperimentsConclusions & Future Work
Hidden Markov ModelsModel Selection in HMM
Hidden Markov Models
Bioinformatics (Secondary structure prediction)
Speech processing (Audio modelling)
Aydın Ulas, Olcay Taner Yıldız MSUMO: An Incremental Model Selection Algorithm for HMM
MotivationMSUMO: A Meta Learning Algorithm for Architecture Selection
ExperimentsConclusions & Future Work
Hidden Markov ModelsModel Selection in HMM
Model SelectionBias-variance tradeoff
Error = Bias + Variance
Small/simple model underfits (bias high, variance low)
Large/complex model overfits (bias low, variance high)
Aydın Ulas, Olcay Taner Yıldız MSUMO: An Incremental Model Selection Algorithm for HMM
MotivationMSUMO: A Meta Learning Algorithm for Architecture Selection
ExperimentsConclusions & Future Work
Hidden Markov ModelsModel Selection in HMM
Model Selection in HMM
Hard to estimate number of hidden states & models ineach state
Need a methodology to optimize the model structure fornovice user
Aydın Ulas, Olcay Taner Yıldız MSUMO: An Incremental Model Selection Algorithm for HMM
MotivationMSUMO: A Meta Learning Algorithm for Architecture Selection
ExperimentsConclusions & Future Work
Structure Learning as State Space SearchMSUMO
Structure LearningState Space Search
Huge search space: All the combinations of number ofhidden states and number of mixtures at each hidden state.
Infeasible to try and evaluate all possible architectures
Heuristic strategy visiting as few as possible states in thesearch space
Aydın Ulas, Olcay Taner Yıldız MSUMO: An Incremental Model Selection Algorithm for HMM
MotivationMSUMO: A Meta Learning Algorithm for Architecture Selection
ExperimentsConclusions & Future Work
Structure Learning as State Space SearchMSUMO
Structure LearningOperators
Define operators. Example: ADD-1 from state HMM4,1 toHMM5,1
Add operators for forward search
Remove operators for backward search
Floating search uses both
Aydın Ulas, Olcay Taner Yıldız MSUMO: An Incremental Model Selection Algorithm for HMM
MotivationMSUMO: A Meta Learning Algorithm for Architecture Selection
ExperimentsConclusions & Future Work
Structure Learning as State Space SearchMSUMO
Structure LearningEvaluation
Compare performance metric of next state with currentstate
Accept or reject operator based on improvement
Takes into account both generalization error andcomplexity
Aydın Ulas, Olcay Taner Yıldız MSUMO: An Incremental Model Selection Algorithm for HMM
MotivationMSUMO: A Meta Learning Algorithm for Architecture Selection
ExperimentsConclusions & Future Work
Structure Learning as State Space SearchMSUMO
Structure LearningDimensions of Search
Initial state (HMM1,1 or HMMN,1)
State transition operators (Add or remove)
Search beam (Single or multiple operator)
State evaluation function (AIC, BIC, or CV)
Termination condition (No improvement or fixed number ofiterations)
Aydın Ulas, Olcay Taner Yıldız MSUMO: An Incremental Model Selection Algorithm for HMM
MotivationMSUMO: A Meta Learning Algorithm for Architecture Selection
ExperimentsConclusions & Future Work
Structure Learning as State Space SearchMSUMO
MSUMO Operators
REMOVE-1: Remove a single hidden state from the HMM.
ADD-1: Add a single hidden state to the HMM.
REMOVE-L: Add a new Gaussian to the mixture.
ADD-L: Remove a Gaussian from the mixture.
Aydın Ulas, Olcay Taner Yıldız MSUMO: An Incremental Model Selection Algorithm for HMM
MotivationMSUMO: A Meta Learning Algorithm for Architecture Selection
ExperimentsConclusions & Future Work
Structure Learning as State Space SearchMSUMO
MSUMO Pseudocode
1 BEST = initial network2 while BEST changed3 for each applicable operator OPERi4 Ci ← OPERi (BEST)5 Sort candidates Ci in the order of complexity6 for i = 1 to number of candidates7 Train and validate Ci on k folds8 if Ci is more complex than BEST
9 Test H0 : µBEST ≤ µCi10 if H0 is rejected11 BEST← Ci12 break13 else14 Test H0 : µCi
≤ µBEST15 if H0 is accepted16 BEST← Ci17 break18 return BEST;
Aydın Ulas, Olcay Taner Yıldız MSUMO: An Incremental Model Selection Algorithm for HMM
MotivationMSUMO: A Meta Learning Algorithm for Architecture Selection
ExperimentsConclusions & Future Work
Structure Learning as State Space SearchMSUMO
Five MSUMO Variants
HMM 1,1
HMM 2,1
….
HMM h ,1
HMM 10,1
HMM 9,1
….
HMM h ,1
HMM 2,1
HMM 1,1
HMM 3,1 HMM 2,2 HMM 1,1
HMM 2,3 HMM 2,1
HMM 2,2 HMM 3,3 HMM 2,4 HMM 1,3
HMM 1,2
HMM 1,2 HMM 10,5
HMM 10,5
HMM 10,4
HMM 9,4
HMM 9,5 HMM 10,4 HMM 8,4
HMM 9,4 HMM 8,3 HMM 7,4 HMM 8,5
HMM 9,5
HMM 2,1
HMM 1,1
HMM 3,1 HMM 2,2 HMM 1,1
HMM 2,3
HMM 2,1
HMM 2,2 HMM 3,3 HMM 2,4 HMM 1,3
HMM 1,2
HMM 1,2
HMM 3,2
1-FW 1-BW FW BW MFW
Aydın Ulas, Olcay Taner Yıldız MSUMO: An Incremental Model Selection Algorithm for HMM
MotivationMSUMO: A Meta Learning Algorithm for Architecture Selection
ExperimentsConclusions & Future Work
Experimental SetupComparison of ArchitecturesResults
Experimental Factors
MSUMO variant used in search (1-FW, 1-BW, FW, BW,and MFW)
Statistical test used in comparison (k-fold paired t test)
Confidence level (1 − α) of the test (0.95)
Correction used when applying multiple tests (Holmcorrection)
Aydın Ulas, Olcay Taner Yıldız MSUMO: An Incremental Model Selection Algorithm for HMM
MotivationMSUMO: A Meta Learning Algorithm for Architecture Selection
ExperimentsConclusions & Future Work
Experimental SetupComparison of ArchitecturesResults
HMM Training
HMM toolbox implemented by Kevin Murphy
Retraining of the architecture when an operator is applied
No probabilities are kept or frozen
Aydın Ulas, Olcay Taner Yıldız MSUMO: An Incremental Model Selection Algorithm for HMM
MotivationMSUMO: A Meta Learning Algorithm for Architecture Selection
ExperimentsConclusions & Future Work
Experimental SetupComparison of ArchitecturesResults