Transcript
Wrappers for feature subset selection
Amir Razmjou
Benefits of Feature Subset Selection
• Too many dimensions; Elimination of the curse of dimensionality problem
• Improved model and classifier performance• Simple Models and elimination of over-
fitting• Faster training times.
The Process
Feature Selection Methods
Wrapperuse a search algorithm to search through the
space of possible features and evaluate each subset by running a model on the subset
Risk of over fitting to the modelComputationally expensive
EmbeddedEmbedded in and specific to a model
FilterSimilar to Wrappers in the search approachSimpler filter is evaluated
Hypothetical Concept
Definition 2
Contradiction 2
Definition 3
Contradiction 3 All features are irrelevant
Definition 4
Contradiction 4every feature is relevant
Definition 5 – Strong Relevance
Definition 6 – Weak Relevance
Conclusion 1
• Forward selection methods: these methods start with one or a few features selected according to a method specific selection criteria. More features are iteratively added until a stopping criterion is met.
• Backward elimination methods: methods of this type start with all features and iteratively remove one feature or bunches of features.
Relief
• Evaluates the worth of an attribute by repeatedly sampling an instance and considering the value of the given attribute for the nearest instance of the same and different class. Can operate on both discrete and continuous class data.
Relief
• Relief does not help with redundant features. If most of the given features are relevant to the concept, it would select most of them even though only a fraction are necessary for concept description
top related