Effects of change propagation resulting from adaptive preprocessing in multicomponent predictive systems

Effects of change propagation resulting from adaptive preprocessing in multicomponent predictive systems

Manuel Martín Salvador, Marcin Budka, Bogdan Gabrys{msalvador,mbudka,bgabrys}@bournemouth.ac.uk

Data Science Institute. Bournemouth University

KES-2016, York, UKSeptember 7th, 2016

Outline1. Prologue2. Introduction to MCPS3. Motivation4. Reactive adaptation of MCPS5. Experiments6. Conclusion

PROLOGUE

Butterfly effectSmall causes can have large effects

— Edward Lorenz (1917 - 2008)

Source: GloWings

Change propagationControlled change management in a system

CC by TheGiantVermin

Data streams

“Infinite” number of records

Continuously arriving to the system at different or same rates

Can be stationary or evolving

Data streams

Examples:

● Sensors in manufacturing industry● Traffic monitoring sensors● Event logs in websites● Transactions in the financial sector

“Infinite” number of records

Continuously arriving to the system at different or same rates

Can be stationary or evolving

A single engine of Airbus A320 has more than 1000 sensors

generating 10GB/s!!

INTRODUCTION TO MCPS

Data Stream

Data stream learning for online prediction

PredictiveModel

Online Supervised Learning Algorithm

Predictions

True labels

t+k

t

Data Stream

Data stream learning for online prediction

PredictiveModel

PredictionsPreprocessing Postprocessing

Multicomponent Predictive System (MCPS)

MCPS composition

Manual● WEKA● RapidMiner● Knime● IBM SPSS

Automatic● Auto-WEKA (Bayesian optimisation)● Auto-sklearn (Bayesian optimisation + Meta-learning)● TPOT (Genetic programming)● e-Lico IDA (Ontologies + Planning)

Example of WEKA workflow

Formalising MCPS

otoken(data) i

place

transition

Well-handled and Acyclic Workflow Petri net (WA-WF-net)MCPS = (P, T, F)

Formalising MCPS

o predictioni

place

transition

Well-handled and Acyclic Workflow Petri net (WA-WF-net)MCPS = (P, T, F)

“Automatic composition and optimisation of multicomponent predictive systems” @ IEEE TNNLS (under review) http://bit.ly/automatic-mcps-tnnls

Formalising MCPS

Classifier

o

Replace missing values

Dimensionality reduction

Outlier handling

token(data) i

place

transition

Well-handled and Acyclic Workflow Petri net (WA-WF-net)MCPS = (P, T, F)

“Automatic composition and optimisation of multicomponent predictive systems” @ IEEE TNNLS (under review) http://bit.ly/automatic-mcps-tnnls

MOTIVATION

Data changes over time

Snapshot of SYN dataset at different times

Need of model adaptation

Streaming error (mean over last 10 samples)SYN dataset with GFMM classifier

GFMMZ-Score PCA Min-Max

Wrongly classified

Need of preprocessing adaptation

Streaming error (mean over last 10 samples)SYN dataset with GFMM classifier

GFMMZ-Score PCA Min-Max

Wrongly classified (out of [0,1])

New hyperboxes

Main strategies for MCPS adaptation

Adaptation strategies GLOBAL LOCAL

Re-composition Full Partial

Hyperparameter optimisation (keep components) Full Partial

Parameterisation (keep components and hyperparameters) Full Partial

Main strategies for MCPS adaptation

Adaptation strategies GLOBAL LOCAL

Re-composition Full Partial

Hyperparameter optimisation (keep components) Full Partial

Parameterisation (keep components and hyperparameters) Full Partial

“Adapting Multicomponent Predictive Systems using Hybrid Adaptation Strategies with Auto-WEKA in Process Industry” @ AutoML / ICML 2016 http://bit.ly/adapting-mcps-paper

This work!

Need of change propagation

Streaming error (mean over last 10 samples)SYN dataset with GFMM classifier

GFMMZ-Score PCA Min-Max

Inconsistent hyperboxesdue to a different input space

REACTIVE ADAPTATION OF MCPS

Reactive adaptation of MCPSGFMMZ-Score PCA Min-Max

Time

i p1 p2 p3 o

[-3.1, 2.7]

x1 = 3.6

Reactive adaptation of MCPSGFMMZ-Score PCA Min-Max

Time

i p1 p2 p3 o

data

meta-data

[-3.1, 2.7]

x1 = 3.6

[-3.1, 3.6]

Reactive adaptation of MCPSGFMMZ-Score PCA Min-Max

Time

i p1 p2 p3 o

data

meta-dataprediction

[-3.1, 2.7]

x1 = 3.6

[-3.1, 3.6]

Updating a component: GFMM

0 1

1

0

(-3.1) (2.7)x1

x2

0 1

1

0

(-3.1) (3.6)x1

x2

Hyperboxes are mapped to the new

input space

EXPERIMENTS

Experiments

Name # Attr # Class Type

SYN 2 2 Synthetic

ELEC 7 2 Real

COVERTYPE 54 7 Real

GAS 128 6 Real

Datasets Scenarios

Id Adap.Model

Adap.Prepro.

ChangePropagation

#1 No No No

#2 Yes No No

#3 Yes Yes No

#4 Yes Yes YesFirst 200 samples for initial training, rest 400 for testing and online learning

GFMMZ-Score PCA Min-Max

Results

#3 crashes due to lack of change propagation when changing PCA components

CONCLUSION

Conclusion

Only model adaptation may not be enough to cope with evolving data streams, adaptive preprocessing should be considered.

However, “blind” adaptation of components can result in inconsistent models or even in a system crash.

Local adaptation of a component may require adapting further components. Therefore, a system must be reactive and propagate changes.

The definition of MCPS has been extended to support change propagation using a new token for meta-data in a coloured Petri net (cMCPS).

Future work

Large study to measure the actual cost of adaptation.

Open questions:

● How to handle propagation requiring changes of the Petri net structure?● How to handle transformations in systems with nonlinear components?● How to order components to reduce the cost of adaptation?● Can a meta-data token be removed at an early stage instead of being fully

propagated?

Thanks!

Paper: http://bit.ly/change-propagation-mcps

Slides: http://www.slideshare.net/draxus

Manuel <[email protected]>

@draxus