DICE Horizon 2020 Research & Innovation Action Grant Agreement no. 644869 http://www.dice-h2020.eu Funded by the Horizon 2020 Framework Programme of the European Union Configuration Optimization Tool Pooyan Jamshidi Imperial College London
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
DICEHorizon2020Research&InnovationActionGrantAgreementno.644869http://www.dice-h2020.eu
FundedbytheHorizon 2020FrameworkProgrammeoftheEuropeanUnion
ConfigurationOptimizationTool
Pooyan JamshidiImperialCollegeLondon
BigDataTechnologies
Cloud(Priv/Pub)`
DICEFramework
2©DICE
DICEIDE
Profile
Plugins
Sim Ver Opt
DPIM
DTSM
DDSMTOSCAMethodology
Deploy Config Test
Mon
AnomalyTrace
Iter.Enh.
DataIntensiveApplication(DIA)
Cont.Int. FaultInj.
WP4
WP3
WP2
WP5
WP1 WP6- Demonstrators
ConfigurationOptimizationToolThe problem:o BigDatatechnologieshave
100sof tuneableparameterso Aknowledgegapisfacedby
SMEsinconfiguringthesetechnologies
What does the tool do?o Automatically runs
experimentsonDIAso Returnsrecommended
configuration parametersforBigDatatechnologies
3©DICE
Innovation:o AutomateDIAconfiguration
acrossreleasecycleso Priorartfocusesonmanual
configuration
Impact & stakeholders:o Reducecost andtimeof
testing betweenreleaseso SupportoperatorsofDIAs
COToolArchitecture
4©DICE
Configuration Optimisation Tool
performance repository
Monitoring
Deployment Service
Data Preparation
configuration parameters
values
configuration parameters
values
Experimental Suite
Testbed
Doc
Data Broker
Tester
experiment timepolling intervalprevious versions
configurationparameters
GP model
Kafka
Vn
V1 V2
System Under Test
historical data
WorkloadGenerator
Technology Interface
Stor
m
Cas
sand
ra
Spar
k
TwoimpementationsofCO:BO4CO,TL4CO
ToolInput(ParametersandOptions)
5©DICE
1- Information about the experiment: budget, config file, duration of each experiment
2- Information about the configuration parameters and their options that testers determine
COToolArchitecture
6©DICE
Configuration Optimisation Tool
performance repository
Monitoring
Deployment Service
Data Preparation
configuration parameters
values
configuration parameters
values
Experimental Suite
Testbed
Doc
Data Broker
Tester
experiment timepolling intervalprevious versions
configurationparameters
GP model
Kafka
Vn
V1 V2
System Under Test
historical data
WorkloadGenerator
Technology Interface
Stor
m
Cas
sand
ra
Spar
k
OptimizationComponent(Matlab)
7©DICE
- This component select the next configuration to experiment considering the current measurements,
- This continues until optimum configuration located or experimental budget finished. Theoptimizationoverhead
isnegligablecomparingwithmeasurements
This componly relies on rayality free MCRcomponent
COToolArchitecture
8©DICE
Configuration Optimisation Tool
performance repository
Monitoring
Deployment Service
Data Preparation
configuration parameters
values
configuration parameters
values
Experimental Suite
Testbed
Doc
Data Broker
Tester
experiment timepolling intervalprevious versions
configurationparameters
GP model
Kafka
Vn
V1 V2
System Under Test
historical data
WorkloadGenerator
Technology Interface
Stor
m
Cas
sand
ra
Spar
k
ExperimentalSuite
9©DICE
Thiscomponentrunstheexperimentsandmeasurestheperformanceofthesystemundertest,thedataareflushedtocsvfileandcommunicatedwiththeoptimizationcomponent
COToolArchitecture
10©DICE
Configuration Optimisation Tool
performance repository
Monitoring
Deployment Service
Data Preparation
configuration parameters
values
configuration parameters
values
Experimental Suite
Testbed
Doc
Data Broker
Tester
experiment timepolling intervalprevious versions
configurationparameters
GP model
Kafka
Vn
V1 V2
System Under Test
historical data
WorkloadGenerator
Technology Interface
Stor
m
Cas
sand
ra
Spar
k
PerformanceRepository
11©DICE
spouts max_spout sorters emit_freq chunk_size message_size throughput latency1 10 1 1 1.00E+05 1000 22657 3.98331 10 1 1 1.00E+05 10000 3596.3 18.4151 10 1 1 1.00E+05 1.00E+05 112.56 217.631 10 1 1 1.00E+06 1000 12273 5.19521 10 1 1 1.00E+06 10000 1174.9 24.2471 10 1 1 1.00E+06 1.00E+05 111.88 205.491 10 1 1 2.00E+06 1000 12024 5.29351 10 1 1 2.00E+06 10000 1151.3 25.0391 10 1 1 2.00E+06 1.00E+05 94.294 220.621 10 1 1 1.00E+07 1000 11552 6.28671 10 1 1 1.00E+07 10000 1228.1 24.9751 10 1 1 1.00E+07 1.00E+05 102.29 236.191 10 1 10 1.00E+05 1000 25978 3.47821 10 1 10 1.00E+05 10000 10112 9.28471 10 1 10 1.00E+05 1.00E+05 1023.8 83.2361 10 1 10 1.00E+06 1000 24147 3.65941 10 1 10 1.00E+06 10000 8400.2 11.8041 10 1 10 1.00E+06 1.00E+05 1197.4 73.7861 10 1 10 2.00E+06 1000 22858 3.71511 10 1 10 2.00E+06 10000 7141.3 10.7551 10 1 10 2.00E+06 1.00E+05 1095.1 78.6241 10 1 10 1.00E+07 1000 22693 4.36371 10 1 10 1.00E+07 10000 6281.5 14.3081 10 1 10 1.00E+07 1.00E+05 951.27 71.4921 10 1 60 1.00E+05 1000 25862 3.85211 10 1 60 1.00E+05 10000 10859 8.64521 10 1 60 1.00E+05 1.00E+05 1128.8 79.8621 10 1 60 1.00E+06 1000 23553 3.90481 10 1 60 1.00E+06 10000 9734.3 9.3451 10 1 60 1.00E+06 1.00E+05 982 66.8521 10 1 60 2.00E+06 1000 25408 3.57381 10 1 60 2.00E+06 10000 7993.9 9.2784
ConfigurationMetrics Measured
The performance repository mediates between the optimization component and experimental suite
ConfigurationParameters(Output)
12©DICE
TechnologySupport
13©DICE
Configuration Optimisation Tool
performance repository
Monitoring
Deployment Service
Data Preparation
configuration parameters
values
configuration parameters
values
Experimental Suite
Testbed
Doc
Data Broker
Tester
experiment timepolling intervalprevious versions
configurationparameters
GP model
Kafka
Vn
V1 V2
System Under Test
historical data
WorkloadGenerator
Technology Interface
Stor
m
Cas
sand
ra
Spar
k
Technologies:Storm,Spark,Cassandra
Related Documents
