Improving Kieker's Scalability by Employing Linked Read-Optimized ...
Post on 14-Feb-2017
214 Views
Preview:
Transcript
Improving Kieker’s Scalability by Employing Linked Read-Optimized and Write-Optimized NoSQL Storage
Armin Moebius | IBAK H. Hunger GmbH & Co. KG, Kiel, Germany Sven Ulrich | IBAK H. Hunger GmbH & Co. KG, Kiel, Germany
7th Symposium on Software Performance 2016
08.11.2016
Regression Benchmarking Execution Environment
Agenda
• Kieker extension for Apache Cassandra data storage
• Regression Benchmarking Execution Environment (RBEE) architecture excerpt
• Data storage with RBEE Monitoring Log
• Evaluation scenarios and details
• Conclusion
2 / 22 7th Symposium on Software Performance 2016 | Moebius & Ulrich| 08.11.2016
Introduction
7th Symposium on Software Performance 2016 | Moebius & Ulrich| 08.11.2016
• Bring regression benchmarks into continuous integration environments
• Improve Kieker’s scalability
• Create a gateway between Kieker and our RBEE Regression Benchmarking Execution Environment
kieker-monitoring.net rbee.io
3 / 22
RBEE Architecture (Excerpt)
7th Symposium on Software Performance 2016 | Moebius & Ulrich| 08.11.2016
SUT
Communication Backend
Control Center
MLog
Com. API Com. API
Cassandra Extension
Benchmark Result
Repository
Com. API
Com. API
Microservices as self-contained Systems Virtual Machine
4 / 22
GUI
UI UI
Other SCS
Own Data Storage
Kieker Extension
7th Symposium on Software Performance 2016 | Moebius & Ulrich| 08.11.2016
• Enabling Kieker to store its monitoring output in Apache Cassandra
• Extending Kieker’s SyncDbWriter and AsyncDbWriter classes
• New classes: – CassandraDb: basic implementation to access Apache Cassandra
– CassandraSyncDbWriter: - extends Kieker’s AbstractMonitoringWriter - newMonitoringRecord() overwritten
– CassandraAsyncDbWriter: extends Kieker’s AbstractAsyncWriter - asynchronous implementation of our writer
– CassandraWriterThread: workerthreads for persisting monitoring data
• Added configuration parameters to kieker.monitoring.properties file
5 / 22
Data Storage
7th Symposium on Software Performance 2016 | Moebius & Ulrich| 08.11.2016
MLog
Com. API
• Containerized system based on Docker
• Employs polyglot persistence – Apache Cassandra: write-optimized wide-column noSQL data storage
– ElasticSearch: read-optimized document store noSQL data storage
• Scales horizontally
• CTE pushes data asynchronously from Cassandra to ElasticSearch
6 / 22
Linking Read- and Write-Optimized Storage
7th Symposium on Software Performance 2016 | Moebius & Ulrich| 08.11.2016
RBEE Monitoring Log
Keyspace
Col 2
Col n
Col 1
Tab 1 Tab 2 Tab n
Col 2
Col n
Col 1 Col 2
Col n
Col 1
{
"_index": "tab1",
"_type" : "tab1",
"_id" : "AVgf49kPr3sjN1Qg0hWa",
"_score" : 1.0,
"_source" : {
"col1" : 1,
"col2" : 2,
"coln" : "n"
}
}
Connecting to both
querying table names
fetching data for each table
sending data of each table
7 / 22
Evaluation
7th Symposium on Software Performance 2016 | Moebius & Ulrich| 08.11.2016
• System under test (SUT) – Java-based employee work time registration system
– Swing-based GUI
– Exactly reproducible monitoring output is generated by executing a Marathon script
8 / 22
Marathon </>
type numbers
Evaluation
7th Symposium on Software Performance 2016 | Moebius & Ulrich| 08.11.2016
• Test scenarios:
a) Non instrumented setup
b) Kieker file-based setup
c) Non containerized setup with read- and write-optimized data storage
Containerized setups
d) 1 container on 1 container host
e) 6 containers on 1 container host
f) 6 containers on 3 container hosts with 2 containers per container host
9 / 22
Evaluation
7th Symposium on Software Performance 2016 | Moebius & Ulrich| 08.11.2016
• Test hardware:
– HP Workstation Z420 • Intel Xeon E5-2670, 32 GB RAM, SATA HDD, 1 GBit/s Lan
– HP Proliant DL380 G7 (Server) • 2x Intel Xeon X5650, 144 GB RAM, 16 SAS 10K HDDs, 1 GBit/s LAN
– HP ProCurve 2910al-24G (Switch) • Dedicated 24 Port Gigabit Switch
10 / 22
Evaluation Results Part (a) – Execution Time
7th Symposium on Software Performance 2016 | Moebius & Ulrich| 08.11.2016
• Non-instrumented System Under Test (SUT)
11 / 22
Evaluation Results Part (b) – Execution Time
7th Symposium on Software Performance 2016 | Moebius & Ulrich| 08.11.2016
• Kieker instrumented system under Test (SUT)
• Logging to files on local file system
12 / 22
Evaluation Results Part (c) – Execution Time
7th Symposium on Software Performance 2016 | Moebius & Ulrich| 08.11.2016
• Instrumented with Kieker
• No containerization
• Read- and write optimized data storage
HP ProCurve Gigabit Switch (Dedicated Ethernet)
Ubuntu 16.04 LTS
HP ProLiant DL380 G7
Instrumented System Under Test
13 / 22
Evaluation Results Part (d) – Execution Time
7th Symposium on Software Performance 2016 | Moebius & Ulrich| 08.11.2016
• Containerization on physical hardware • Employing RBEE Monitoring Log (1 instance)
Ubuntu 16.04 LTS
Docker Engine
bins / libs
HP ProLiant DL380 G7
Container Host 1
HP ProCurve Gigabit Switch (Dedicated Ethernet)
Instrumented System Under Test
14 / 22
Evaluation Results Part (e) – Execution Time
7th Symposium on Software Performance 2016 | Moebius & Ulrich| 08.11.2016
Ubuntu 16.04 LTS
Docker Engine
bins / libs
HP ProLiant DL380 G7
Container Host 1
• Containerization on physical hardware • Employing RBEE Monitoring Log (6 instances)
HP ProCurve Gigabit Switch (Dedicated Ethernet)
Instrumented System Under Test
15 / 22
Evaluation Results Part (f) – Execution Time
7th Symposium on Software Performance 2016 | Moebius & Ulrich| 08.11.2016
Ubuntu 16.04 LTS
Docker Engine
bins / libs
HP ProLiant DL380 G7
Ubuntu 16.04 LTS
Docker Engine
bins / libs
HP ProLiant DL380 G7
Ubuntu 16.04 LTS
Docker Engine
bins / libs
HP ProLiant DL380 G7
Container Host 1 Container Host 2 Container Host 3
D o c k e r S w a r m
• Containerization & Clustering • Employing RBEE Monitoring Log (3x2 instances)
HP ProCurve Gigabit Switch (Dedicated Ethernet)
Instrumented System Under Test
16 / 22
Evaluation Results – Execution Times - Summary
7th Symposium on Software Performance 2016 | Moebius & Ulrich| 08.11.2016
• Speedup of method’s execution times in relation to Kieker’s file based instrumentation:
– (c) RBEE Monitoring Log without containerization 8% faster
– (d) RBEE Monitoring Log Docker image (1 host | 1 instance) 12% faster
– (e) RBEE Monitoring Log Docker image (1 host | 6 instances) 14% faster
– (f) RBEE Monitoring Log Docker image (3 hosts | 2 instances each) 22% faster
17 / 22
Evaluation Results Part (d) – Transfer Time
7th Symposium on Software Performance 2016 | Moebius & Ulrich| 08.11.2016
• Containerization on physical hardware • Employing RBEE Monitoring Log (1 instance)
Ubuntu 16.04 LTS
Docker Engine
bins / libs
HP ProLiant DL380 G7
Container Host 1
HP ProCurve Gigabit Switch (Dedicated Ethernet)
Instrumented System Under Test
18 / 22
Evaluation Results Part (e) – Transfer Time
7th Symposium on Software Performance 2016 | Moebius & Ulrich| 08.11.2016
Ubuntu 16.04 LTS
Docker Engine
bins / libs
HP ProLiant DL380 G7
Container Host 1
• Containerization on physical hardware • Employing RBEE Monitoring Log (6 instances)
HP ProCurve Gigabit Switch (Dedicated Ethernet)
Instrumented System Under Test
19 / 22
Evaluation Results Part (f) – Transfer Time
7th Symposium on Software Performance 2016 | Moebius & Ulrich| 08.11.2016
Ubuntu 16.04 LTS
Docker Engine
bins / libs
HP ProLiant DL380 G7
Ubuntu 16.04 LTS
Docker Engine
bins / libs
HP ProLiant DL380 G7
Ubuntu 16.04 LTS
Docker Engine
bins / libs
HP ProLiant DL380 G7
Container Host 1 Container Host 2 Container Host 3
D o c k e r S w a r m
HP ProCurve Gigabit Switch (Dedicated Ethernet)
• Containerization on physical hardware • Employing RBEE Monitoring Log (3x2 instances)
Instrumented System Under Test
20 / 22
Conclusion
7th Symposium on Software Performance 2016 | Moebius & Ulrich| 08.11.2016
• Employing linked write-optimized and read-optimized data storage for storing Kieker’s monitoring output reduce monitoring overhead
• The usage of RBEE Monitoring Log, built upon Docker, Apache Cassandra and ElasticSearch improves Kieker’s data storage scalability.
21 / 22
Thank you
7th Symposium on Software Performance 2016 | Moebius & Ulrich| 08.11.2016
Kieker: http://www.kieker-monitoring.net
RBEE: http://www.rbee.io
22 / 22
top related