WBDB 2014 Benchmarking Virtualized Hadoop Clusters

Benchmarking Virtualized Hadoop Clusters

Todor Ivanov, Roberto V. Zicari Big Data Lab, Goethe University Frankfurt

Alejandro Buchmann Database and Distributed Systems, TU Darmstadt

1 5th Workshop on Big Data Benchmarking 2014

Outline

• Virtualizing Hadoop

• Measuring Performance – Iterative Experimental Approach – Platform Setup – Experiments – Summary of Results

• Lessons Learned

• Next Steps

5th Workshop on Big Data Benchmarking 2014 2

Virtualizing Hadoop

• Motivation

– Hadoop-as-a-service (e.g. Amazon Elastic Map Reduce)

– Automated deployment and cost-effective management

– Dynamically scalable cluster size (e.g. # of nodes, resource allocation)

• Challenges

– I/O overhead

– Network overhead (message communication and data transfer)

• Related Work: virtualized vs. physical Hadoop Virtualized Hadoop has an estimated overhead ranging between 2-10%

(reported in [1], [2], [3])

5th Workshop on Big Data Benchmarking 2014 3

[1] Buell, J.: A Benchmarking Case Study of Virtualized Hadoop Performance on VMware vSphere 5. Tech. White Pap. VMware Inc. (2011). [2] Buell, J.: Virtualized Hadoop Performance with VMware vSphere ®5.1. Tech. White Pap. VMware Inc. (2013). [3] Microsoft: Performance of Hadoop on Windows in Hyper-V Environments. Tech. White Pap. Microsoft. (2013).

Objectives of Our Research

Investigate and compare the performance between

standard and separated data-compute cluster configurations.

• How does the application performance change on a data-compute cluster?

• What type of applications are more suitable for data-compute clusters?

5th Workshop on Big Data Benchmarking 2014 4

Standard

Cluster Data-Compute Cluster

Methodology: Iterative Experimental Approach

I. Choose a Big Data Benchmark

II. Configure Hadoop Cluster

III. Perform Experiments

IV. Evaluate

Results

5th Workshop on Big Data Benchmarking 2014 5

Step I: Intel HiBench

• Benchmark suite for Hadoop (developed by Intel in 2010) (Huang et al. [4])

• 4 categories, 10 workloads & 3 types

• Metrics: Time (Sec) & Throughput (Bytes/Sec)

Category No Workload Tools Type

Micro Benchmarks

1 Sort MapReduce IO Bound

2 WordCount MapReduce CPU Bound

3 TeraSort MapReduce Mixed

4 TestDFSIOEnhanced MapReduce IO Bound

Web Search 5 Nutch Indexing Nutch, Lucene Mixed

6 Page Rank Pegasus Mixed

Machine Learning 7 Bayesian Classification Mahout Mixed

8 K-means Clustering Mahout Mixed

Analytical Query 9 Join Hive Mixed

10 Aggregation Hive Mixed

5th Workshop on Big Data Benchmarking 2014 6

[4] Huang, S. et al.: The HiBench benchmark suite: Characterization of the MapReduce-based data analysis. Data Engineering Workshops (ICDEW), 2010

Step II: Platform Setup

• Platform layer (Hadoop Cluster) – vSphere Big Data Extension integrating Serengeti Server (version 1.0) – VM template hosting CentOS – Apache Hadoop (version 1.2.1) with default parameters:

• 200MB Java Heap size • 64MB block size • 3 replication factor

• Management layer (Virtualization) – VMWare vSphere 5.1 – ESXi and vCenter Servers

• Hardware layer - Dell PowerEdge T420 server – 2 x Intel Xeon E5-2420 (1.9 GHz), 6 core CPUs – 32GB RAM – 4 x 1 TB, WD SATA disks

Hardware

Management (Virtualization)

Application (HiBench Benchmark)

Platform (Hadoop Cluster)

CPUs Memory Storage

5th Workshop on Big Data Benchmarking 2014 7

(Known) Limitations

• Single physical server (no physical network)

• VMWare ESXi server hypervisor

• Testing with default configurations (Serengeti & Hadoop)

• Time constraints: – Input data sizes: 10/20/50GB

– 3 test repetitions

5th Workshop on Big Data Benchmarking 2014 8

Step II: Comparison Factors

The number of utilized VMs in the compared clusters should be equal.

• Each additional VM increases the hypervisor overhead (reported in [2], [5], [6])

• Utilizing more VMs may improve the overall system performance [2]

The utilized hardware resources in a cluster should be equal.

5th Workshop on Big Data Benchmarking 2014 9

[2] Buell, J.: Virtualized Hadoop Performance with VMware vSphere ®5.1. Tech. White Pap. VMware Inc. (2013). [5] Li, J. et al.: Performance Overhead Among Three Hypervisors: An Experimental Study using Hadoop Benchmarks. Big Data (BigData Congress), 2013 [6] Ye, K. et al.: vHadoop: A Scalable Hadoop Virtual Cluster Platform for MapReduce-Based Parallel Machine Learning with Performance Consideration. Cluster Computing Workshops (CLUSTER WORKSHOPS), 2012

Step II: Comparison Standard1/Data-Compute1

Standard

Cluster Data-Compute Cluster

1) of the utilized hardware resources 2) of the utilized VMs

∆ – difference in performance

5th Workshop on Big Data Benchmarking 2014 10

Step II: Comparison Standard2/Data-Compute3

Standard Cluster Data-Compute

Cluster

1) of the utilized hardware resources 2) of the utilized VMs

∆ – difference in performance

5th Workshop on Big Data Benchmarking 2014 11

Step II: Comparison Data-Compute1/2/3

Data-Compute Cluster Data-Compute

Cluster

1) of the utilized hardware resources

∆ – difference in performance

5th Workshop on Big Data Benchmarking 2014 12

Step II: All Cluster Configurations

5th Workshop on Big Data Benchmarking 2014 13

Step III & IV: CPU Bound - WordCount

• Configuration: 4 map/1 reduce tasks, 10/20/50 GB input data sizes

• Times normalized with respect to baseline Standard1

• 38-47% better performance for Data-Compute cluster

• Data-Compute1 (2CW & 1DW) ≈ Data-Compute2 (2CW & 2DW)

Equal Number of VMs

3 VMs 6 VMs

DataSize (GB)

Diff. (%) Standard1/

Data-Comp1

Diff. (%) Standard2/

Data-Comp3

10 -40 -38

20 -41 -42

50 -43 -47

5th Workshop on Big Data Benchmarking 2014 14

1.00 1.00 1.00

1.75 1.74 1.74

0.71 0.71 0.70 0.71 0.71 0.70

1.26 1.22 1.19

0

0.5

1

1.5

2

10 20 50Data Size (GB)

Standard1 Standard2 Data-Comp1 Data-Comp2 Data-Comp3

Rat

io t

o S

tan

dar

d1

Step III & IV: Read I/O Bound – TestDFSIOEnh (1)

• Configuration: 100MB file size, 10/20/50 GB input data sizes

• Read times normalized with respect to baseline Standard1

• Standard1 (Standard Cluster) performs best

Equal Number of VMs

3 VMs 6 VMs

Data Size (GB)

Diff. (%) Standard1/

Data-Comp1

Diff. (%) Standard2/

Data-Comp3

10 68 -18

20 71 -30

50 73 -46 Rat

io t

o S

tan

dar

d1

5th Workshop on Big Data Benchmarking 2014 15

1.00 1.00 1.00

1.83 1.93 1.87

3.08 3.39

3.66

1.51 1.71 1.78

1.55 1.48 1.28

0.0

1.0

2.0

3.0

4.0

10 20 50Data Size (GB)

Standard1 Standard2 Data-Comp1 Data-Comp2 Data-Comp3

Step III & IV: Read I/O Bound – TestDFSIOEnh (2)

• Configuration: 100MB file size, 10/20/50 GB input data sizes

• Read times normalized with respect to baseline Standard1

• Data-Comp1 (2CW & 1DW) > DC2 (2CW & 2DW) > DC3 (3CW & 3DW)

More data nodes improve read performance in a Data-Compute cluster.

Different Number of VMs

3 VMs 4 VMs

4 VMs 6 VMs

Data Size (GB)

Diff. (%) Data-

Comp1/2

Diff. (%) Data-

Comp2/3

10 -104 3

20 -99 -15

50 -106 -39

5th Workshop on Big Data Benchmarking 2014 16

1.00 1.00 1.00

1.83 1.93 1.87

3.08 3.39

3.66

1.51 1.71 1.78

1.55 1.48 1.28

0.0

1.0

2.0

3.0

4.0

10 20 50Data Size (GB)

Standard1 Standard2 Data-Comp1 Data-Comp2 Data-Comp3

Rat

io t

o S

tan

dar

d1

Step III & IV: Write I/O Bound – TestDFSIOEnh (1)

• Configuration: 100MB file size, 10/20/50 GB input data sizes

• Write times normalized with respect to baseline Standard1

• Data-Compute cluster (Data-Comp1, Data-Comp3) performs better

Equal Number of VMs

3 VMs 6 VMs

Data Size (GB)

Diff. (%) Standard1/

Data-Comp1

Diff. (%) Standard2/

Data-Comp3

10 -10 4

20 -21 -14

50 -24 -1

5th Workshop on Big Data Benchmarking 2014 17

1.00 1.00 1.00

0.84

1.08 1.00

0.91 0.83 0.81

0.73 0.86

0.95 0.87

0.95 0.99

0.0

0.5

1.0

1.5

10 20 50Data Size (GB)

Standard1 Standard2 Data-Comp1 Data-Comp2 Data-Comp3

Rat

io t

o S

tan

dar

d1

Step III & IV: Write I/O Bound – TestDFSIOEnh (2)

• Configuration: 100MB file size, 10/20/50 GB input data sizes • Write times normalized with respect to baseline Standard1

• Data-Comp1 (2CW & 1DW) < Data-Comp3(3CW & 3DW) Having 2 extra Data Worker nodes increases the write overhead up to

19% in a Data-Compute cluster.

• Data-Comp3 (6VMs) outperforms Standard1 (3VMs)

Different Number of VMs

3 VMs 6 VMs

3 VMs 6 VMs

Data Size (GB)

Diff. (%) Data-

Comp1/3

Diff. (%) Standard1/

Data-Comp3

10 -4 -15

20 13 -6

50 19 -1

5th Workshop on Big Data Benchmarking 2014 18

1.00 1.00 1.00

0.84

1.08 1.00

0.91 0.83 0.81

0.73 0.86

0.95 0.87

0.95 0.99

0.0

0.5

1.0

1.5

10 20 50Data Size (GB)

Standard1 Standard2 Data-Comp1 Data-Comp2 Data-Comp3

Rat

io t

o S

tan

dar

d1

Summary of Results

• Compute-intensive (i.e. CPU bound) workloads are suitable for Data-Compute clusters. (up to 47% faster)

• Read-intensive (i.e. read I/O bound) workloads are suitable for Standard clusters.

– For Data-Compute clusters adding more data nodes improves the read performance. (up to 39% better e.g. Data-Compute2/Data-Compute3)

• Write-intensive (i.e. write I/O bound) workloads are suitable for Data-

Compute clusters. (up to 15% faster e.g. Standard1/Data-Compute3 )

– Lower number of data nodes result in better write performance.

5th Workshop on Big Data Benchmarking 2014 19

Lessons Learned

• Factors influencing cluster performance*:

– Overall number of virtual nodes (VMs) in a cluster

– Choosing cluster type (Standard or Data-Compute Hadoop cluster)

– Number of nodes for each type (compute and data nodes) in a Data-Compute cluster

* note: Limitations known! (slide 9)

5th Workshop on Big Data Benchmarking 2014 20

Next Steps

• Repeat the experiments on virtualized multi-node cluster

• Evaluate virtualized performance with other workloads

• Experiments with larger data sets

• Repeat the experiments using other hypervisors (e.g. OpenStack)

5th Workshop on Big Data Benchmarking 2014 21

Thank you!

Questions & Feedback are very welcome!

Contact info:

Todor Ivanov todor@dbis.cs.uni-frankfurt.de http://www.bigdata.uni-frankfurt.de/

5th Workshop on Big Data Benchmarking 2014 22