University of Minnesota Optimizing MapReduce Provisioning in the Cloud Michael Cardosa, Aameek Singh†, Himabindu Pucha†, Abhishek Chandra http://www.cs.umn.edu/~cardosa Department of Computer Science, University of Minnesota †IBM Almaden Research Center
Optimizing MapReduce Provisioning in the Cloud. Michael Cardosa, Aameek Singh†, Himabindu Pucha †, Abhishek Chandra http://www.cs.umn.edu/~cardosa Department of Computer Science, University of Minnesota † IBM Almaden Research Center. MapReduce Provisioning Problem. Platform: - PowerPoint PPT Presentation
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
University of Minnesota
Optimizing MapReduce Provisioningin the Cloud
Michael Cardosa, Aameek Singh†,Himabindu Pucha†, Abhishek Chandra
http://www.cs.umn.edu/~cardosa
Department of Computer Science, University of Minnesota
†IBM Almaden Research Center
University of Minnesota
MapReduce Provisioning Problem Platform:
Virtualized Cloud Environment, which enables
Virtualized MapReduce Clusters Several MapReduce Jobs from different
users Goal: Optimize system-wide metrics, such
as: throughput, energy, load distribution, user costs
Problem: At the Cloud Service Provider level, how can we harvest opportunities to increase performance, save energy, or reduce user costs? 2
University of Minnesota
MapReduce Platform: Hadoop Open-source implementation of MapReduce
distributed computing framework Used widely: Yahoo, Facebook, NYT, (Google)
InputData
University of Minnesota
Hadoop Clusters
4
Distributed data Replicated chunks
Distributed computation Map/reduce tasks
Traditional: Dedicated physical nodes
University of Minnesota
Virtual Hadoop Clusters
5
Run Hadoop on top of VMs E.g.: Amazon Elastic MapReduce =
Hadoop+AmazonEC2
Server Pool
VM Pool
Hadoop Processes
University of Minnesota
Roadmap Intro & Problem Platform Overview Spatio-Temporal Insights for
Provisioning Building Blocks for MapReduce
Provisioning Case Study: Performance optimization Case Study: Energy optimization
6
University of Minnesota
Spatio-Temporal Insights for Provisioning
Initial Focus: Energy Savings Goal: Minimize energy usage
Energy+cooling ~ 42% of total cost [Hamilton08]
Problem: How to place the VMs on available physical servers to minimize energy usage? Minimize Cumulative Machine Uptime (CMU)
7
University of Minnesota
VM Placement: Spatial Fit
8
Job 1 Job 2 Job 3 Job 4
Co-Place complementary
workloads
University of Minnesota
Which placement is better?
9
20min
10min
100min
20min20min
20min
SHUTDOWN SHUTDOWN
A B
University of Minnesota
Time Balancing
10
20 25
90
20 25 20 25
20 25
30
20 25
30
20 25
30
Time Balance
University of Minnesota
Building Blocks for Provisioning
11
Objective-drivenresource provisioning
MapReduce Jobs
Jobprofiling
Clusterscaling Migration
Cloud Execution Environment
Initial Provisioning Continuous Optimization
University of Minnesota
Building Blocks for Provisioning Job Profiling: MapReduce job runtime
estimation Based on number of VMs allocated to job Based on input data size Offline and Online Profiling
Cluster Scaling: Changing number of VMs allocated to a particular MapReduce job Affects runtime of job; relies on Job Profiling
model Migration: Useful for continuous
optimization Load balancing, VM consolidation
12
University of Minnesota
Job Profiling: Runtime Estimation Based on Number of VMs
13
University of Minnesota
Job Profiling: Runtime Estimation Based on Input Data Size