Data-Intensive Cloud Control for GENI GEC 10 Orca control framework March 15 th, 2011 Michael Zink, Prashant Shenoy, Jim Kurose, David Irwin and Emmanuel.

Data-Intensive Cloud Control for GENI

GEC 10Orca control framework

March 15th, 2011

Michael Zink, Prashant Shenoy, Jim Kurose, David Irwin and Emmanuel

[email protected]

2

DiCloud Project Overview

Support researchers conducting data-intensive experiments

• Sensors network storage volumes processing

Extend Orca with

• Data-centric Slices: storage as a first-class resource with Amazon

Simple Storage Service (S3) and Elastic Block Storage (EBS)

• Cloud Computing: connect Amazon Elastic Compute Cloud (EC2)

resources to GENI

3

Demo – Step 1 (Allocate resources)

ViSE Sensor NetworkS3 (storage)

EC2 servers

Researcher creates a slice with:• weather radar • storage space on S3• compute servers on EC2

OrcaDiCloud

4

Demo – Step 2 (Storing data)

ViSE Sensor NetworkS3 (storage)

Weather radar data is stored in S3DiCloudS3 Proxy

5

Demo – Step 3 (Processing data)

S3 (storage)EC2 servers

• Get radar data from S3• Process data on EC2 server• Put generated image in S3

DiCloudAWS

accounting

00101010110101110011001010001110000

6

Demo – Step 4 (Visualizing data)

S3 (storage)

DiCloudS3 Proxy

• Get generated image in S3

7

Technical details

Implemented on top of Amazon Web Service API

DiCloud monitors

• EC2 Server hourly usage

• Network usage (in and out traffic using CloudWatch statistics)

• S3 storage space and put/get operations using a dedicated proxy

• EBS disk usage (storage space and number of IOs using

CloudWatch statistics)

All accounting operations are logged in a database

Resources are automatically revoked when budget has expired

8

CF and GENI AM API Integration

DiCloud implements building blocks for using Clouds

• Proxy to monitor cloud API calls

• Accounting software to track $ spent

• Mechanisms to prevent users from exceeding budget

Software sits beneath the CF/GENI AM API

• Orca AM implements GENI AM API

• DiCloud hooks into Orca join/leave handlers

• Internal Orca AM interfaces

• Other CFs may plug in at different places

9

Challenges

Network connectivity with AWS

Monitoring

• CloudWatch does not differentiate between free and paying network

traffic or disk IO

• S3 Proxy in the cloud would save network traffic and cost

What security model for storage

• How to share data with others?

• Storage leases can last years

What budget to allocate to make this useful?