Sponsored by the National Science Foundation GEC 12 DiCloud Tutorial University of Massachusetts Amherst November 2 nd, 2011 GEC12 DiCloud Tutorial.

Sponsored by the National Science Foundation

GEC 12 DiCloud Tutorial

University of Massachusetts AmherstNovember 2nd, 2011

GEC12 DiCloud Tutorial

Sponsored by the National Science Foundation 2

Project Summary

• DiCloud enables GENI experiments to incorporate cloud computing resources– Focuses on Amazon web services– EC2 (VMs), EBS (block storage), S3 (object storage)

• How are cloud resources different from other GENI resources?– They cost money!

• Requires active Amazon account and credit card

– Require authentication with Amazon

Project Summary

• DiCloud tracks per GENI-user Amazon costs– Amazon charges can be complicated

• Pay per VM time used• Pay for data transfer in and out• Pay for IOs to block devices

– No current facilities for tracking per user costs

• Enables controlled access to cloud resources– Stand-alone: setup your own DiCloud instance to track

your own costs– GENI CF-mode: place “underneath” GENI CF, where

AM pays the bills, but gives GENI users access

Elastic Compute Cloud (EC2)

• Rent VMs from Amazon “on-demand”– http://wwww.amazon.com/ec2

Elastic Compute Cloud (EC2)

• Rent VMs from Amazon “on-demand”– Pay per hour based on VM performance– Pay for data transfer in/out

Elastic Block Store (EBS)

• Rent block storage from Amazon– Pay per GB-month (currently $0.10)– Also pay per 1 million IO requests (currently $0.10)

Simple Storage Service (S3)

• Rent object storage from Amazon– Pay per GB-month – Pay for data transfer in/out

Tutorial Overview

• Setup DiCloud server for stand-alone mode• Access DiCloud through console

– Learn how to create VMs, block devices, storage objects– Learn how to check our balance and set quotas

• Access DiCloud through web portal– Learn how to create users and edit quotas

• Learn how to give users access to cloud resources

– Learn how to console actions through web portal

• Time permitting– Use your Amazon resources in a simple GENI

experiment using Gush

Tutorial Step 1: Let’s get started!

• Trac page with both setup and tutorial instructions– http://groups.geni.net/geni/wiki/DICLOUD/

GEC12tutorial– Setup instructions done for you– Tutorial instructions step-by-step in slides– Download GEC12tutorial.txt file with commands

• Pre-requisites– Download/start your Virtual Box image

• Login as user: geni

– Open a Firefox web browser– Open 3 terminal windows and go to ~/Tutorials/DiCloud

• $ cd ~/Tutorials/DiCloud

Tutorial Step 1a: Common Errors

• Be careful cutting and pasting– Doesn’t work from Trac!– Sometimes even txt file inserts hidden characters– Some lines require edits; don’t cut and paste the

newline

• Make terminal windows big– Sometimes terminal gets corrupted when pasting a line

that runs to the next line (especially when backspacing)

• On the website– Remember to hit down before hitting enter if the

browser auto-fills in a text box (from its history)

Tutorial Step 2: Getting your Keys

• We’ve created temporary keys for you to access Amazon using a UMass account– Run the following commands:

• $ cd ~/Tutorials/DiCloud• $ ./key_install.sh geni.cs.umass.edu

– Verify Success:geni@geni-tutorial:~/Tutorials/DiCloud$ ls -la ec2/keys/

total 40

drwxr-xr-x 2 geni geni 4096 2011-10-24 11:30 .

drwxr-xr-x 6 geni geni 4096 2011-10-17 09:21 ..

-rw------- 1 geni geni 120 2011-10-24 11:26 credentials.csv

-rw-r--r-- 1 geni geni 91 2011-10-24 11:29 geni_aws_credentials.txt

-rw-r--r-- 1 geni geni 1375 2011-10-24 11:26 geni_cert.pem

-rw-r--r-- 1 www-data www-data 1375 2011-10-24 11:30 geni_cert_web.pem

-rw------- 1 geni geni 1675 2011-10-24 11:26 geni-keypair.pem

-rw-r--r-- 1 geni geni 400 2011-10-24 11:26 geni-keypair.pub

-rw-r--r-- 1 geni geni 887 2011-10-24 11:26 geni_pk.pem

-rw-r--r-- 1 www-data www-data 887 2011-10-24 11:30 geni_pk_web.pem

Tutorial Step 3: DiCloud Server

• Run following command to start DiCloud server: – $ cd ~/Tutorials/DiCloud– $ source ./environment– $ cd dicloud/bin– $ ./dicloud_server.sh– Verify Success

11:20:31,376 INFO root Reading properties from dicloud.properties

11:20:31,466 INFO root Starting RMI registry on port 1099

11:20:31,474 INFO root Registering DiCloud server

11:20:33,253 INFO root Registering AWS accouting service

11:20:33,260 INFO root Registering EC2 handler

11:20:33,275 INFO root Registering S3 handler

11:20:33,289 INFO root Registering EBS handler

11:20:33,302 INFO root Waiting for console connections

• Move terminal to background but keep DiCloud server running

Tutorial Step 3: DiCloud Server

• Switch to another terminal– $ cd ~/Tutorials/DiCloud– $ source ./environment– $ cd dicloud/bin– $ cat ec2pricing.properties

cloudwatch=-0.015

network.in.per.gb=-.10

network.out.per.gb=-.15

s3.storage.per.gb=-.15

s3.put=-.00001

s3.get=-.000001

ebs.storage.per.gb=-.10

ebs.io.per.million=-.10

t1.micro,us-west-1=-.01

m1.small,us-west-1=-.042

m1.large,us-west-1=-.16

m1.xlarge,us-west-1=-.325

m2.xlarge,us-west-1=-.23

m2.2xlarge,us-west-1=-.571

……..

Tutorial Step 4a: DiCloud Console

• Now open a new terminal and start console:– $ cd ~/Tutorials/DiCloud– $ source ./environment– $ ./dicloud_console.sh– Verify Success

geni@geni-tutorial:~/Tutorials/DiCloud$ ./dicloud_console.sh

dicloud - not connected>

• Now we are in the DiCloud console– Type “help” to see a list of the possible console

commands

Tutorial Step 4b: DiCloud Console

• Now connect to server and start creating resources– dicloud - not connected> connect localhost 1099

Connecting to //localhost:1099/dicloud/server

dicloud - //localhost:1099>

– dicloud - //localhost:1099> get balanceCurrent balance is: $0.0

– dicloud - //localhost:1099> add credit 100

Current balance is: $0.0

New balance is: $100.0

– dicloud - //localhost:1099> get balanceCurrent balance is: $100.0

• Next we are going to create an S3 storage object, an EC2 VM, and an EBS storage volume

Tutorial Step 4c: S3 objects

<rand> must be random

(S3 namespace globally shared)

• Create S3 storage object– dicloud - //localhost:1099> s3 create gec12-tutorial-<rand> us– dicloud - //localhost:1099> s3 put gec12-tutorial-<rand> test

/home/geni/Tutorials/DiCloud/s3_test

Uploading file /home/geni/Tutorials/DiCloud/s3_test as object test in bucket gec12-tutorial-<rand>

– dicloud - //localhost:1099> s3 get gec12-tutorial-<rand> test /home/geni/Tutorials/DiCloud/test

Downloading file /home/geni/Tutorials/DiCloud/test from object test in bucket gec12-tutorial-<rand>

– dicloud - //localhost:1099> s3 delete gec12-tutorial-<rand> us– dicloud - //localhost:1099>get balance

Tutorial Step 4d: EC2 VMs

• Create EC2 VM– dicloud - //localhost:1099> ec2 start /geni_cert.pem /geni_pk.pem ami-

013d6c44 1 m1.small us-west-1 geni-keypairi-970bb3d0

– dicloud - //localhost:1099> ec2 probe /geni_cert.pem /geni_pk.pem i-970bb3d0 us-west-1 ascii

Instance Id: i-970bb3d0AMI

Id: ami-013d6c44

Public name: ec2-50-18-35-162.us-west-1.compute.amazonaws.com

Private name: ip-10-166-207-140.us-west-1.compute.internal

Public IP: 10.166.207.140

Private IP: ebs

Instance State: runningInstance

Type: m1.small

Key pair: geni-keypair

Start time: 2011-10-24T18:02:43+0000

Availability zone: us-west-1a

Tutorial Step 4e: EC2 VMs

• Logging into your EC2 VM– Open another terminal window– $ ssh –i /geni-keypair.pem root@ec2-50-18-35-162.us-west-

1.compute.amazonaws.com

__| __|_ )

_| ( / Amazon Linux AMI

___|\___|___|

See /usr/share/doc/system-release/ for latest release notes.

No packages needed for security; 10 packages available

[ec2-user@ip-10-166-207-140 ~]$

• Next lets create an EBS storage volume and attach it to our VM

Tutorial Step 4f: EBS storage volume

• Create EBS storage volume– dicloud - //localhost:1099> ebs create /geni_cert.pem /geni_pk.pem 1 us-

west-1 us-west-1<availability zone>vol-7f2ca91e

– dicloud - //localhost:1099> ebs attach /geni_cert.pem /geni_pk.pem us-west-1 vol-7f2ca91e i-970bb3d0 sdg

attaching

• Switch back to the EC2 VM’s terminal– [ec2-user@ip-10-166-207-140 ~]$ ls /dev/sdg

/dev/sdg

Tutorial Step 4g: Console Scripting

• DiCloud’s console is scriptable– $ cd ~/Tutorials/DiCloud/dicloud/bin– $ nano getbalance.txt

connect localhost 1099

get balance

– $ ./dicloud_console_script.sh < getbalance.txt

dicloud - not connected>Connecting to //localhost:1099/dicloud/server

dicloud - //localhost:1099>Current balance is: $0.0

dicloud - //localhost:1099>

• Also able to hook into DiCloud programmatically using Java

Tutorial Step 4h: Clean-up

• Detach and destroy EBS storage volume– dicloud - //localhost:1099> ebs detach /geni_cert.pem /geni_pk.pem us-

west-1 vol-7f2ca91edetaching

– dicloud - //localhost:1099> ebs delete /geni_cert.pem /geni_pk.pem vol-7f2ca91e us-west-1

• Terminate EC2 VM instance– dicloud - //localhost:1099> ec2 stop /geni_cert.pem /geni_pk.pem i-

970bb3d0 us-west-1 geni-keypair– dicloud - //localhost:1099> get balance

Tutorial Step 5: Web Portal

• Open Firefox web browser– Go to http://localhost/vise/dicloud– Login as user:admin, pw:geni

• First, lets create a new non-admin user

• Give the new user the name geni with pw geniuser– Change access rights to view billing history

Tutorial Step 5: Web Portal• Now click on “change” under “Certificates” for user geni• Enter file names and click “Update user certificates”

– cert file: /geni_cert_web.pem– key file: /geni_pk_web.pem– keypair name: geni-keypair

Tutorial Step 5: Web Portal• Now logout and log back in as user “geni”

– Then click on “Request EC2 resources”– Use ami-013d6c44 as the AMI id

Tutorial Step 5: Web Portal• Now logout and log back in as user “geni”

– Then click on “Request EC2 resources”– Use ami-013d6c44 as the AMI id

• Now go back to your user page– Click on status next to the VM you just created

• Now lets create an S3 bucket– Click on “Request new S3 buckets”

• Once back at the user page, put something in your bucket– Select the s3_test file in ~/Tutorials/DiCloud/s3_test

• Now get that same thing back from your bucket

• Now lets create an EBS volume and attach it to our VM– Click request new EBS volumes– Make it 1GB, in us-west-1, in us-west-1<availability num>

• Attach the volume to our VM– Click on attach, enter instance id, and sdg for device

• If we ssh into our VM we can see that the volume is now attached

Tutorial Step 5: Web Portal Cleanup

• EBS– Detach and delete EBS volume

• S3– Delete S3 storage bucket

• For EC2…..– Lets create 1 more VM, so we can test our Gush experiments

across 2 VMs– Take note of the IP addresses for each machine

• ec2-50-18-30-192.us-west-1.compute.amazonaws.com• ec2-50-18-138-224.us-west-1.compute.amazonaws.com

Tutorial Step 6: Using Gush

• Setup keys and fix hostnames – $ cp ~/Tutorials/DiCloud/ec2/keys/geni-keypair.pem ~/.ssh/id_rsa– $ cp ~/Tutorials/DiCloud/ec2/keys/geni-keypair.pub ~/.ssh/id_rsa.pub– $ keychain ~/.ssh/id_rsa– $ ssh root@amazon.machine.ip “hostname amazon.machine.ip”– $ ssh root@amazon.machine2.ip “hostname amazon.machine2.ip”

• Add EC2 VMs to Gush’s directory.xml file– $ cd ~/Tutorials/DiCloud/gush– $ nano directory.xml

<?xml version="1.0" encoding="UTF-8"?>

<gush>

<resource_manager type="ssh">

</resource_manager>

</resource_manager>

</gush>

• Run Gush experiment – $ cd ~/Tutorials/DiCloud/gush– $ ./gush –P 15000– gush> info nodes

There are 2 known nodes:

[ U ] root@ec2-184-72-17-161.us-west-1.compute.amazonaws.com:15400(pref=0) (Disconnected.)

– gush> load simple.xml Project "simple" is selected.Experiment "simple" is selected.

– gush> run

<< see next slide for correct output >>– gush> disconnect

gush> root@ec2-50-18-78-110.us-west-1.compute.amazonaws.com:15400 has decided to leave the mesh.

root@ec2-184-72-17-161.us-west-1.compute.amazonaws.com:15400 has decided to leave the mesh.

– gush> quit

• Run Gush experiment – gush> runStarting experiment run.

Running experiment simple...

gush> The configuration matcher has finished matching.The resource allocator has finished successfully.

root@ec2-184-72-17-161.us-west-1.compute.amazonaws.com:15400 has joined the mesh.

The file transfer of Package to ec2-184-72-17-161.us-west-1.compute.amazonaws.com has been completed.

The software installation of Package on ec2-184-72-17-161.us-west-1.compute.amazonaws.com was successful.

root@ec2-184-72-17-161.us-west-1.compute.amazonaws.com:15400,2261: Hello World! Your experiment was successful

root@ec2-50-18-78-110.us-west-1.compute.amazonaws.com:15400,2313: Hello World!

Your experiment was successful!

The experiment has ended.

Tutorial Step 6: Using Gush with Orca

• Run Gush experiment using Orca and DiCloud resources – $ cd ~/Tutorials/DiCloud/gush– $ cat omni_config

[ setup omni_config with correct ProtoGENI credentials ]– $ nano directory.xml

[ change directory.xml to add orca slice ]<?xml version="1.0" encoding="UTF-8"?>

<gush>

<resource_manager type="geni-orca">

<port_map slice="gush-orca-david" port="11000"/>

</resource_manager>

</resource_manager>

</resource_manager>

</gush>

• Add both Amazon and Orca keys to keychain– $ cd ~/.ssh/; keychain --clear– $ keychain ~/.ssh/id_rsa_geni; keychain ~/.ssh/id_rsa– $ cd ~/Tutorials/DiCloud/gush– $ cp simple.xml simple-orca.xml; nano simple-orca.xml

[ Add orca resources to simple.xml example ]<?xml version="1.0" encoding="utf-8"?>

<gush>

<num_hosts>4</num_hosts>

</rspec>

</resources>

</gush>

Tutorial Step 6: Using Gush with Orca• Create orca-new.rspec

– $ nano orca-new.rspec<?xml version="1.0" encoding="UTF-8"?>

<rspec type="request" xsi:schemaLocation="http://www.protogeni.net/resources/rspec/2

http://www.protogeni.net/resources/rspec/2/request.xsd"

xmlns:flack="http://www.protogeni.net/resources/rspec/ext/flack/1"

xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

xmlns=http://www.protogeni.net/resources/rspec/2>

<sliver_type name="raw-pc">

<disk_image name="http://geni-images.renci.org/images/gush/gush-deb5-i386.xml" version="25f53b64cfe44dd1604447f04b7b533bb67dea1e" />

</sliver_type>

</interface></node>

<sliver_type name="raw-pc">

<disk_image name="http://geni-images.renci.org/images/gush/gush-deb5-i386.xml" version="25f53b64cfe44dd1604447f04b7b533bb67dea1e" />

</sliver_type>

</interface></node>

<interface_ref client_id="geni1:0" />

<interface_ref client_id="geni2:0" />

</link>

</rspec>

• Create Orca slice and slivers; check status– $ helper-scripts/handle-geni.py -n -f pg -a

https://www.emulab.net/protogeni/xmlrpc/am createslice gush-orca-david

INFO:omni:Loading config file omni_config

INFO:omni:Using control framework pg

Created slice with Name gush-orca-david, URN urn:publicid:IDN+emulab.net+slice+gush-orca-david

– $ helper-scripts/handle-geni.py -n -f pg -a https://geni.renci.org:11443/orca/xmlrpc createsliver gush-orca-david orca-new.rspec

INFO:omni:Loading config file omni_config

Asked https://geni.renci.org:11443/orca/xmlrpc to reserve resources. Result: Request id: 8eaacd08-4f4b-4ac6-be0a-ce78dbd77893[ Slice UID: 4e8cec5d-a28a-49df-a485-0a03c46f3a95 | Reservation UID: f3cc03c5-6902-4fac-9f96-d6f94d1a43a1 | Resource Type: acisrencivmsite.vm | Resource Units: 1 ]

[ Slice UID: 4e8cec5d-a28a-49df-a485-0a03c46f3a95 | Reservation UID: 2d6fc0e5-77bc-41a9-9c93-7e2d46134b9a | Resource Type: acisrencivmsite.vm | Resource Units: 1 ]

[[ Slice UID: 4e8cec5d-a28a-49df-a485-0a03c46f3a95 | Reservation UID: 02db97c4-1ff1-492d-9c51-5736061682c0 | Resource Type: acisrencivmsite.vlan | Resource Units: 1 ]

• Create Orca slice and slivers; check status (cont’d)– $ helper-scripts/handle-geni.py -n -f pg -a

https://geni.renci.org:11443/orca/xmlrpc sliverstatus gush-orca-david<?xml version="1.0"?>

<gush>

</slice>

</gush>

– $ helper-scripts/handle-geni.py -n -f pg -a https://geni.renci.org:11443/orca/xmlrpc sliverstatus gush-orca-david

<?xml version="1.0"?>

<gush>

</slice>

</gush>

• Must fix hostnames as before– $ ssh root@152.54.10.11 “hostname 152.54.10.11”– $ ssh root@152.54.10.10 “hostname 152.54.10.10”

• Run Gush experiment as before– $ ./gush –P 15000 – gush> Gush has learned about the slice gush-orca-david.– gush> Updated information on the slice gush-orca-david – gush> info nodesThere are 4 known nodes:

[ P ] root@152.54.10.10:11022(pref=0) (Disconnected.)

[ P ] root@152.54.10.11:11022(pref=0) (Disconnected.)

– gush> load simple-orca.xml

Project "simple" is selected.

Experiment "simple" is selected.

– gush> run

[ output on next slide ]

Tutorial Step 6: Using Gush with Orca• Run Gush experiment as before

– gush> runStarting experiment run.

Running experiment simple...

gush> The configuration matcher has finished matching.

The resource allocator has finished successfully.

root@152.54.10.10:11022 has joined the mesh.

The file transfer of Package to 152.54.10.10 has been completed.

The software installation of Package on 152.54.10.10 was successful.

root@152.54.10.11:11022 has joined the mesh.

The file transfer of Package to 152.54.10.11 has been completed.

The software installation of Package on 152.54.10.11 was successful.

root@152.54.10.10:11022,2053: Hello World! Your experiment was successful!

root@152.54.10.11:11022,2028: Hello World! Your experiment was successful!

root@ec2-50-18-28-196.us-west-1.compute.amazonaws.com:15400,3220: Hello World!

Your experiment was successful!root@ec2-204-236-143-92.us-west-1.compute.amazonaws.com:15400,2281:

Hello World! Your experiment was successful!

The experiment has ended.

• Disconnect from gush and delete sliver– gush> disconnectgush> root@152.54.10.10:11022 has decided to leave the mesh.

root@152.54.10.11:11022 has decided to leave the mesh.

– gush> quit– $ helper-scripts/handle-geni.py -n -f pg -a

https://geni.renci.org:11443/orca/xmlrpc deletesliver gush-orca-davidINFO:omni:Loading config file omni_config

Deleted sliver urn:publicid:IDN+emulab.net+slice+gush-orca-david on unspecified_AM_URN at https://geni.renci.org:11443/orca/xmlrpc

Tutorial Step 6: Terminating EC2 VMs

• Go back to the Firefox browser and terminate your VMs

Sponsored by the National Science Foundation GEC 12 DiCloud Tutorial University of Massachusetts Amherst November 2 nd, 2011 GEC12 DiCloud Tutorial.

amazon slide

amazon resources

geni resources

geni users access slide

tutorialsdicloud slide

geni experiments

gush slide

comec2 slide

Documents

GIMS Tutorial - GEC 12 Packet Capture for GENI

Gec report

RSPO MANUAL - GEC

GEC 2014 Intro

#GEC12: Martin Edmondson

GEC Magazine 5

Scanned by CamScanner of... · DAC study Card 8257 study...

GEC MAGAZINE

GEC Approach

Gec Bilaspur

Gec presentation gec-nyc

Gec showers catalogue

GEC Packaged Boilers

GEC Training

GIMI Tutorial GIMI Team University of Massachusetts Amherst....

GEC 220 NOTE_1