CLOSER 2012 - Apr 2012 glideinWMS - Ownership of VM 1 CLOSER 2012 The glideinWMS approach to the ownership of System Images in the Cloud World presented by Igor Sfiligoi 1 co-authors A.Tiradani 2 , B.Holzman 2 and D.C.Bradley 3 1 UC San Diego, 2 Fermilab, 3 UW Madison
29
Embed
The glideinWMS approach to the ownership of System Images in the Cloud World
Scientific communities that are accustomed to use Grid resources are now considering the use of Cloud resources. However, moving from the Grid to the Cloud brings along the need for the creation and maintenance of the system image used to configure the provisioned resources, and this presents both opportunities and problems for the users. The impact is especially interesting in the context of glideinWMS due to its layered architecture. This presentation describes the various options available to the glideinWMS project team, their advantages and disadvantages, and explains why one of them is to be preferred.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
CLOSER 2012 - Apr 2012 glideinWMS - Ownership of VM 1
CLOSER 2012
The glideinWMS approachto the ownership of System Images
in the Cloud World
presented by Igor Sfiligoi1
co-authors A.Tiradani2, B.Holzman2 and D.C.Bradley3
1UC San Diego, 2Fermilab, 3UW Madison
CLOSER 2012 - Apr 2012 glideinWMS - Ownership of VM 2
Our users
● Our primary target audience is the scientific community● In particular, those communities that need
massive amounts of CPU cycles
● Large number of users - O(1k)● Organized in groups (known as VOs)
● Batch processing is a must● Typical user task requires > 1 CPU year● Users understand the need of splitting the problem
in many (semi-)independent tasks
Virtual Organizations
CLOSER 2012 - Apr 2012 glideinWMS - Ownership of VM 3
Our environment
● World-wide distributed computing a must● No single resource provider can provide enough
CPU for all the users– Both due to logistical and political constraints
● Until now this meant Grid computing● i.e. federation of independent batch sys. providers● and essentially all research-funded
● Our VOs are now considering Cloud computing, too● Both commercial and research-funded
Cloud==IaaS
CLOSER 2012 - Apr 2012 glideinWMS - Ownership of VM 4
CLOUDGRID
Grid vs Cloud
● Similarities● Both are a way to provision resources
● Differences relevant to this talk
● Only bare (virtualized) hardware provided by the resource provider
● Must install OS before running the actual payload– But more flexibility
● OS and system libraries provided by the resource provider
● User just executes his payload– Must play well with
provided OS
CLOSER 2012 - Apr 2012 glideinWMS - Ownership of VM 5
CLOUDGRID
Grid vs Cloud
● Similarities● Both are a way to provision resources
● Differences relevant to this talk
● Only bare (virtualized) hardware provided by the resource provider
● Must install OS before running the actual payload– But more flexibility
● OS and system libraries provided by the resource provider
● User just executes his payload– Must play well with
provided OS
Some user groups findthe cloud model easier
CLOSER 2012 - Apr 2012 glideinWMS - Ownership of VM 6
Why can Cloud be easier?
● Counterintuitive● Installing and maintaining a whole OS a big task!
● Yet, easier than trying to adapt existing scientific application● Most code not actively maintained● Often written making system assumptions
● Plus, most (virtualized) hardware uniform● And HW API variations are usually much smaller
than OS API variations
CLOSER 2012 - Apr 2012 glideinWMS - Ownership of VM 7
Why can Cloud be easier?
● Counterintuitive● Installing and maintaining a whole OS a big task!
● Yet, easier than trying to adapt existing scientific application● Most code not actively maintained● Often written making system assumptions
● Plus, most (virtualized) hardware uniform● And HW API variations are usually much smaller
than OS API variations
But someone stillhas to manage the batch jobs
EnterglideinWMS
CLOSER 2012 - Apr 2012 glideinWMS - Ownership of VM 8
glideinWMS
● An overlay batch system on top of dynamic resources● Based on the pilot principle
● Hides provisioning from final users● Looks and feels like
a batch systemon top of dedicated resources to them
Provider A
Provider BProvider CProvider AProvider A
Glideinresource pool
http://tinyurl.com/glideinWMS Sfiligoi, I. et al., (2009). The pilot way to grid resources using glideinwms. In Computer Science and Information Engineering, 2009 WRI World Congress on, 2, pp.428-32. doi:10.1109/CSIE.2009.950
CLOSER 2012 - Apr 2012 glideinWMS - Ownership of VM 9
glideinWMS architecture
● Three independent components● Glidein Factory – interface to resources● VO Frontend – resource provisioning logic● The actual WMS – seen by the final users
ResourceProviders
VO Frontend
GlideinFactory
WMS
PilotPilotPilot
PilotJob
Job Pilot
Monitor Request
Workload Management System == Batch system
CLOSER 2012 - Apr 2012 glideinWMS - Ownership of VM 10
glideinWMS architecture
● Three independent components● Glidein Factory – interface to resources● VO Frontend – resource provisioning logic● The actual WMS – seen by the final users
ResourceProviders
VO Frontend
GlideinFactory
WMS
PilotPilotPilot
PilotJob
Job Pilot
Monitor RequestVO Frontend
GlideinFactory
WMS
GlideinFactory VO Frontend
VO Frontend
WMS
N-to-Mrelationship
Workload Management System == Batch system
CLOSER 2012 - Apr 2012 glideinWMS - Ownership of VM 11
Multiple operation teams
● Few groups operate the whole glideinWMS● Typically separate groups for
● Glidein Factory● VO Frontend + WMS
Sfiligoi, I. et al., (2011). Reducing the human cost of grid computing with glideinWMS. In CLOUD COMPUTING 2011, pp. 217-21.
CLOSER 2012 - Apr 2012 glideinWMS - Ownership of VM 12
Multiple operation teams
● Few groups operate the whole glideinWMS● Typically separate groups for
● Glidein Factory● VO Frontend + WMS
● Glidein Factory typically generic● Essentially an abstraction layer to resources● Was the interface to Grid resources
– Adding the support for Cloud resources now
Sfiligoi, I. et al., (2011). Reducing the human cost of grid computing with glideinWMS. In CLOUD COMPUTING 2011, pp. 217-21.Andrews, W. et al., (2011). Early experience on using glideinWMS in the cloud. J. Phys.: Conf. Ser. 331 062014
CLOSER 2012 - Apr 2012 glideinWMS - Ownership of VM 13
Multiple operation teams
● Few groups operate the whole glideinWMS● Typically separate groups for
● Glidein Factory● VO Frontend + WMS
● Glidein Factory typically generic● VO Frontend the actual brain
● Contains resource provisioning logic● Provides pilot credential(s)● Customizes the pilot
Sfiligoi, I. et al., (2011). Adapting to the Unknown With a few Simple Rules: The glideinWMS Experience. In ADAPTIVE 2011, pp. 25-28.Jan 2012 VO Frontend Training Session at UCSD