1 Applications of Intelligent Agent Technology to the Grid Carl Kesselman Information Sciences Institute University of Southern California Globus Alliance In collaboration with Nick Jennings & Ian Foster 2 Why the Grid? Origins: Revolution in Science Pre-Internet Theorize &/or experiment, alone or in small teams; publish paper Post-Internet Construct and mine large databases of observational or simulation data Develop simulations & analyses Access specialized devices remotely Exchange information within distributed multidisciplinary teams
18
Embed
Applications of Intelligent Agent Technology to the Grid
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
1
Applications of Intelligent Agent Technology to the
Grid
Carl Kesselman
Information Sciences Institute
University of Southern California
Globus Alliance
In collaboration with Nick Jennings & Ian Foster
2
Why the Grid?Origins: Revolution in Science
Pre-InternetTheorize &/or experiment, aloneor in small teams; publish paper
Post-InternetConstruct and mine large databases of observational or simulation data
Develop simulations & analyses
Access specialized devices remotely
Exchange information within distributed multidisciplinary teams
2
3
Why the Grid?New Driver: Revolution in Business
Pre-InternetCentral data processing facility
Post-InternetEnterprise computing is highly distributed, heterogeneous, inter-enterprise (B2B)
Business processes increasingly computing- & data-rich
Outsourcing becomes feasible service providers of various sorts
Growing complexity & need formore efficient management
4
Common Requirements
Dynamically link resources/servicesFrom collaborators, customers, eUtilities, …(members of evolving “virtual organization”)
Into a “virtual computing system”Dynamic, multi-faceted system spanning institutions and industries
Configured to meet instantaneous needs, for:
Multi-faceted QoX for demanding workloadsSecurity, performance, reliability, …
3
5
The Grid“Resource sharing & coordinated problem solving in dynamic, multi-institutional virtual organizations”
1. Enable integration of distributed resources
2. Using general-purpose protocols & infrastructure
3. To achieve better-than-best-effort service
6
0
10
20
30
40
50
60
70
8:00
8:30
9:00
9:30
10:0
010
:30
11:0
011
:30
12:0
012
:30
13:0
013
:30
14:0
014
:30
15:0
015
:30
16:0
016
:30
17:0
017
:30
18:0
018
:30
Num
ber o
f Par
ticip
ants
UIUCColorado
Grid in Practice:Earthquake Engineering
Example
Illin
ois
Colo
rado
Illinois (simulation)
4
Grid2003: An Operational Grid28 sites (2100-2800 CPUs) & growing400-1300 concurrent jobs7 substantial applications + CS experimentsRunning since October 2003
Koreahttp://www.ivdgl.org/grid2003
8
(Very Small)Example
Workflows
Genome sequence analysis
Physicsdata
analysis
Sloan digital sky
survey
5
10
How it Really Happens(with the Globus Toolkit)
WebBrowser
ComputeServer
GlobusMCS/RLS
DataViewerTool
CHEF ChatTeamlet
MyProxy
CHEF
ComputeServer
Resources implement standard access & management interfaces
Collective services aggregate &/or
virtualize resources
Users work with client applications
Application services organize VOs & enable access to other services
Applications of the framework(Compute, network, storage provisioning,
job reservation & submission, data management,application service QoS, …)
12
The Need for Automation:Critical if Grid is to Scale
Who contributes & gets to consume what?Policy negotiation, enforcement, auditing
How do I schedule jobs & data movement?Adaptive scheduling
Who can be trusted to do what?Community membership, reputation, trust negotiation, intrusion detection
Why do things fail?Failure detection, problem determination, fault isolation, system adaptation
7
13
Virtual Organizations
VOs play a central role in GridsWidely adopted approach in scientific applns.
Manually established (acceptable use policy, resource allocation, membership, services available)
Some infrastructure support (e.g. VO based discovery, membership services)
More automation needed Need autonomous VO formation
14
VO FormationNeed to:
Determine policy (negotiation, trust management)Determining membership and role (coalition formation)Creating VO wide services (global behaviors)Managing work (collaborative problem solving, workflow management)Evolution
Related to coalition formation (Lesser’s talk yesterday)May be driven strictly by policy
“Coalition of the willing….”
May be driven by contribution or utility to goalse.g. N lowest bidders for a service (Normana, Preecea, et. a.l., 2004),
May be dynamic and nestedMapping of workflow to defined agent setCreation of global services
8
15
Policy Federation
Policy negotiation between membersCreation of coalition policy (e.g. Ao & Minsky 2003)
Reasoning about trustRule based, or logic based decision making (e.g. KAoS ontologies, policy services)
Application of policyAssignment of role (Sims, Corkill, and Lesser 2004)Learning and adaptation of trust & reputation (Ramchurn, Huynh, Jennings, 2004)
16
Community Authorization Service
Domain A
PolicyAuthority
Domain B
Sub-Domain A1 Sub-Domain B1
CAS identity"trusted"
Requestor
Server
request +CAS assertions
VirtualOrganization
Domain
capabilityassertions
CommunityAuthorization Svc enforcement
on CAS-identity andrequestor's capabilities
9
17
Using KAoS Policy Services
Stub
Grid Service
KAoSGridServiceCredential
Credential
Client
KAoS
(Checks whether the arrows match)
(The arrows represent SOAP messages)
(if authorized)
WS SecurityRequest Handler
(From M. Johnson, 2003)
18
Large number of users utilizing resources either through direct job-submission or through intermediaries (brokers)
Issues of how to:
identify potential providers
choose between alternatives
Grid Resource Management
10
19
Resource selection: Matchmaking
Classified Advertisement (ClassAds) describe:Attributes of consumers & providers
Constraints on compatible entities
Preference (rank)
Matchmaker matches compatible ClassAdsSymmetric match on attributes & constraints
Selection based on ranking function
(Raman, Livny and Solomon 1998)
20
Matchmaking: The Protocol
Matchmaker
Resource(Provider)
Customer(Requester)
1. Advertise 1. Advertise
2. Match
3. Notify 3. Notify
4. Claim
Providers and requesters advertise to a Matchmaker
Matchmaker discovers compatible advertisements
Matchmaker notifies compatible parties
Notified entities claim each other and collaborate
11
21
Matchmaking as a Contract Net
Matchmaker acts as communication hub
Can apply more sophisticated evaluation criteria
Auction, market based, …
Still need to address claiming protocolMatch is only advisory
“UMM: A dynamically adaptive, unstructured multicast overlay”
M. Ripeanu et al.
A
E
B
D
C
A’
E’
B’
D’
C’
A”
E”
B”
D”
C”
Applicationoverlay
Baseoverlay
Physicaltopology
0
2
4
6
8
10
024
048
072
096
012
0014
4016
8019
2021
6024
0026
4028
8031
2033
6036
0038
40
Time (sec)
RD
P
0
2
4
6
8
10
12
Max
xim
umlin
k st
ress
.
MaxRDP95% RDP90%RDPStress
10 nodes fail then
rejoin 900s later
RDP=1
RDP=2
34
Application of Agents
Similarity to Agent-based Information Retrieval
Coalition of indexes
Distribution of index and catalog function (Zhang & Lesser)
Domain optimized, distributed searchDynamic deployment, tasking and configuration
18
35
SummaryAgents & Grid share a common interest in robust, scalable open distributed systems
Large-scale scientific and commercial Grid infrastructure being build with expanding applications space
Many places where agent technology can “drop in”
Problem perspective and details sometimes different
May need to build understanding across communities
36
For More InformationBrain meets brawn: Why Grid and agents need each other, Proc. 3rd Int. Conf. on Automated Agents and Multi-Agent SystemsGlobus Alliance