1 Distributed Software Distributed Software Components, Components, Grid Web Services and Science Grid Web Services and Science Portals for Scientific Portals for Scientific Applications Applications Dennis Gannon Department of Computer Science And Pervasive Technology Laboratories Indiana University And NCSA Alliance & DOE SciDAC
51
Embed
1 Distributed Software Components, Grid Web Services and Science Portals for Scientific Applications Dennis Gannon Department of Computer Science And.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
1
Distributed Software Components,Distributed Software Components,Grid Web Services and Science Grid Web Services and Science
Portals for Scientific ApplicationsPortals for Scientific Applications
Dennis GannonDepartment of Computer Science
AndPervasive Technology Laboratories
Indiana UniversityAnd
NCSA Alliance & DOE SciDAC
2
Outline
• Building Grid Applications as “web service” enabled software components.– Composing Applications from components– How to create and deploy a component– How does it communicate with us?
• Using XML Message & Event Systems from Apps.
• Wrapping Scientific Apps as Grid Web Service component.– Scripting applications
• Grid Application Factories– A web service approach to launching apps
• Science Portals – the notebook approach.
3
The Big Picture
Resource layer1000s of PCs ->massive supercomputers
Information/NamingServices
Information/NamingServices
(co-)schedulingService
(co-)schedulingService
AccountingService
AccountingService
SecurityService
SecurityService
Event/MesgService
Event/MesgService
Discoveryservice
Discoveryservice
User HelpServices
User HelpServices
MonitoringService
MonitoringService
Peer Creation& resolution
Services
Peer Creation& resolution
Services
InformationRouting
InformationRouting
P2P / Grid Services LayerGrid Web Services Layer
User Portals/ Science Portals
Grid Application Factory ServiceLaunch, configure
And control
Application Factory
App InstanceApp InstanceApp Instance
4
Theme of this hour
• Encapsulating an application as a software component/grid service– Managing the application– Interacting with events/message system– Providing “remote control”
• Building an Application Service Factory as a secure web service– Encapsulating the hard parts of grid
application design and execution
• Accessing Applications from a Science WEB Portal
5
Grid Web Services Layer
• Major Grid Frameworks are evolving to support Web Service Interfaces– Avaki
• Provides an integration platform for web services
– Extensible Naming, Security,– Location transparency and support for migration
– Globus• OGSA: Refactor of basic architecture to support
WS interfaces.– New WS based versions of Gram, MDS, GridFTP.
– See Grid Forum presentations
6
Software Components & Web Services
• A Software Component Framework is– A standard for defining and composing and
interacting with a family of objects called components.
– A Component is a entity which satisfies the rules of behavior of a particular component architecture.
• Having an object model is not sufficient! • CAs have very specific rules for interaction and execution
environment.
• Examples:– Enterprise Java Beans, COM components, DOE
SciDAC CCA components, Corba Components.– Web services almost a component architecture.– Much of CCA can be built on top of WS technology
7
What about CAs for the Grid?
• Several good projects– Corba Based CAs: IRISA/INRIA, Gateway,
Miss. State, Rutgers– Imperial College
• Look at US Dept. of Energy CCA.– Designed for both parallel and Dist. Apps.– Grid version based on web services
standards.
8
The DOE Common Component Architecture
• A specification for a CA for large scale scientific computation– Both a DOE project and an open forum
• Sandia, llnl, argonne, pnnl, ornl, nasa (icase), utah, indiana, Maryland, ncsa, … about 50 people.
• Key Idea: dynamic composition by linking “ports” – Provides ports: interface of “services” provided– Uses ports: a call-site/use of a service to be provided
by another component.
Uses port-a call site for a remote functioninvocation
ProvidesPort –A set of functions which can be invoked remotely
9
Building Applications by Composition
• Connect uses Ports to Provides Ports.
getImage()
adjustColor()
Image tool graphical interface component
setImage(…)
Directconnect
Single address space
Remote component
Image ProcessingComponent
doFFT(…)
ACMEFFTComponent
ImageDatabaseComponent
10
CCA Application ExampleSolving a Time Dependent PDE from an Unstructured FEM Mesh
Lois McInnes, Lori Freitag, Boyana Norris, David Bernholdt, Jim KohlBen Allen and Rob Armstrong
11
Grid Component Communication
• How do grid components communicate?– Use a simple Remote Procedure Call Mechanism
• We Use Simple Object Access Protocol SOAP– Uses XML over HTTP and XML Schemas.– Java RMI implemented over SOAP and C++ bindings– .NET compatible. Oh, Boy! Also works well with WSDL.– Very robust but slooooow.
– Events/Messages• Objects encoded as XML Docs.
– Compatible with Soap
• Each provides port is a complete web service!– WSDL automatically generated when component
instantiated.
12
Programming A Simple Example
• Typical Scenario– A remote component that is “Grid Service”– A user Interface Component
• Launches grid component with “gram”• Connects to it.• Interacts with it.
Webster ServerComponent: A dictionary look-up service
Webster Client andUser Interface
public interface Define_idl{ public String define(String word);}
defineport
13
Class Hierarchy• To Program Client and Server
– Implement the define port object– Install it in a component.– Invoke it from a Client
Define_idl
ProvidesDefineUsesDefine
ProvidesPortUsesPort
ProvidesDefineImpl
ServerComponent
ClientComponent
14
Implementing the Server Component
• Class ProvidesDefineImpl must be implemented– This port just looks in the on-line
dictionary and returns the definition.
• Each component must have a method setServices( Services core){ … }– Where component declares and
installs ports.– Declare and create an instance of the
ProvidesDefineImpl class.
• Component is now a web service.
Webster Server
Component:
port
dictionary
15
The Webster Service Component
public interface ProvidesDefine extends Define_idl, ProvidesPort { }
public class ProvidesDefineImpl extends UnicastRemoteObject implements ProvidesDefine{ public String define(String word) { lookup word in dictionary and return defn}}
public class WebsterServer implements Component{
public void setServices (cca.Services core) { core.addProvidesPort(new ProvidesDefineImpl(), new PortInfoImpl("DefinePort", "http://www.extreme.indiana.edu/webster/wsdl#provides")); } …}
• Contact creation service to create instance of WebsterServer Service– uses component metadata to do this.
• Contact connection service to dynamically link its own uses port to provided port of Service
• Based on user input invoke remote service
Here is someText thatisDisplayed.
word
Webster Client
Creation service
Connection service
User interface
18
Clientpublic class WebsterClient implements Component{ public void setServices( cca.Services core ){ // declare UsesDefine uses port } public void instantiateRemote( ) { } public String getDefinition(String word) { UsesDefine definer = (UsesDefine) serv.getPort("UsesDefinePort"); return definer.define(word); } public static void main( String[] args) { serv = new ServicesImpl(); WebsterClient wc = new WebsterClient( ); wc.setServices(serv); wc.instatiateRemote( ); // set up and run user interface }
Where we usethe grid to launchthe server componentand connect to it.
Calls wc.getDefinition( word )and displays results
19
InstantiateRemote
ComponentWrapper remotec = new ComponentWrapper( "c:\\dbg\\xcat_tutorial\\xml\\webster.xml"); remotec.setMachineName("rainier.extreme.indiana.edu"); remotec.setCreationMechanism("gram"); remotec.createInstance(creationService);
Sample Script• Find an event channel and create a simple method for sending text messages• Use a process manage to route stdout and stderr to event stream
• Simple Python Grid Scripts are good for– Managing predefined sequences of grid
operations based on simple inputs from user• Select resources • Supply application parameters and launch remote
jobs• Establish peers and do peer discovery/resolution• Establish interaction with desktop apps (.Net/COM)• Subscribe to Event streams and respond to remote
application events• Launch and couple together remote
components/agents
– Can WSFL do all this?
28
Application Factory Service
• A service that understands a how to instantiate a running instance of an app component.– You provide it with appropriate
requirements initial conditions, etc. via an XML file
– The service• checks you credentials and
authorization• May consult resource broker• launches the app or runs the
appropriate grid script.
Appfactory
AppInstance
Provide me with an instance of application X
29
Anatomy of a Componentized Grid Service Application
• User Configures Application from Web Browser
• Sets application parameters• Launches job User Portals/ Science Portals
WebServerWeb
Server
• Web Server contacts appropriate application factory service (FS)• Supplies FS with task parameters • FS contacts Resource Broker and secures job
launches.• Returns App WSDL to server to browser
•Job begins execution •Publishes its contact point (WSDL) to discovery service•Begins publishing status events to event channel.
•Web server discovers application •Allows user to interrogate it or retrieve event traces
Appfactory
ResourceBroker
AppInstance
Eventchannel
Discserv
30
An Application Instance• Typical Instance
– Publishes event stream to “well known” channel.
– Has a Control Interface to allow portal level interrogation or remote steering
– May be linked to other components/services
LinksToOtherApps/services
Control Interface• check status• control messages
ApplicationInstance
Event stream
31
A Science Portal View of “programming” the Grid
• A Science Portal is a Web server that– Uses a “prepackaged” set of scripts to
• Get the users proxy cert from a cert repository• Configure a specific application to users needs• Contact the appropriate app factories• “prepackated scripts organized as “notebook”
– Look for event histories of application execution– Allow the user to contact and control the app.
User Portals/ Science Portals
Launch, configureAnd control
Appfactory
AppInstance
32
Security• The Problem
– When two components are linked how does the provider authenticate the user?
– When I wish others to use my component instance how do I handle authorization?
• Must be done on a port-by-port basis.• Same problems as faced by web services
• Solutions– Using SSL with globus proxy certs.– Components have a “control port” which can be
used to load authorized user lists.• Legion/Akenti/CAS projects have some good lessons to
teach us.
33
Some Examples
• To see what is required for a Grid programming, need to look at real examples.– Materials Science– Remote instrument control– Orbiting Space Junk– Grappa. The Atlas Griphyn portal
34
An Industrial Application• Materials ScienceMaterials Science
– NCSA Chemical Eng. TeamNCSA Chemical Eng. Team– Semiconductor ProcessingSemiconductor Processing
• Copper metalization on SiliconCopper metalization on Silicon• Optimizing Semicon Fab ProcessesOptimizing Semicon Fab Processes
– The Problem: The Problem: • Process Engineer at company X wants Process Engineer at company X wants
to try an experimentto try an experiment– to simulate proposed changes in chip to simulate proposed changes in chip
fabracation by changing some of process fabracation by changing some of process parameters.parameters.
– To accomplish this twoTo accomplish this twodifferent simulationsdifferent simulationsneeded to be coupled.needed to be coupled.
35
The Problem in Greater Depth– It is a Grid coupled multi-scale simulation problem
• The Monte Carlo simulation is at University Y and the Continuum simulation is at Company Z
– They decide to collaborate but• 1. The effort must protect security for each peer.
– Codes may be used but not moved– Data must be protected.
• 2. How does Engineer at X do this?
Engineer’s Grid Script1 Launch MC at Y2. Launch Cont at Z3. Link MC to Cont4. Execute MC & Cont5. Store results at X Y
Z
The GridThe Grid
X
ChemThis is a scientificnotebook examplethere is text andthere is also some interesting graphics
and some scripts
36
App Requirements and Observations
• Requirements– Strong coupling between two
simulation components– Combined simulation under
“remote control”– Security a top concern– Remote user is NOT a grid
programmer
• Observations– Requires “wrapping”
conventional apps as components
– On-the-fly component instantiation and coupling required
R. Bramley, D. F. McMullen, J. Huffman Indiana University I. Foster, G. von Laszewski, M. WestbrookArgonne National LaboratoryE. Westbrook,Lawrence Berkeley Laboratory
Remote Instrument Control And on-line data analysis
38
Application
Beam Source
Sample Crystal
Goniostat
CCD Array
LocalStore
CCD DataCollector
User CtlScan Set
StructureAnalysis
Local DSP,Buffering
CollaborativeVisualization
Image Framesto Struct. Analysis
Image Framesfrom Local St
Image Frames
Ctl Signals
Status
User Ctl
Munged Frames
39
Application 3: Tracking Space Junk
• Approximately 9,000 objects larger than 3 cm orbiting earth with speeds averaging about 10 Km/sec
• The Problem:– To launch something you
need to know if you will hit the space junk.
– Must predict thousands of orbit paths.
• (an homage to Entropia and Condor)
40
Computation Based on Peer to Peer Model
• The Algorithm– Divide objects into groups of size < 100– Send each group to an idle machine to compute orbit
trajectories– Report potential collisions back to root component.– Launch master and workers– Load tasks into workers. Workers report results back
to user as collision events.
OrbitCalculator
OrbitcalculatorThis is a scientificnotebook examplethere is text andthere is also some interesting graphics
and some scripts
EventChannel
OrbitCalculator
OrbitCalculator
OrbitCalculator
Master
41
GRAPPA
• Atlas Experiment– Large Hadron Collider
• 100 MB/sec (10PB/year)
• Grid Access Portal for Physics Apps– To provide Atlas physicists
with an point of access to Grid resources
• Manage submission of Athena and AtlSim jobs to condor cluster
• Manage interaction of analysis with physics data and grid services
events
detector
histogram
Mesg servc
Job op servc
Particle prop
Analysismodule
Grappa NotebookApp ManagerGrappa
This is a scientificnotebook examplethere is text andthere is also some interesting graphics
and some scripts
42
Conclusions
• Building an application programming environment for the Grid is very hard– Applications are distributed and very
complex. • Need reliability, security and programability
• Good news: 3 new technologies, when combined, show great promise– Software component architectures– Emerging Web Services Model– Peer-to-Peer frameworks
43
Dustbin
• Following slides are not used
44
Lots of Research Still needed
• Mobility– Component Migration (Condor moved my
component!)• How do remote references get updated?
• Efficiency– Need high throughput extensions to soap
• Make this work with WSDL?
– Protocol negotiation in the RMI stack.
• Application Execution Resource Estimation.– Negotiation of performance contracts.
• Need ways to automatically deploy grid apps/components on remote grid resources
45
More Topics• Lowering the barrier to Grid Computing
– Need to learn from P2P • Easy to install grid application gateways.• Works with DHCP, Firewalls, NATs, etc.
– Rapid Deployment of Secure Peer Networks• Jxta is very interesting!
• More work on dealing with Faults– Condor people understand this. – Self healing distributed apps
• Web Services & Work Flow– W3C & IBM WSFL– Is there a general workflow language?
• Understanding how to manage vast quantities of experimental data!
46
Something New: Web Services
• Making B2B really work requires more than traditional web server.
• A Web Service is– A network service that provides a functional
interface to remote clients. • designed to allow composition of services
• Big collaboration between big players– IBM, Microsoft, Oracle, HewlitPaq, …, sun
• Central to MS .NET and IBM services plans
lookUp(tickerSymbol )returns price
lookUp( SGI)
$1.73
47
Central Web Services Standards• Web Services Description Language (WSDL)
– An XML document that • Describes the interfaces types for each port
– The contents of messages it receives and sends• Describes the bindings of interfaces to protocols
– SOAP – XML over HTTP is default, others possible– Describes the access points (host/port) for the protocol
bindings
• Web Services Flow Langauge (WSFL)– An XML document that describes how a business
process invokes a set of services
• UDDI– A mechanism to discover WSDL docs for services
• JXTA– A suite of protocol services for building P2P
application– Peering an important concept. Protocols:
• Peer Membership – how do we form a peer group?• Peer Discovery- where are my peers?• Peer Resolution- sending queries to peers• Peer Information – peer status
– Communication• Pipe Binding – establish a virtual channel between peers• Endpoint Routing – establish communication path through
peer routers.
– Search: Query Routing Protocol
• Experiments with using P2P in Grids just starting