Holding slide prior to starting show
Dec 31, 2015
Holding slide prior to starting show
Portals for Workflow Composition and Service
Management
David W. Walker
School of Computer Science
Cardiff University
http://www.cs.cf.ac.uk/User/David.W.Walker/
System Portals
• We need portals to be able to handle tasks such as: :– Problem specification, resource discovery and job
composition– Service deployment and lifecycle management– Submission, execution, monitoring, and (maybe)
interaction
• These tasks are quite generic, rather than application-specific.
• Portal can behave rather like a “Grid Shell”
Web Services
• Everything is a (Web/Grid) service.
• This includes:– Computation routines– Access to files and databases– Components of the Grid infrastructure,
such as workflow enactment engines, resource monitors, etc.
A Common Approach to Workflow
• Visual service composition.
• Service interfaces and other metadata expressed in an XML-based service description document.
• Services registered with, and discovered through, a registry.
Visual Service Composition Portlet
• Applications and high-level services are composed from interacting services.
• Composition can be done graphically using a “drag and drop” style of interface to draw a workflow.
• In a workflow graph the nodes represent services and the edges represent data flow or control flow.
• Need special nodes to represent loop and conditional constructs.
• Need query mechanism to discover services
Workflow Description
• A workflow can be represented with an XML-based workflow description language such as WSFL, SWFL, SCUFL, BPEL4WS, etc.
• A workflow description document can be passed to a workflow enactment engine, which provides an execution environment for the workflow.
Two Hard Problems
• Semantic specification of applications• Scheduling of workflow nodes on distributed
resources.– Early binding model: bind to specific service/platform
at composition time (“validation”).– Intermediate binding model: bind at “compile” time
(when converting from XML to executable form).– Late binding model: bind dynamically at runtime.
• Later binding allows the use of more up-to-date information to make scheduling decisions.
Mobility
• Should (some) services be mobile and/or clonable?
• Allows the automatic creation of new services that are heavily used.
• May be important for autonomic grids.
• Generalises current data migration and replication concepts.
Communication and Notification
• Another important issue is how to carry out the data flow and control flow in a workflow.
• Need an asynchronous messaging system to support interoperability and to handle connection problems.
Portlets
• Portlet technology endorsed by GCE working group of GGF, e.g., JetSpeed
• Web services have user-facing ports that provide content to users.
• Portlets can be regarded as interfaces to that content, specified via an extension to WSDL.
• Portals can be built from collections of portlets, e.g., as a set of panels for job submission, job status, visualisation, etc.
• Portlets provide as component model for user interfaces in the same way that web services provide a component model to middleware.
Grid-Oriented Portal Work at Cardiff
• Visual Service Composition Environment (VSCE) for building applications based on service workflow.
• Service Workflow Language (SWFL) for describing composite applications.
• Tools for transforming between workflow graph SWFL executable code.
• Jini Service-oriented Grid Architecture (JISGA). An execution environment taking SWFL as input.
• Java-C Automatic Wrapper (JACAW). Wraps C code as Java using JNI for deployment as Web service in JISGA.
• Grid-Service-in-a-Box (GSiB). Supports service creation, deployment, management, and monitoring.
Motivation and Targets
• Easy-to-use tools are needed to allow scientists to take full advantage of the new Grid without them needing detailed knowledge of the underlying infrastructure.
• GSiB project is developing a visual problem-solving environment/portal for all service users (clients and providers).
Service Deployment
• A web service– Its public interfaces and bindings are described in
an XML document.– It is registered in a service registry to allow service
discovery and query.
• Usually additional code or wrapper is needed to make existing software available as a Web service – a lot of repeated work.
• Legacy problems exists.
Service Deployment
• JACAW (Java-C Automatic Wrapper) automatically generates the JNI wrapping code for a legacy code.
• Axis Java2WSDL utility generates WSDL document from the Java code.
• UDDI registry server.• UDDI4J provides a Java API for
interacting with a UDDI registry.
Service Deployment
JNI Wrapper
Legacy code
WSDL Document
UDDI Registry
JACAW
JAVA2WSDL
UDDI4J
Service Composition
• Graphical symbols for constructing a flow model.
Data input port
Data output port
Control input port
Control output port
Data link
Control link
Normal and assignment activity
Control activity
An Example
A
B C
For
retA = activityA(..);
for( i=0; i<100; i++){
activityB(retA, i);
}
retC = activityC();
retA
YES
NO
retA i
Service Workflow Language (SWFL)
<activity name="for"> <for setParallel="no"> <input message=“ForInput”/> <expression><![CDATA[index=0;index<100;index++]]></expression> </for></activity>
<dataLink name=“A_For” source=“A” target=“for”> <swflMap sourceMessage=“AOutput” targetMessage=“ForInput”>
<part> <sourcePart name=“retA”/><targetPart name=“retA”/> </part> </swflMap></dataLink>
To be continued
Service Workflow Language (SWFL)
<dataLink name=“For_B” source=“for” target=“B”> <swflMap sourceMessage=“ForOutput” targetMessage=“BInput”> <part> <sourcePart name=“retA”/><targetPart name=“retA”/> <sourcePart name=“i”/><targetPart name=“i”/> </part> </swflMap></dataLink
<controlLink name=“A_For” source=“A” target=“for” /><controlLink name=“For_B” source=“for” target=“B” controlPort=“YES”/><controlLink name=“For_C” source=“for” target=“C” controlPort=“NO”/><controlLink name=“B_For” source=“B” target=“for” />
Service Workflow Language (SWFL)
• SWFL is XML based for describing interacting Web services.
• Extends Web Service Flow Language (WSFL) by – supporting the application of all the
conditional and loop control constructs of the Java language to the composition of Web services.
– allowing more general data mappings.
Another Picture
ValidatorValidator
Create SWFLCreate SWFLDrawing ScreenDrawing Screen
swflJobswflJob
WorkflowEngine
WorkflowEngine
Visual Service Composition Environment
Execution Environment
SWFL2Java
• It converts the description of a job in SWFL into executable Java code.
SWFLDescriptionDocument
Java Executable
Code
DataflowGraph
ControlFlowGraph
FlowModel
Visual Service Composition Environment
• Collaborative code development environment using a visual programming tool for seamlessly integrating services into higher-level services or complete applications.
• Applications are created by plugging together services to form a workflow graph. Service metadata is used to check if interfaces are compatible.
• Different developers can place their components in a shared repository, e.g, UDDI or Jini lookup service.
• Output is in the form of SWFL.
JISGA
• JISGA consists of two main parts:– A WorkflowEngine service– A JobProcessor service
• Grid application is submitted to a WorkflowEngine service as SWFL.
• Sequential jobs are handled directly by the WorkflowEngine service.
• Parallel jobs involve multiple JobProcessor services.
What the Workflow Engine Is
• Processes service-based applications described in an XML-based workflow language.
• Determines the order of execution and generates the harness code.
• Executes the code which includes discovering services, invoking services, receiving results and sending results to clients.
What the Workflow Engine Does
• Converts a SWFL description of a composite service-based job into an executable Java code, and executes it.
Java Executable Code
<?xml…><JFlowModel……………………….……….….
SWFL description
IntermediateFlowModel
object
SWFL2Graph Graph2Java
SWFL2Java
To be continued…
Future Research• Further development of GSiB for creating,
deploying, managing and monitoring Grid applications.
• More experiments in integration of Jini services and Web services in JISGA.
• Demo applications to illustrate use of JISGA on UK e-Science Grid.
• Integration of better scheduling in JISGA.
Final Comments
• Service-oriented approach to Grid computing is generally agreed on
• But there are many options for how such Grids might be build (GT3, Jini, JXTA, etc)
• Ability to interoperate between different Grids is important.
• Need to get more experience with actually using services for e-Science.