The FI-WARE Project – Mediator GE Webinar
June 2013
Mediator GE Webinar
The FI-WARE Project – Mediator GE Webinar
Agenda
2
Mediator GE Overview
Mediator GE: Mediation Engine (TI)
Mediator GE: Dynamic Mediation API (Thales)
Demo: Event Brokering
Questions & Answers
The FI-WARE Project – Mediator GE Webinar
Overview
FI-WARE platform should be able to support services exposed through different protocols and technologies and enable the creation of new composite services. The mediator shall provide the glue between the service layer and the composition layer in order to enhance the composition capabilities of the compositor GEs.
The FI-WARE Project – Mediator GE Webinar
Capabilities
4
The main capability of the Mediator GE is to build an interoperability layer providing implementation for various enterprise integration patterns and data and protocol transformationsIt is a kind of broker between the service consumer and the service provider
Particularly it provides Data Mediation (Transformation), Protocol Mediation and routing capabilities using various mechanisms:
Using a library of built-in transformers in case of well-known formats Using templates for more flexible data/protocol transformation capabilities Using XML DSL (Domain Specific Language) in order to fulfill more
complex protocol/data transformation requirements or complex integration patterns (split/aggregations)
Using custom code (various programming languages are supported) to reach maximum flexibility in most complex scenarios
The FI-WARE Project – Mediator GE Webinar
Role in FIWARE
5
Basically the Mediator GE could play the role of centralizing and abstracting the access towards the heterogeneity of different devices or target applications exposing different capabilities or the same capabilities through different interfaces and data models.
In this scenario the mediator GE can be configured to implement different functionalities:
exposing a uniform interface and data model towards a plethora of heterogeneous devices and sensors
routing incoming messages towards the right target device or service based on the content of the message or other criteria
performing the role of splitting a coarse grained task into many specific items that shall be executed by different actors (device or applications), dispatching the split tasks and aggregating all results in order to return the whole result to the caller.
performing the role of event broker. A client application interested in events from a particular set of devices or other applications can subscribe to them using a particular service exposed by the mediator where all relevant event sources are configured.
providing an API for automatically solving particular mediation issues (based on semantic meta-data)
The FI-WARE Project – Mediator GE Webinar
Architecture
The mediator offers a set of available mediation tasks: the set of mediation capabilities that can be used via the mediator.
The mediator allows users to create and manage their mediation services: a mediation service is a virtual proxy towards a web service that executes a mediation task between the caller and the target service. The mediation task must be chosen from the set of available mediation tasks.
The mediator GE is used mainly by the Compositor Editor and Compositor Engine GEs within the FI-WARE platform. It provides a layer of virtual proxies to be used by the compositor instead of the target services in order to make the compositor able to support various kind of target services. Besides the FI-WARE platform also Future Internet applications or composite services on top of the FI-WARE platform can use the mediator as a service for their own purpose.
The FI-WARE Project – Mediator GE Webinar
Mediation Engine (TI)
The reference implementation of GE-Mediator provided by Telecom Italia is based on two open source frameworks:
•WSO2 ESB (http://wso2.com/products/enterprise-service-bus)
•Apache Camel (http://camel.apache.org)
Particularly the value added by Telecom Italia is related to:
•A custom mediation module based on Apache Camel in order to enhance the mediation capabilities of WSO2 ESB.
• Basically it is possible to add Camel route to a running instance of WSO2 ESB through a customization of the WSO2 ESB Web Admin GUI
•An enhanced per user logging capability• Basically it provides data related the who is using what inside the ESB
• The integration with a monitoring and disaster recovery tool based on JMX
The FI-WARE Project – Mediator GE Webinar
WSO2 Enterprise Bus Overview
The mediation built-in capabilities are provided through:
Service VirtualizationMessage Mediation
The main mediation capabilities are:Data TransformationProtocol mediationLoad Balancing and FailoverManipulating SOAP Headers (WS-Security) Iterate/clone messagesPolicy enforcement
The FI-WARE Project – Mediator GE Webinar
How to configure Mediators
WSO2 ESB provide a built in Mediator Catalog Usually a mediator is configured using XMLWSO2 ESB provides a Web GUI to help to configure Mediators without or less XML ESB mediation framework is extensible. It is possible to write custom mediators using:
Pure Java Spring Bean Scripting Languages: javascript, ruby
Examples of Mediator functionalities: Drop Mediator : Drops a message, stopping further processing of the message Send Mediator :Send the message out, using information inside the message or according to some external
configuration Script Mediator : Executes a scripting language with the message Filter Mediator : Filter messages using XPath XSLT Mediator : Do XSLT transformations on messages
The FI-WARE Project – Mediator GE Webinar
Mediators catalog: core mediators
Name Description
Send Send a message out
Log Logs a message
Property Set or remove properties associated with the message
Sequence Refer a sequence
Event Send event notifications to an event source
Drop Drops a message
Enrich Enriches a message
Enqueue Create an enqueue mediator
The FI-WARE Project – Mediator GE Webinar
Mediators catalog: filter mediators
Name Description
Filter Filter a message using Xpath (if else logic)
Out Inbuilt filter for choosing messages in ESB out path
In Inbuilt filter for choosing messages in ESB path
Switch Filter a message using Xpath (switch logic)
Router Route messages based on XPath filtering
Conditional Router Route messages based on 'Condition'
Validate Schema validation for messages
The FI-WARE Project – Mediator GE Webinar
Mediators catalog: transform mediators
Name Description
XSLT XSLT Transformations
XQuery XQuery Transformations
Header Set or remove SOAP Headers
Fault Create or remove SOAP Faults
Rewrite Create a rewrite mediators
The FI-WARE Project – Mediator GE Webinar
Mediators catalog: advanced mediators
Name Description
Cache XSLT Transformations
Clone Clones a message
Iterate Splits a message
Aggregate Combines messages
Callout Blocking web services calls
Transaction Executes a set of mediators transactionally
Throttle Limit the number of messages
RMsequence Reliable messaging
DBReport Writes data to a database
DBLookup Retrieve data froma database
Rule Execute Rule
Other…
The FI-WARE Project – Mediator GE Webinar
Mediators catalog: extensions
Name Description
Class Create and execute a custom mediator written in Java
POJOCommand Execute a custom command
Script Execute a mediator written in scripting language
Spring Create a mediator managed by spring
Other…
The FI-WARE Project – Mediator GE Webinar
Example 1: managing WS-Security
<proxy xmlns="http://ws.apache.org/ns/synapse" name="smsService_ws_security" Transports="https" statistics="disable" trace="disable" startOnLoad="true">
<target> <outSequence> <send /> </outSequence> <endpoint> <address uri="http://localhost:9000/services/SmsMessageAdapter" /> </endpoint> </target> <enableSec /> <policy key="conf:/repository/axis2/service-groups/smsService_ws_security/
services/smsService_ws_security/policies/UTOverTransport" /></proxy>
The FI-WARE Project – Mediator GE Webinar
Example 2: Data Transformation SOAP2REST
<proxy xmlns="http://ws.apache.org/ns/synapse" name="SoapToRest_Sms" transports="http" statistics="enable" trace="enable" startOnLoad="true"> <target> <inSequence> <property name="Accept" value="application/xml" scope="transport" /> <log level="full" /> <router> <route xmlns:soaptorest="http://telecomitalia.it/ictservice/smsService/soaptorest" expression="$body/soaptorest:sendSms"> <target sequence="anon"> <sequence> <xslt key="gov:/transformations/trasformSOAP2RESTPayload" /> <property name="HTTP_METHOD" value="POST" scope="axis2" type="STRING" /> <log level="full" /> <send> <endpoint> <address uri="http://polaris.cselt.it:9763/SmsServiceRest/rest/sms" format="rest" /> </endpoint> ….........................................
</proxy>
The FI-WARE Project – Mediator GE Webinar
Example 2: Data Transformation SOAP2REST XSLT
<?xml version="1.0" encoding="UTF-8"?><xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform"><!--Our SoapToRest service, built to expose an existing Rest service as a Soap service, for each method uses as payload the Rest service payload incapsulated in an element whoose name is that we gave to the soap method.Here we remove this incapsulation, to get payload that we have to send to the Rest service.-->
<xsl:output method="xml" indent="yes"/>
<xsl:template match="/node()/node()"><xsl:copy>
<xsl:copy-of select="./*"/></xsl:copy>
</xsl:template>
</xsl:stylesheet>
The FI-WARE Project – Mediator GE Webinar
Example 3: Protocol Transformation TCP2HTTP
<proxy xmlns="http://ws.apache.org/ns/synapse" name="TCPtoHTTP_mail" transports="tcp" statistics="enable" trace="enable" startOnLoad="true"> <target endpoint="fiwareMailproxy"> <inSequence> <log level="full" /> </inSequence> <outSequence> <send /> </outSequence> </target></proxy>
The FI-WARE Project – Mediator GE Webinar
Example 4: Throttling capability
The throttling capability means that it is possible to restrict the usage of a mediation service using different parameters.
Range - The IP address range or the domain is restricted from accessing the service. Requests from such clients will be restricted based on the specified values.
The restriction can be : Deny or Control
Control means that it is possible to set usage limitation based on: - Maximum Request Count (MRC) - Prohibit Time Period (PTP)
The FI-WARE Project – Mediator GE Webinar
Example 5: Loadbalancing capability
The loadbalancing capability means that it is possible to configure a mediation service to use different service endpoints in order to balance the calls using a round robin algorithm. More complex loadbalancing algorithms currently can be configured only using a custom mediator.
<session type="http|simpleClientSession"/>? <loadBalance [policy="roundRobin"]> <endpoint .../>+ </loadBalance>
The FI-WARE Project – Mediator GE Webinar
Dynamic Mediation API (DMA)Optional Library provided by Thales
22
Les
info
rma t
ion s
co
nten
ues
d an s
ce
doc
um
en t
de
meu
ren
t la
pro
prié
té e
xclu
siv e
du
gro
upe
TH
AL
ES
et n
e d
oiv
en t
pa
s ê
tre
divu
lgu
é es
à l 'e
x té
ri eu r
du
Gro
up
e .
Dynamic Mediation API (DMA)
Main features and assumptions:• Optional library delivered as part of the Mediator GE
• Can be used “internally” by the Mediator as part of a created “Mediation Service”
• API can also be used directly by third-parties for solving particular mediation issues (as shown in this presentation)
• FI-PPP projects: use under the conditions established in the FI-PPP Collaboration Agreement. Distributed under Thales license as a binary-only closed-source library
• Useful when syntactic mismatch occurs (at design or runtime) between Service consumer’s needs and Target Service offer.
• Additional semantic information (meta-data) + domain ontology is used automatically by the DMA for bridging the syntactic gap between a Service Consumer and a Target service => automatic data conversion
23
Les
info
rma t
ion s
co
nten
ues
d an s
ce
doc
um
en t
de
meu
ren
t la
pro
prié
té e
xclu
siv e
du
gro
upe
TH
AL
ES
et n
e d
oiv
en t
pa
s ê
tre
divu
lgu
é es
à l 'e
x té
ri eu r
du
Gro
up
e .
DMA Overview
SSL - Smart Systems Lab
Ontology
Automaticconversion
Mismatch
Service Request
TargetService
Data Request Data Target
DMA features
(1) Automatically deal with service (request / target) syntactic mismatch
Use semantic-based characterization of service features (methods, parameters,…)
Supports direct target service invocation through library API
Communication protocol agnostic (SOAP, REST)
(2) Automatic data mediation between services Data must be described using XSD and
semantically-annotated (3)Use of ontology as business model reference
(1) Automatically deal with service (request / target) syntactic mismatch
Use semantic-based characterization of service features (methods, parameters,…)
Supports direct target service invocation through library API
Communication protocol agnostic (SOAP, REST)
(2) Automatic data mediation between services Data must be described using XSD and
semantically-annotated (3)Use of ontology as business model reference
DMA Library
1
2
3is_equivalent
24
Les
info
rma t
ion s
co
nten
ues
d an s
ce
doc
um
en t
de
meu
ren
t la
pro
prié
té e
xclu
siv e
du
gro
upe
TH
AL
ES
et n
e d
oiv
en t
pa
s ê
tre
divu
lgu
é es
à l 'e
x té
ri eu r
du
Gro
up
e .
DMA API typical use cases
1. Consumer needs to invoke a specific operation on an available target service and already has a structured payload he wants to send
Potential mismatch at data-model level payload needs to be automatically converted by the DMA
Semantic annotation of consumer-side payload and target service offer mandatory DMA will find the semantically matching operation in the target service
2. Consumer needs to invoke a specific operation on an available target service and doesn’t have an imposed structured payload to send
Consumer will provide DMA with a list of parameters with simple data type Consumer-side data-model undefined payload will be automatically created by
the DMA Semantic annotation of parameters and target service offer still mandatory DMA will find the semantically matching operation in the target service
3. In both cases, DMA can be used either to build the right payload (and stop), or to also deal with target service invocation
25
Les
info
rma t
ion s
co
nten
ues
d an s
ce
doc
um
en t
de
meu
ren
t la
pro
prié
té e
xclu
siv e
du
gro
upe
TH
AL
ES
et n
e d
oiv
en t
pa
s ê
tre
divu
lgu
é es
à l 'e
x té
ri eu r
du
Gro
up
e .
DMA API (1/4)
DMA provides 4 Dynamic Mediation Tasks that are accessible as Java methods or SOAP endpoints:
String dynamicallybuildServicePayloadNoOPWithParameters( String usdlURI, String serviceID, String operationConcept , List<SemanticParamItem> inputs)
throws MediatorException;
Method description:• Purpose: using a list of unstructured data as input parameters, dynamically building an XML payload adapted to a given service and linked to an operation concept.
26
Les
info
rma t
ion s
co
nten
ues
d an s
ce
doc
um
en t
de
meu
ren
t la
pro
prié
té e
xclu
siv e
du
gro
upe
TH
AL
ES
et n
e d
oiv
en t
pa
s ê
tre
divu
lgu
é es
à l 'e
x té
ri eu r
du
Gro
up
e .
DMA API (2/4)
String dynamicallybuildServicePayloadNoOPWithPayload( String usdlURI, String serviceID, String operationConcept , String saxd_payload, String payload )
throws MediatorException;
Method description:• Purpose: using a structured payload (with its associated data schema) as input parameter, dynamically building an XML payload adapted to a given service and linked to an operation concept.
27
Les
info
rma t
ion s
co
nten
ues
d an s
ce
doc
um
en t
de
meu
ren
t la
pro
prié
té e
xclu
siv e
du
gro
upe
TH
AL
ES
et n
e d
oiv
en t
pa
s ê
tre
divu
lgu
é es
à l 'e
x té
ri eu r
du
Gro
up
e .
DMA API (3/4)
String dynamicDataMediationInvocationNoOPWithParameters ( String usdlURI, String serviceID, String operationConcept , List<SemanticParamItem> inputs , List<SemanticParamItem> awaitedOutput)
throws MediatorException;
Prototype description:• Purpose: dynamically searches and invokes a service capability based on the given operation concept. The operation of the service, if found, is called with a dynamically built payload based on the list of unstructured data passed as input parameters. • Note: the expected data format of the result(s) is defined by the last input parameter awaitedOutput. The result is then adapted before being returned.
28
Les
info
rma t
ion s
co
nten
ues
d an s
ce
doc
um
en t
de
meu
ren
t la
pro
prié
té e
xclu
siv e
du
gro
upe
TH
AL
ES
et n
e d
oiv
en t
pa
s ê
tre
divu
lgu
é es
à l 'e
x té
ri eu r
du
Gro
up
e .
DMA API (4/4)
String dynamicDataMediationInvocationNoOPWithPayload ( String saxsd_payload, String serviceID, String payload, String wanted_response, String operationConcept)
throws MediatorException;
Prototype description:• Purpose: dynamically searches and invokes a service capability based on the given operation concept. The operation of the service, if found, is executed using the input payload, first adapted to the service. • Note: the data format of both input payload and expected result(s) is defined by the input parameter saxd_payload. The result is then adapted before being returned.
29
Les
info
rma t
ion s
co
nten
ues
d an s
ce
doc
um
en t
de
meu
ren
t la
pro
prié
té e
xclu
siv e
du
gro
upe
TH
AL
ES
et n
e d
oiv
en t
pa
s ê
tre
divu
lgu
é es
à l 'e
x té
ri eu r
du
Gro
up
e .
A: Consumer needs to invoke a specific operation on an available target service and doesn’t have an imposed structured payload to send
IMediator mediator = MediatorEntry.getInstance();;
String operationConcept = "http://www.example.com/Calculator.owl#Addition";
SemanticParamItem param1 = new SemanticParamItem();
param1.setConcept("http://www.example.com/Calculator.owl#OperandeGauche");
param1.setValue("12");
SemanticParamItem param2 = new SemanticParamItem();
param2.setConcept("http://www.example.com/Calculator.owl#OperandeDroite");
param2.setValue("5");
SemanticParamItem param3 = new SemanticParamItem();
param3.setConcept("http://www.example.com/Calculator.owl#Somme");
List<SemanticParamItem> result = null;
result = mediator.dynamicDataMediationInvocationNoOPWithParameters(usdlURI, serviceID, operationConcept, inputs, outputs);
for (SemanticParamItem item : result){
assertTrue(item.getValue().equals("17"));
System.out.println(item.getConcept());
System.out.println(item.getValue());
}
Use case: simple calculator exemple – Case 1
30
Les
info
rma t
ion s
co
nten
ues
d an s
ce
doc
um
en t
de
meu
ren
t la
pro
prié
té e
xclu
siv e
du
gro
upe
TH
AL
ES
et n
e d
oiv
en t
pa
s ê
tre
divu
lgu
é es
à l 'e
x té
ri eu r
du
Gro
up
e .
B: Consumer needs to invoke a specific operation on an available target service and already has a structured payload he wants to send
IMediator mediator = MediatorEntry.getInstance();;
String wanted_response = "AdditionResponse";
String payload = Util.getLocalPathFromResource("fiware/payload.xml");
SemanticParamItem param3 = new SemanticParamItem();
param3.setConcept("http://www.example.com/Calculator.owl#Somme");
String saxsd_payload = Util.getLocalPathFromResource("fiware/saxsd_inputs_outputs.xml");
System.out.println(saxsd_payload);
String operationConcept = "http://www.example.com/Calculator.owl#Addition";
String result = null;
result = mediator.dynamicDataMediationInvocationNoOPWithPayload(
saxsd_payload, usdlURI, serviceID, payload, wanted_response, operationConcept);
assertTrue(result != null);
assertTrue(result.contains("15"));
Use case: simple calculator exemple – Case 2
31
Les
info
rma t
ion s
co
nten
ues
d an s
ce
doc
um
en t
de
meu
ren
t la
pro
prié
té e
xclu
siv e
du
gro
upe
TH
AL
ES
et n
e d
oiv
en t
pa
s ê
tre
divu
lgu
é es
à l 'e
x té
ri eu r
du
Gro
up
e .
Hypothesis of work:• Payload:
<Addition>
<operandeg>12</operandeg>
<operanded>3</operanded>
</Addition>
• Annotated schema for the payload and/or request/response using SAWSDL:<xs:element name="Addition">
<xs:complexType>
<xs:sequence>
<xs:element name="operandeg" sawsdl:modelReference="http://www.example.com/Calculator.owl#OperandeGauche" type="xs:string"/>
<xs:element name="operanded" sawsdl:modelReference="http://www.example.com/Calculator.owl#OperandeDroite" type="xs:string"/>
</xs:sequence>
</xs:complexType>
</xs:element>
<xs:element name="AdditionResponse">
<xs:complexType>
<xs:sequence>
<xs:element name="resultat" sawsdl:modelReference="http://www.example.com/Calculator.owl#Somme" type="xs:string"/>
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:schema>
Use case: simple calculator exemple – Case 2
The FI-WARE Project – Mediator GE Webinar
DEMO
The FI-WARE Project – Mediator GE Webinar
Example Scenario: Task Dispatcher and Event Brokering
33
The FI-WARE Project – Mediator GE Webinar
Example Scenario: Task Dispatcher and Event Brokering
34
In this scenario the mediator can play the roles of task dispatcher using the split/aggregator pattern and event broker.
As a task dispatcher the mediator receive a coarse grained task from the task manager and , based on some criteria related to the content of the task, split it into specific items and dispatch them to the right target (human or application)
As an event broker the mediator exposes a service in order to receive events from one or more event sources and allows applications to subscribe to an event topic in order to get related events when they are received by the broker. In this way the event broker collect events from devices, optionally filter them and send them to the current subscribers.
Moreover, in this demo scenario the mediator play also the role of centralizing the calls to third party services used by the client (subscribed) applications (i.e. email and sms services).
In order to show the load-balancing capability of the mediator GE, the email service mediation can be configured to use two different endpoints balanced through a simple round robin algorithm
The FI-WARE Project – Mediator GE Webinar
References
Fiware Apps Wiki Architecture https://forge.fi-ware.eu/plugins/mediawiki/wiki/fi-ware-private/index.php/FIWARE.OpenSpecification.Apps.Mediator
Fiware Apps Wik Open Spec: https://forge.fi-ware.eu/plugins/mediawiki/wiki/fi-ware-private/index.php/FIWARE.OpenSpecification.Apps.Mediator
WSO2 ESB docs site: http://wso2.com/products/enterprise-service-bus