ESB 10 weken theorie en praktijk Introductie MOM (Message Oriented Middleware) Verdieping mogelijk in een afstudeerstage Verdiepen kennis van XML.

ESB• 10 weken theorie en praktijk• Introductie MOM (Message Oriented

Middleware)• Verdieping mogelijk in een

afstudeerstage• Verdiepen kennis van XML

Agenda

Week 1: Theorie MOM + demo Tibco BusinessWorksWeek 2: Intermezzo XML + hands-on XML + WSDL Week 3: Theorie Fiorano (1 uur) + hands-onWeek 4: Handson Fiorano Hands-on (deel II)Week 5: Theorie + introductie Webmethods + indelen

in groepen + uitdelen groepsopdrachtWeek 6: College WebmethodsWeek 7: Hands-on Webmethods (I) – groepsopdrachtWeek 8: Hands-on Webmethods (I) - groepsopdrachtWeek 9: Hands-on Webmethods (I) - groepsopdrachtWeek 10: Hands-on Webmethods (IV) –

groepsopdracht

Even voorstellen

Marcel GrauwenLead Consultant

Ir. M.A. GrauwenA Hofmanweg 5a2031 BH, HaarlemThe Netherlands

Phone:  +31 23 750 5175Fax:   +31 23 750 5176Mobile: +31 651 827 062E-mail: [email protected]:http://www.glomidco.com

Global Middleware Consultancy

Guihen AbrialConsultant

Guihen AbrialA Hofmanweg 5a2031 BH, HaarlemThe Netherlands

Phone:  +31 23 750 5175Fax:   +31 23 750 5176Mobile: +31E-mail: [email protected]:http://www.glomidco.com

Global Middleware Consultancy

BPM• Hoe zien jullie BPM?

SOABPM

BAM

1SOA

BPM

BAM

2

SOA BPM

BAM

3 SOA BPM

BAM

4MOM

What is Middleware?•Which types of Middleware can be identified?

–Message Oriented Middleware (MOM)–Object Oriented Middleware (OOM)

•GloMidCo’s definition of MOM:

“ Standardized integration that combines: messaging, web services, data transformation, workflow and intelligent routing to reliably connect and coordinate the interaction of applications across the enterprise ”

Introduction MOMMessage Oriented Middleware

(MOM) defined as the whole of:• Enterprise Applicatation Integration

(EAI)• Service Oriented Architecture (SOA)• Business Process Management (BPM)• Business to Business (B2B)• Business Activity Monitoring (BAM)

MOM overview

Provision-ing

BillingInventory CRM1TT CRM2CallCenter

Common Data Model

EAIFine

Grained Service

Course Grained Service

AddCustomerPOSOA Governance

BPM

BAM

(web) Service enabled

Policy Management

Service (UDDI)

Registry

Service Monitoring

Enterprise Service Bus

Orchestration

Mediation Routing Transport

BPM Service Callout

StateFull Concept of steps

TasksUsers

Manual Steps

ESB triggers workflow

Collect Transactions

Check Business

Rules

Business Logic Rules Engine

trigger corrective

actions

B2B

Concepts of Partners

Usage of standard protocols

Internet

Multiple transport channels

PKI security.

Popular MOM products 2005

• Sonic• Tibco• SeeBeyon

d• …• Fiorano **

** no NL customers

Popular MOM products 2008SOA suites:

• Sonic (Progress)• Tibco• Software AG /

Webmethods• BEA/Oracle• IBM Websphere• …

SOA Management:-Amberpoint (OEM)-Actional (Progress)

Popular BPM products 2007

• Tibco iProcess (staffware)

• Software AG / Webmethods

• Cordys• Lombardi• ….

Business Process (BP)• Business Process: a set of activities consisting of a set of

logical related tasks that, when performed in the appropriated sequence and according to the correct business rules, produce a business outcome. (Business process is stateless)

• Implements deployed process flow definitions• Event triggers (forks) an instance of BP• Each instance (also often called job) is stateless• Each step within the BP consists of evaluating a sequence of

activities (Tibco and SeeBeyond terminology) or services (Sonic terminology)

• The component in which the BP’s are running is generally called a process engine (PE)

Tibco: BusinessWorksFiorano: Fiorano ESBSonic: Sonic ESB

Default the ESB is state-less (note: a workflow process is in contrary state-full!)

Workflow (BPM)• Workflow add state-fullness to MOM (e.g. using a DB)• Used for long term processes• Concept of: Users, Task, Processes• Compare with State Machine WorkflowNote: What the vendors call BPM, is in fact an advanced

workflow implementation that can use SOA capabilities and is based on modeling paradigms like BPEL

• Business Process Management (BPM) is a set of software systems, tools, and methodologies that addresses how organizations identify, model, develop, deploy and manage Business processes. (BPM enhances state fullness and includes workflow )

• BPEL (Business Process Execution Language) models the Business Processes (see also WS-BPEL for web-services)

• Business Process Modelling Notation (BPMN) is an open (OMG) notation standard for designing vendor independed BMP processes

ESB: from “Spagetti” to “Lasagna”

Provision-ingBilling

Inventory

CRM1

TT

CRM2

CallCenter

Common Data Model

Inventory TT

ProvisioningBillingCRM1 CRM2

CallCenter

DataW/house

Adapters

Enterprise Service

Bus

all Application data transformed

(mapped) to one canonical Common

Data Model (and visa vera)

ProvisioningBilling

Inventory

CRM1

TT

CRM2

CallCenter

DataW/house

Common Data Model

map am2cdm

Appl.A adapter

Common Data Model

Wire FormatAPImap

cdm2amESB ESB

Application Model

ABusiness

Logic

Application Model

B

ESB adapter Appl.B

Wire FormatXML

BAM Operational Monitoring

SLA reporting

“Klant” in Application A

(AM)

is mapped to “Customer”

(CDM format)

Customer” is mapped back into “Client” in Application B

(AM)

The company is using “Customer”

as the Data Dictionary Model

(CDM)All applications in the company are mapped to the

CDM (Canonical Data

Exchange)

Business Logic should be applied in the CDM layers only!

Mapping:Ability to convert data formats between applications

Distributed data transformation

Distributed implies multiple ‘layers’ of data transformation

A layered service approach generally separates the business logic from application code

Publish and Subscribe message bus • Uses subject based addressing e.g. RSF.REC.KLM.AS• Real time• In principle non-transactional (can be made transactional with an

additional layer)• Possible decentralized (peer 2, e.g. Tibco RV) or centralized (JMS

server)• Possible fully distributed (e.g. with Tibco RV)• One to many• Fully de-coupled• Using broadcast or multicast

Sonic: Sonic MQ (JMS Topic)Tibco: RendezVous (RV) or EMS(JMS Topic)Fiorano: Fiorano MQ (Topic)

Event-Driven(Publish - Subscribe)

Producer ConsumerA

ConsumerB

ConsumerC

ESB type Pub/Sub

ESB type – Message Queue

Queue based message bus• Uses Topics to denote the queues• Queue is like a stack (FIFO)• Centralized server components required (node)• Client connects to a node• De-Coupling (not fully implemented)• Supports the HUB and SPOKE model• Optionally multiple nodes possible which can forward queues• Optionally supports two face commit (with or without external TM)• Optionally prioritizing

Sonic: Sonic MQTibco: RVCM, RVTX, EMS (queue)SeeBeyond: JMS(Queue)Fiorano: Fiorano MQ (queue)

Also: Websphere MQ, JMS queue, Oracle AQ, MSMQ, ….

Integration Components• Message bus drivers (deamons, client software)• Adapters (e.g. file, JMS, Http, RV, SAP rfc, Peoplesoft, MQ)• Routing components• Mapping components (!)• Process (flow) engine components• Agents (e.g. monitoring, process control)• Central repository components (e.g. storing configurations)• Correlation components (e.g. BAM)• Workflow components (state-full components, BPM)• Business Connectors (e.g. SOAP communication with

external associates)• Portal components and plug-ins

Adapters• Adapters are an important part of MOM• COTS (commercial of the shelve)

component which connects an external API to the ESB API

• Many adapters are available, often 3rd party adapters are used

• Adapters are created with an SDK, some vendors provide the SDK to create your own legacy adapter

• Adapters can be both FT or Load balanced• Adapters optionally contain mapping

functionality

Business Connector• B2B, B2C component• COTS component used to connect the ESB via a standard

component to the outside world• Support for industry standards like:RosettaNet, CIDX, xCBL,

cXML, ChemXML, ebXML, OASIS, Siebel UAN, EDI etc• Control of messages send and received, certificates

administration, and other B2B facilities.• Transports supported: (s)http, ftp, soap, as2, email, etc.• Often equipped with additional transformation capabilities

(private processes)

Tibco: BCSonic: Collaboration Server (obsolete)SeeBeyond: eXchange

Asynchronous vs Synchronous

UML sequence diagramComponent A

∆ T

…

…

Component B Component C Component D

… …

synchronous

asynchronous

asynchronous

correlation?

Asynchronous process are de-Coupled (no waiting process, just fire and forget)

Synchronous processes only when Request/Reply interactions are required. (timeouts can occur when reply is not received in time)

Generally try to avoid synchronous transactions, or break them into two asynchronous transactions if possible!

ReliabilitySeveral levels of Message reliability:• Best delivery mode: – no guaranteed delivery,

limited retry period (e.g. Tibco RV, SOAP, Webservices)

• Guaranteed delivery: - message will be delivered minimal once (duplication detection required) (e.g. Tibco RV Certified, JMS, WS-ReliableMessaging)

• Transactional integrity: one face commit or with the use of an External TransAction Monitor (TM) even two face commit (e.g. MSMQ with MSTM, Tuxedo)

Each mode has PRO’s and CON’sGeneral rule: More reliability means less throughput

Fault tolerant vs load balanced

• FT means one active (primary), multiple components hot standby (secondary)

• FT often implemented with heartbeats• FT solutions guarantees the order of the

messages• FT is often a fail-over mechanism to archive an

higher uptime.• Load balanced, multiple instances running at

once• Load balanced often requires a load balancer

component, or a scheduler component• Load balancing doesn’t guarantee that the order

of the messages will be kept• Load balanced is often realized by clustering

Message containerMOM transports text based messages which can belong to certain

message types

• Self subscribing (e.g. name value pairs, nested structures)• Form classes (e.g. Market Data updates)• XML (e.g. ebXML, rosettanet, chemXML etc)• EDI• SOAP (envelop for XML messages)• Tibco AE• (X)SQL• Binary Blob's (ASCII representation)• Sonic Propriety message types (e.g. XQ format)• YAML• …Note: many project create a default message header and put the

container as payload. Advanced tracking and tracing, correlation, complex logging and recording are made easier.

Real Time• Real time in principle only possible

for Pub/Sub MOM. • Queue based MOM can be near real

time.• “ the event drive enterprise”• Most processes in the majority of the

companies are batched, and not even near real time

• “Enterprise nervous system”

Remote configuration and management

• Central repositories hold all the MOM configuration• Changes can be made in real time• Deployments and control of the processes are

controlled by administration tools

Tibco: AdministratorFiorano: Admin ToolsSonic: Management Controle

Levels of Security• Message Level Security (e.g. encryption of

message content)• Connection Layer Security (e.g. authentification) • Transport Layer Security (e.g. encryption of

messages on the message bus)• Network Level Security (e.g. firewalls)

Authentication – who?Authorization – what?Integrity – was it changed?Privacy – can anyone see it?

Spectrum of Messaging Architectures

Tightly Coupled

Loosely Coupled

Not Coupled (Decoupled)

Objects Services Events

OO SOA EDA

Intra-App Inter-Application

What are design patterns? (intermezzo)

• Repeatable solution to a software design problem

• Design “template”• Relationships & interactions between

classes and/or objects• Situation dependent, must be

adapted• Not all patterns are “design” patterns

– Architectural pattern– Code pattern

Finding patterns•60+ Patterns

•600+ Pages

•Icon language

http://www.enterpriseintegrationpatterns.com/

Message Bus pattern“What is an architecture that enables

separate applications to work together, but in a decoupled fashion such that applications can be easily added or removed without affecting the others?” (http://integrationpatterns.com/MessageBus.html)

Guaranteed Delivery pattern

“How can the sender make sure that a message will be delivered, even if the messaging system fails?”

“Use Guaranteed Delivery to make messages persistent so that they are not lost even if the messaging system crashes.”

CBR pattern“The Content-Based Router examines the message

content and routes the message onto a different channel based on data contained in the message. The routing can be based on a number of criteria such as existence of fields, specific field values etc.”

Incoming message routed to exactly one destination based on the content of the message

Recipient List pattern

Incoming message routed to one or more destinations based on the content of the message

Benefits of design patterns

• Accelerates the development process• Tested & proven approaches• Familiar to developers• Provides a lexicon, facilitates

communication• Basis for standards and

documentation

SCA (intermezzo)• Service Component Architecture

• OASIS standard

• Note: Fiorano supports SCA

SCASCA Model: Interacting components

SCARecursive composition

SCAModelling the enterprise

SCAModeling application distribution

SCAOperational view

Architecture (Intermezzo)

What is Architecture?• Physical Structure — Components and their physical

arrangement

– Software on machines, machines on networks

– Machines, networks, and minor components form the infrastructure

• Logical Organization — Component roles and responsibilities with respect to the business process(es) the system is intended to support

– Who performs what work, when, and under what conditions?

– Who owns what information, and how is it accessed and distributed?

– Who monitors, manages, and reports on the overall work process?

Debunking Architecture Myths I

• There is no “one size fits all” architecture• Each architecture is specific to the business processes it supports– The architecture of a house doesn’t make a good

factory– The architecture of a banking system won’t run an

airline– A TELCO looks nothing like a $20B Real-estate

financing operation

• Even within the same vertical market there is considerable variation in business processes, and therefore in architecture– Differences in primary business processes provide

competitive differentiation

Debunking Architecture Myths II

• Architectural styles are not architectures• Styles define principles around which architectures can be organized– Strengths and weaknesses can be generalized

• e.g. A three-tier client-server approach is an architectural style, not an architecture– It does not define the actual system components or their

responsibilities

• Technologies are not architectures• Message-oriented middleware (MOM) is not an architecture– “Anything can talk to anything anywhere” is an important

enabling technology, but its unconstrained use leads to bad architectures

• J2EE, CORBA, BPEL, BPMS, BPMn etc. are not architectures

Positioning Architecture in the Project Life Cycle

Agile Architecture Method

RankingUse Case Example: ATM

TransactionsWithdraw Cash

Make Deposit

Transfer Funds

Check Balance

Bank Customer

Ranking by Volumetrics

Peak Rate / Second

Peak Rate Rank

1 or less 1

10 2

100 3

1000 4

10,000 or over 5

Data Size (Bytes)

Data Size Rank

100 or less 1

1K 2

10K 3

100K 4

1MB or over 5

Ranking by Complexity

Source of data

Source of Activity and Process Triggers

Monitoring of Processes

Management of Processes

Complexity Ranking

Users Users Users Users 1

Users and Systems

Users Users Users 2

N/A Users and Systems

Users Users 3

N/A N/A Users and Systems

Users 4

N/A N/A N/A Users and Systems

5

Rating by Business Impact

Risk Category

Minor to business if process is not successfully executed

Business Impact Rank

None No measurable impact 1

Minor Barely observable impact on the departmentalbottom line, no observable impact on the enterprisebottom line

2

Significant Significant impact on a departmental bottom line,barely observable impact on the overall enterprisebottom line.

3

Major The failure of a single execution is survivable, butinability to execute or errors in the process will resultin failure to comply with regulations and/or a majorimpact on the enterprise bottom line

4

Catastrophic A single process execution failure can result inunrecoverable business failure and/or loss of life

5

Overall Business Process Ranking

Use Case Business Process (Scenario)

Frequency Rank

Data Size Rank

Complexity Rank

Risk Rank Overall Rank

Withdraw Cash

ATM 2 2 3 4 48

Make Deposit

ATM 1 2 3 4 24

Withdraw Cash

Make Deposit

Bank Customer

48

24

Monitoring & Correlation I

Component A Component B Component C

= Functional Error

Monitor / Correlation

= Technical Error

= Advisory

= OR

= Begin: Component A Process X, Job y, messageID z

= End: Component A Process X, Job y, messageID z

Asynchronous processes

Correlation on MessageID

= Message Listener

Monitoring & Correlation II

Component A Component B Component C

= Functional Error

Monitor / Correlation

= Technical Error

= Advisory

= OR

Asynchronous processes

Listener on Messages

= Message Listener

Monitoring & Correlation III

Component A Component B Component C

= Functional Error

Monitor / Correlation

= Technical Error

= Advisory

= OR

Asynchronous processes

Listener on Messages&Correlation on Messages

= Message Listener

= Begin: Component A Process X, Job y, messageID z

= End: Component A Process X, Job y, messageID z

Successful transaction A-> C -> A

Monitoring & Correlation IV

Component MessageType Topic ProcessID JobID MessageID Advisory

A X01 Y01 A-S1 01 Uniq001 Begin

A X01 Y01 A-S1 01 Uniq001 End

B X01 Y01 B-R1 01 Uniq001 Begin

B X01 Y01 B-R1 01 Uniq001 End

B X02 Y02 B-S2 01 Uniq001 Begin

B X02 Y02 B-S2 01 Uniq001 End

C X02 Y02 C-R1 01 Uniq001 Begin

C X02 Y02 C-R1 01 Uniq001 End

C X03 Y03 C-S2 01 Uniq001 Begin

C X03 Y03 C-S2 01 Uniq001 End

B X03 Y03 B-R3 01 Uniq001 Begin

B X03 Y03 B-R3 01 Uniq001 End

B X04 Y04 B-S4 01 Uniq001 Begin

B X04 Y04 B-S4 01 Uniq001 End

A X04 Y04 A-R2 01 Uniq001 Begin

A X04 Y04 A-R2 01 Uniq001 End

Successful transaction A-> C -> A

Monitoring & Correlation V

• Operational monitoring• Functional monitoring• Tactical monitoring (e.g. monthly SLA

reporting)• “track and trace errors” (functional

and technical)• Automated corrections possible

SLA reporting• Service level agreements very common

with MOM (and very useful)• Agreement on:

– How many errors / time are allowed– How many messages lost/time– How many messages throughput/time– Reaction time on errors

• Depends on metering tools• Depends for metering data on a monitor

and correlation tool (BAM tool)• Automatic reporting possible

Operational awareness• Technical monitoring - technical exception

management e.g. JMX plug-ins. • Functional (operational) monitoring (e.g. Tibco OPS

factor; very common is to add additional processes into architecture to handle functional exceptions)

• Business process and Activity Monitoring (e.g. correlation with the use of e.g. Tibco BAM, Seebeyond eBAM, Sonic Apama, etc)

• SLA reporting; storing the exceptions and correlation into a DB and run on regular interval reports for the management

• SOA Governmence• Distinction between: technical errors, functional-

technical errors and real functional errors (three state error logic)

Types of testing• Unit Test

– Ensures individual component satisfies its requirements – generally not an architecture issue

• Integration Test– Make sure that the components “play together” properly– Defining a reasonable order-of-assembly is definitely an architecture

issue• Implications for environments must be assessed

• Functional Test– Once the components are assembled, exercise the functionality. Other

than defining an appropriate environment, this is generally not an architecture issue

A Test methodology (e.g. TMAP) should be a critical part of every MOM project

Automated MOM test tools: GreenhatConsulting GHtester, Ikto Lisa, HP Mercury

MOM development IDE• Most MOM products have still propriety IDE

(e.g. TIBCO Designer)• Conversion towards standardised IDE (only

valid for the new versions of the MOM products)

Fiorano(2008Q4): Eclipse plug-inSonic (V7): Eclipse plug-inTIBCO (V6 2009): Eclipse plug-in

Cordys is an exception, which has its IDE based on Ajax/Activex.

Demo Tibco BW• Rv messages explained• Simple BW design• Complex BW design• Create deployment file• Deployment• Test

Tibco BWStart is process (job) starter

(Successful) end is finish of process (job)

BW components are tasks

Each started job is a process (thread) in a JVM

No message queues between the tasks

This example arrival of message is trigger to start job

Zero codingMapping is “drag & drop”

All transformations based on Xpath

Mapping is translated to XSLT (normally hidden under Tibco BW designer)

Compare BW to Fiorano

Fiorano has • concept of services

(compare Tibco tasks, where the whole process might be a service)

• Each services calling a service (black arrows) has an underlying JMS queue or topic (red arrow)

• No concept of jobs, each component/service exists once (if you want load balancing you have to draw a component twice, WYSIWYG)

• Data flow driven architecture• Not always completely following standards (e.g. usage of Tivosi

in stead of pure xml)

Each Process Engine tool from any vendors has pro’s and con’s.

Conceptual knowledge will make it possible to work with multiple vendors! That is the goal for this course!