From Software Selection to Mission-Critical Deployment · ESB Evolution •Enterprise Service Bus concept: David Chapelle, Progress Software •Mule ESB: Ross Mason, real-world implementation

ESB HOWTOFrom Software Selection

toMission-Critical Deployment

Eugene CiuranaDirector, Systems Infrastructure

LeapFrog Enterprises, USA

About Eugene Ciurana

• 15+ years of experience building mission-critical, high-availability systems infrastructure

• 10+ years of Java work

• Engaged by the largest companies in the world

• Retail

• Finance

• Oil industry

• Background ranges from industrial robotics to on-line retail systems

Do You Have an Integration Problem?• If you are in a mid-sized to large enterprise, you have many

applications that must inter-operate

• Custom

• Third-party

• Legacy

• Integration is point-to-point

• Applications communicate over dedicated links

• File exchange, database, API calls, RPC

• Hard and expensive to respond quickly to business demand

Do You Have an Integration Problem?• Multiple tiers

• Each tier is implemented with different technology

• Coexistence of legacy and new systems

• Different skills for each tier

• Multiple physical locations

• Multiple protocols?

• Ad hoc interfaces between systems?

• Data and process consolidation

• Point-to-point integration is a huge problem!

Point-to-Point Integration Problems

• Every problem becomes a new project

• Different systems may support different interoperability protocols

• JMS

• SOAP

• REST

• BPEL

• etc.

SOAP

• Creates vendor lock-in or artificial dependencies between systems

Point-to-Point Integration Problems• Point-to-Point

integration is more expensive

• Requires more regression testing

• Consumes more resources

JMS

JMS

JMS

FTP

SOAP

REST

Application

A

Application

B

ApplicationC

Application

D

Application

E

Application

F

Integration Through Services• SOA = Services-Oriented Architecture

• Collection of services that communicate with one another

• No dependencies on other services

• Self-contained

• Messaging: mechanism for communication between two or more services

• Real-time, asynchronous, synchronous

• May occur over different transports

• HTTP, FTP, JMS, RMI, CORBA, etc.

Integration Through Services

• Web services are the current mechanism for data exchange between applications

• Shorthand: web services are a tangible implementation of SOA concepts

• Some people think that SOAP == web services

• That’s a very limited view of the universe

• Web services may be implemented in a variety of ways:

• HTTP: SOAP, REST

• JMS: synchronous and asynchronous queueing

Can you think of others?

ESB Evolution• First... came the queueing architectures

• Message queueing (MQ, Sonic MQ, JMS) was not enough

• Service consumers and providers must implement it

• Expensive to maintain two different interoperability layers for the same service (e.g. JMS, REST)

• Messaging buses evolved to accommodate other protocols

• Different approaches by different vendors/dev groups

• ESB ::= Enterprise Service Bus

• Queuing + messaging + support for multiple protocols

ESB Evolution• Enterprise Service Bus concept: David Chapelle, Progress

Software

• Mule ESB: Ross Mason, real-world implementation in parallel

• Open-source, battle-tested... it works!

• Commercial ESBs

• Open-source ESBs

• Commercial and open-source ESBs all claim to do the same thing... Mule is often selected because it delivers on the promises made by commercial vendors and is more mature and robust than other open-source offerings

Use Commercial or Open-Source?

• Motivation: commercial vendors want to lock you in

• Their ESBs work great with other products in their catalogue, not so great or not at all with third-party software

• Motivation: open-source seamless integration

• Many are immature or marred by philosophical baggage

• This presentation is about Mule ESB: open-source, great support organization behind it, active volunteer community, and an enterprise track record

Geeking Out... Let’s Do ESB!

• What you need to know:

• Java programming (intermediate to advanced)

• Basic concepts about messaging

• Web services, HTTP, or JMS

• “Enterprise Integration Patterns”

• Gregor Hohpe, Bobby Wolf, et. al.

• What you’ll need:

• Java 5

• Mule 1.4.1 or later

Getting Started• Get Mule:

• http://mule.codehaus.org

• tar xzf mule-1.4.2.tar.gz

• ln -s ./mule-1.4.2 mule (sym links are useful)

• Environment:

• export MULE_HOME=~/development/mule

• export PATH=“$PATH:$MULE_HOME/bin”

• export JAVA_HOME=/usr/local/java

• Create a work directory and go there:

• mkdir myproject; cd myproject

You’re readyto

Mule!

Sample Project• Need to create an OpenLaszlo component from a content

management system

• Day Software’s Communiqué (CQ)

• Building OpenLaszlo integration into CQ is expensive and time-consuming

• Time critical: develop an OpenLaszlo compiler module and integrate this with CQ using Mule

• Total development and integration time: 3 days

• Components are fully reusable!

• Implemented as services

Architecture Overview

Mule ESB

CMS AuthoringRIA

ComponentEngine

Customer

Master

Single Sign-On System

Active

Directory

CMS Publishing

E-Commerce Site

HostedMicro-site

Internet

Mule Environment• Mule relies on $MULE_HOME to find its libraries

• Three very important directories under Mule:

• lib/mule - core Mule libraries

• lib/opt - third-party libraries such as Axis, Apache Commons, dom4j, log4j, Jetty, JUnit, etc. - almost everything you need is here!

• lib/user - this is where you’ll store your .jar(s) with your own components

• OpenLaszlo doesn’t ship with Mule, so in lib/opt:• lrwxr-xr-x lps-4.0.3.jar -> /opt/LPS/lps-4.0.3/Server/WEB-INF/lib/lps-4.0.3.jar

Mule In Action• Applications are at the endpoints

• Configuration describes endpoints, channels, connectors, transformers, and routers

• Service objects provide functionality to “tie” things together

• Written by Mule, third-parties, or your dev team

Application Channel

Transformers

Message

Sender

Connector

Inbound

Router Service

ObjectComponent

Outbound

Router

Transformers

Message

Receiver

Connector

Channel Application

Mule

SWFRendereringEngine

Defining Models

Communiquésource files

list

FileRetrieverComponent

SourceFiles

Repository

File system

SWFCompiler

BuiltFiles

Repository

SWFRendereringEngine

FileRetrieverComponent

SWFCompiler

SourceFiles

Repository

BuiltFiles

Repository

File system

Communiquésource files

list

Defining Models<model name=“SWFRenderingEngine”>

<endpoint name=“LPSDrop” value=“http://localhost/lpsdrop">

<mule-descriptor name= “FileRetrievalService” implementation=“com.pkg.FileRetriever”>

Complete file at:http://ciurana.eu/JavaZone2007/swf-engine.xml

A Mule model maps to a composite of components andendpoints.

Defining Endpoints• Endpoints are the interfaces between the Mule and the

applications

• They are abstractions of the underlying messaging system

• An endpoint can abstract any network transport, a database, memory channels, a file system, etc.

<endpoint-identifiers> <endpoint-identifier name="LPSDrop" value=“http://localhost:8016/lpsdrop” /> <endpoint-identifier name="SourceFiles" value="file://./sources" /> <endpoint-identifier name="CompiledFiles" value="file://pubserver/triggers" /> <endpoint-identifier name="ServiceMonitor" value="stream://System.out" /></endpoint-identifiers>

Transformers

• Data may come through an inbound endpoint in a format different than the one expected by the outbound endpoint or the service object

• You can transform the data as it comes into the component or after it leaves

• You may apply transformations to the message itself, the message envelope, or both

• Example: lpsdrop listens for a method POST with a payload in the form: fileCount=0&file1=/path/to/whatever.gif&... - FileRetriever expects a HashMap with these data

Transformers

public class POSTToHashMap extends AbstractTransformer { // Private members and methods . .

// Public methods public Object transform(Object anObject) { HashMap payload = null; if (this.isValidPOST(anObject)) payload = this.extractDataFrom(anObject);

return payload; } // transform}

fileCount=2&file1=x.gif&file2=prog.lzx

Various operations to convert the POST datato a HashMap.

http://ciurana.eu/JavaZone2007/POSTToHashMap.java

+Object transform(Object source);

+String toString();

AbstractTransformer

+Object transform(Object source);

POSTToHashMap

Transformers public Object transform(Object anObject);

Mule transformers resolve these calls through configuration. As a developer/implementer, you must know the return type expected by the component and specify it as part of the configuration.

<transformers> <transformer name="POSTToHashMap" className="com.leapfrog.transformers.POSTToHashMap" returnClass="java.util.HashMap" /></transformers>

Notice that you must specify the return type for the transformation.

Handling Application Messages• You may extend an application’s capabilities through the use

of a service component

• Called Universal Message Objects or UMOs in older versions of Mule - the documentation still refers to them this way

• A service component is a POJO/bean that runs under Mule

• It executes business logic on incoming messages from any destination

• Messages can be dispatched to another endpoint

• There is no Mule-specific code in them

• Configure the bean using the Mule XML file

Implementing Business LogicSWFRendereringEngine

FileRetrieverComponent

SWFCompiler

SourceFiles

Repository

BuiltFiles

Repository

File system

Communiquésource files

list

SWFCompiler

Implementing Business Logic

<<component>>

Build

Files

SWFBuilder<< component>>

OpenLaszlo +setArgList()

+run()

+status()

<<mixin>>

Compiler

<<component>>

Source

Files

Mule

FileConnector

Implementing Business Logicpublic class SWFCompiler extends Service { . . public Object service(UMOEventContext eventContext) { String sourceFileName = this.sourceDir+"/"+sourceFile; byte[] byLaszloSource; if (this.validateEnvironment()) try { byLaszloSource = eventContext.getMessageAsBytes(); this.dumpBytesToFile(byLaszloSource, sourceFileName); this.compile(sourceFile, objectFile); this.packageSWF(); } catch (Exception e) { log.error(e.toString()); } else log.error("Invalid compiler environment.");

return this.fileSWFAsBytes(); } // service

http://ciurana.eu/JavaZone2007/SWFCompiler.java

Implementing Business Logicpublic class SWFCompiler extends Service { . . public Object service(UMOEventContext eventContext) { String sourceFileName = this.sourceDir+"/"+sourceFile; byte[] byLaszloSource; if (this.validateEnvironment()) try { byLaszloSource = eventContext.getMessageAsBytes(); this.dumpBytesToFile(byLaszloSource, sourceFileName); this.compile(sourceFile, objectFile); this.packageSWF(); } catch (Exception e) { log.error(e.toString()); } else log.error("Invalid compiler environment.");

return this.fileSWFAsBytes(); } // service

http://ciurana.eu/JavaZone2007/SWFCompiler.java

This is just a POJO - Service is a superclass that we defined

Message can be Object, String, etc. UMOEventContext always works

Application business logic - do

what you need.

Return an object... send this to an endpoint?

public class SWFCompiler extends Service {

public Object service(UMOEventContext eventContext) {

byLaszloSource = eventContext.getMessageAsBytes(); this.dumpBytesToFile(byLaszloSource, sourceFileName); this.compile(sourceFile, objectFile); this.packageSWF();

return this.fileSWFAsBytes();

<mule-descriptor name="SWFCompilationService" implementation="com.leapfrog.services.SWFCompiler"> <inbound-router> <endpoint address="SourceFiles"> <filter pattern="source_file.lzx" className="org.mule.providers.file.filters.FilenameWildcardFilter"/> </endpoint> </inbound-router>

<properties> <property name="sourceDir" value="./sources" /> <property name="objectDir" value="./triggers" /> <property name="objectFile" value="animation.swf" /> <property name="cacheDir" value="./cache" /> <property name="LPSHome" value="/Users/eciurana/ESB/lps-4.0.3" /> </properties></mule-descriptor>

Implementing Business Logic

private String sourceDir; // in SWFCompiler

public void setSourceDir(String sourceDir) { this.sourceDir = sourceDir;} // setSourceDir

Deploying Mule

• Your components are ready and you configured the models, endpoints, routers, and transformers - what’s next?

• Deploy Mule embedded in another application

• Saves you from writing message, transport code

• Deploy Mule with another framework like Spring

• It becomes another pluggable component

• Deploy Mule stand-alone in a high-performance dedicated deployment

Deploying Mule• The applications are deployed as .jar files to the server

• Satisfy all the package dependenciesSWF Rendering Engine

+SWFEngine-OpenLaszlo

Compilerlzc Main

-MuleCore

VMConnectorFileConnector

SWFBuilder

conf/swf-engine.xml

# OpenLaszloopt/lps-4.0.3.jar

mule/mule-core.jar

user/swfengine.jar

Deploying Mule• Stand-alone server deployment

• Select a fast server, multiple-cores, and lots of memory

• Mule needs to move messages through quickly

• Gigabit connections if set up behind the firewall

• Fail-over: two Mule instances behind a front-end load balancer or router

• Use in-memory caching or JavaSpaces to share state

• Only one instance at a time is active

• Warm server will establish endpoint connections on-demand

Production Deployment

• How many Mule instances in a server?

• Deploy a single instance with large config file?

• Deploy multiple instances, each for a subsystem?

• Multiple JVMs?

• Multiple virtual servers?

• Xen, VMWare

• Easier administration

Mule

Server

Production DeploymentSWFRenderer

jar

mulecore

jar

config.xml

CMS server

<uses>

author

Mule Server

Mule Server

Mule Server

lazlo-runtime

jar

Production Deployment

Complete Sample Architecture

Firewall

External

Device

PC

application

Web

browser thesite.comhttp

Cart,

session

data

Dedicated StoreJava 6, Wicket, Tomcat,

etc.

SOAP REST

SOAP

REST

SOAP REST

Mule ESB

SOAP, REST, JMS, MQ, BPEL, JDBC, caching, in-memory

SOAP REST

REST

Microsite

Java 6

Wicket

CMS

feed

httpInternet

SOAP

REST

SOAP

REST

ERP

SOAP

REST

CMS ContentRepository

JDBC

Customer

Master

SOAP

REST

Crowd Single Sign-On

Active Directory(domain)

Custom

Merchandizing

Tools

SOAP

REST

REST

CRM

Mule ESB

Conclusions• You probably need an ESB for application integration if your

company is from mid- to enterprise-size

• SOA: avoid marketectures, define a sensible architecture that combines open-source and commercial products, leveraging your domain-specific applications as services

• Using Mule open-source ESB

• Easy configuration, integration, expansion and customization

• Deploying in production: don’t put all the eggs in one basket

• Application monitoring: custom and commercial support

Questions&

Answers

Eugene CiuranaDirector, Systems Infrastructure

LeapFrog Enterprises, [email protected]

pr3d4t0r - irc.freenode.net#esb, ##java, #awk

This presentation is hosted athttp://ciurana.eu/JavaZone2007