Jboss Fuse in Vodafone’s Global Integration Gateway 5 th December 2013 / Bruno Meseguer Jboss Architect Meetup
Nov 11, 2014
Jboss Fuse in
Vodafone’s Global
Integration Gateway
5th December 2013 / Bruno Meseguer
Jboss Architect Meetup
Jboss Architect Meetup | 05 December 2013 2
Contents of the presentation
• Who we are within Vodafone
• The performance challenge
• Selection process for the new technology
• Creation of a new framework
• Some final thoughts
Welcome !
Jboss Architect Meetup | 05 December 2013
Global Integration Gateway (GIG) Vodafone Group Enablers
Jboss Architect Meetup | 05 December 2013 4
Vodafone Group Enablers
• Enablers team, responsible to enable capabilities in Vodafone Group
– Charging
– Messaging
– Customer Registry
GIG
– … and more
• GIG (Global Integration Gateway) (100+ systems)
– Intelligent Gateway which allows flexible Partners/Vodafone integration.
– Local/Global Partners and Vodafone internal systems can access services provided by OpCo’s.
– Opco’s can access services provided by Partners and Vodafone internal systems.
Who we are and what we do
Jboss Architect Meetup | 05 December 2013 5
Vodafone’s Global Integration Gateway (GIG)
HTTP HTTP
Technical capabilities view
Authentication Authorisation
Protocol Bridging
Orchestration Throttling
Routing
Header Enrichment
Error Handling
Data Transformation
Logging & Reporting
Asynchronous
Very fast > 800 TPS
Jboss Architect Meetup | 05 December 2013 6
Vodafone’s Global Integration Gateway (GIG)
HTTP HTTP
Bridge In
(north)
IBM DP
Bridge Out
(south)
IBM DP
Orchestration
(Tibco BE)
Atomic
(Tibco BE)
Atomic
(Tibco BE)
Orchestration
Orchestration
Orchestration
Atomic
Atomic
Atomic
System layers view
Jboss Architect Meetup | 05 December 2013 7
Vodafone Group Enablers
• In GIG’s ecosystem there are currently 100+ systems
• GIG’s current traffic is 1000+ TPS
• GIG’s original design aimed to be capable of handling 10,000 TPS
– This principle has been maintained on every evolutionary step of the solution
– GIG scales horizontally: 10,000 doesn’t represent a barrier
• The latency that GIG introduces is <0.1 seconds on slowest use cases
• The average payload size is around 2KB
– As GIG typically exposes API’s, payloads are not significantly big.
Some GIG numbers
Jboss Architect Meetup | 05 December 2013 8
Vodafone’s Global Integration Gateway (GIG)
• 2008:
– Poor HTTP performance from TIBCO component
– Had to find a fast HTTP gateway
– IBM's DataPower was the answer > 800 TPS
• 2012:
– Vision to port systems to a cloud environment
– IBM's hardware not portable
– the quest to find a replacement for IBM's DataPower starts
A little bit of history
Jboss Architect Meetup | 05 December 2013 9
Vodafone’s Global Integration Gateway (GIG)
• To find a software based solution, current DataPowers not portable to the cloud.
– With very high levels of performance, ideally > 800 TPS
– Fully asynchronous socket handling
– EMS compatible
• Minimise impact on clients.
– Able to maintain the same port definitions as in DataPower
• Minimise migration efforts
– Quick adoption of the technology
– 70+ flows to be migrated
The goals to replace IBM’s DataPowers
Jboss Architect Meetup | 05 December 2013
The performance
challenge
Jboss Architect Meetup | 05 December 2013 11
The performance challenge
• A normal HTTP server retrieves resources or generates some content and quickly responds to the
client
• What happens when a system acts as a pass-through and needs to wait for backend responses?
• delays from the backend could be disastrous to the system
Jboss Architect Meetup | 05 December 2013 12
The performance challenge
• The worker thread-pool gets exhausted, where are the workers?
Quickly running out of workers!
Jboss Architect Meetup | 05 December 2013 13
The performance challenge
• Worker threads need to wait for the backend response to resume.
Threads busy… doing nothing!
Jboss Architect Meetup | 05 December 2013 14
The performance challenge
• A waiting queue of requests rapidly grows
Requests keep coming!
Jboss Architect Meetup | 05 December 2013 15
The performance challenge
• Clients get increasingly angry!
Unhappiness!
Jboss Architect Meetup | 05 December 2013 16
The performance challenge
• The system is exposed:
– Delays from the backend could be disastrous to the system
– The workers thread pool gets exhausted
– Worker threads need to wait for the backend response to resume
– A waiting queue of requests rapidly grows
• Systems will need to be proved to be robust:
– Under high rates of client requests
– Possible delays from back-ends
• Systems need to be Asynchronous
Jboss Architect Meetup | 05 December 2013
Selection Process
Jboss Architect Meetup | 05 December 2013 18
Selection Process
• Direct competitor to DataPower
• Highly recommended by Group Architecture
– Pre-selected by Group Architecture to replace DataPower
• Can be used as a Hardware or Software based solution
• A promise of raw performance, up to 30000 TPS
Layer7… first candidate
Jboss Architect Meetup | 05 December 2013 19
Selection Process
• How can we stress a system capable of 30000 TPS without the need of a farm of HTTP clients?
• There is an old friend to us called Netty used back in 2008 (a beta library back then) to test
DataPower on the lab.
• Using Netty we created a powerful client and stub proved to handle 5000+ TPS per instance
• Netty is an NIO client/server framework (event driven) with HTTP capability:
– Non-blocking socket management
– threads are freed up when posting requests, they don't wait for responses
– difficult to exhaust the thread pool
– responses are taken care of by a second thread pool
– the socket and HTTP context are kept in memory awaiting for a response thread to pick them up and close
Lab preparation
Jboss Architect Meetup | 05 December 2013 20
Selection Process
Lab preparation
• To stress the bridge we bombard it with a constant TPS rate and delay the response by 5 seconds.
Jboss Architect Meetup | 05 December 2013 21
Selection Process
• Within the space of two weeks we realised Layer7 was not the right choice
– By the third day of testing we aborted the performance study for Layer7
• Layer7 proved not to fulfil our performance requirements
– delays at the backend proved to be deadly to Layer7
– it was not asynchronous
Layer7 results
Jboss Architect Meetup | 05 December 2013 22
Selection Process
• As Netty proved to be very efficient we approached Red Hat and asked to provide us with Netty
support
• Initially Red Hat recommends Fuse + Jetty (Netty equivalent)
– 1st round of performance test show the solution works (400+ TPS), and it is indeed asynchronous
• From our Netty benchmarks we knew there was more potentially to exploit
• Red Hat agrees to integrate Netty HTTP into their product (it already included the Netty TCP
component)
– Red Hat delivers a Netty HTTP concept for us to try out
– 2nd round of performance show an improvement by a factor of 2 at least
Red Hat comes into play
Jboss Architect Meetup | 05 December 2013 23
Selection Process
• We triggered parallel assessments for:
– JBoss7
– TIBCO BusinessEvents 5.0
– WSO2
More candidates…
Jboss Architect Meetup | 05 December 2013 24
Selection Process
Performance results
Jboss Architect Meetup | 05 December 2013 25
Selection Process
Performance results
Jboss Architect Meetup | 05 December 2013 26
Selection Process
• Although only one product could get selected, all of them had an opportunity to identify
weaknesses in their product
• They all immediately put engineers at work to focus around those areas to be improved
• They all managed to boost up their engines:
– some more than others (x2, x3, x4)
– some adopted the asynchronous mechanisms (x100)
Everyone wins
Jboss Architect Meetup | 05 December 2013 27
Selection Process
• Fastest NIO product available (Netty enabled)
• Integration product (Camel)
• OSGI container
• Fully covering our migration requirements
• Java based solution
• Red Hat’s engineers: main open source committers
• Red Hat quality assurance
Red Hat’s JBOSS FUSE gets selected
Jboss Architect Meetup | 05 December 2013 28
Selection Process
• It was very critical to make to right choice
• We invested the right amount of time on each one of the steps
• We know we got it RIGHT!
No rushing !
Jboss Architect Meetup | 05 December 2013
Vodafone’s Framework
Jboss Architect Meetup | 05 December 2013 30
Vodafone’s Framework
• Jboss Fuse offering an opportunity to improve DEV, Deployment & Support processes
– Traditionally our developments are based on frameworks
– Upgrades on frameworks require:
from DEV a new build (per component)
and from D&S another testing and deployment process
… per component !
Jboss Fuse’s potential
Jboss Architect Meetup | 05 December 2013 31
Vodafone’s Framework
Traditional build and deployment processes
Customisation
Vodafone’s
Framework
Build Customisation
Vodafone’s
Framework
Deployable Unit
Deployable Unit
Source Code
Vodafone’s
Framework
Customisation Customisation
Customisation
JVM
JVM
Jboss Architect Meetup | 05 December 2013 32
Vodafone’s Framework
• JBoss Fuse container is OSGI based
– It encourages to think differently
– Perhaps Vodafone can exploit the benefits
Could we change our processes to reduce efforts?
Could these benefits help the various involved teams
Development
Deployment
Support
OSGI and bundles
Jboss Architect Meetup | 05 December 2013 33
Vodafone’s Framework
New processes: OSGI orientated
Build
Vodafone’s
Framework
Deployable Unit
Customisation
Deployable Unit
Source Code
Vodafone’s
Framework
Customisation Customisation
Customisation
Customisation
Deployable Unit
JVM
Customisation
Customisation
Vodafone’s
Framework
Jboss Architect Meetup | 05 December 2013 34
Vodafone’s Framework
• The Framework is decoupled from business components and sits on the OSGI container as a
reusable module
• Implemented using Camel route definitions
• It contains 90% of the execution logic of a flow
• The Framework is responsible for
– Standarising flow definitions
– Reduce complexity in customisations
– Include all common functionality
– Ensure stability, robustness and performance
Framework's main bundle
Jboss Architect Meetup | 05 December 2013 35
Vodafone’s Framework
Framework's main bundle
common common common
Direct-VM
JMS
common
Framework 1.0.0
1. Accept internal
invocations.
2. Throtteling.
3. Header
Enrichment.
4. System Logging.
5. Collect Reporting
Information.
6. Bridge to JMS.
Jboss Architect Meetup | 05 December 2013 36
Vodafone’s Framework
• Business components (called Connectors) are configured to communicate with the framework
• Implemented using Camel route definitions
• Connectors are responsible for
– Listening to incoming requests
– Concentrate on the business logic
– Delegate execution control to the Framework
Connector’s bundle
Jboss Architect Meetup | 05 December 2013 37
Vodafone’s Framework
Connector’s bundle
HTTP
custom
Direct-VM
out
Flow A
1. Open an HTTP
port.
2. Custom manipulation, typically
extraction of the Service ID, MSISDN,
and country code.
3. Invoke the Framework
route definition.
Jboss Architect Meetup | 05 December 2013 38
Vodafone’s Framework
• Connectors are configured to communicate with the framework
• N connectors talk to the same single Framework module within the same OSGI container
• Upgrades on the framework do not require new deliveries from Development and do not require
redeployments
Framework and connectors interaction
Jboss Architect Meetup | 05 December 2013 39
Vodafone’s Framework
Framework and connectors interaction
Jboss Architect Meetup | 05 December 2013 40
Vodafone’s Framework
• Upgrades on the Framework do not cause an impact on Connectors
• When a Framework fix is delivered, Connectors are simple reconfigured
• Multiple Framework versions can coexist if necessary
Framework upgrades
Jboss Architect Meetup | 05 December 2013 41
Vodafone’s Framework
Framework upgrades
common common common
Direct-VM
in
JMS
common
Framework 1.0.0
custom
HTTP
Direct-VM
out
HTTP
custom
Direct-VM
out
common common common
Direct-VM
in
JMS
common
Framework 2.0.0
Flow A
Flow B
Jboss Architect Meetup | 05 December 2013 42
Vodafone’s Framework
• An early performance round shows a degradation (when
compared to POC results) well within the acceptable
margin
– around 10-15% loss only to the raw potential available
– with 5 seconds delay at the back and 2K payload we obtain 700
TPS
Note: The Lab environment used is SLOW !
We think the Lab environment is around half the speed of a
modern environment.
• An official data gathering is pending to be collected
Framework's performance
0
100
200
300
400
500
600
700
800
PoC Framework
TPS
Jboss Architect Meetup | 05 December 2013
Final overview
Jboss Architect Meetup | 05 December 2013 44
Current status of the project
• Vodafone’s Framework
– Implemented internally at
– Nearly 100% of the functionality completed
• Migration of DataPower flows
– The migration is outsourced to our trusted vendor Bulgaria
– Sprint 6 out of 8 is on its way
– 70+ flows in total
• Deployment
– First migrated flows started to be deployed in the pre-production environment
Jboss Architect Meetup | 05 December 2013 45
How JBoss Fuse compares to DataPower?
• Of course, JBoss Fuse is software based instead of Hardware based
– this means it is “cloudable”
– this means we are free to choose the hardware to run on
– it means the hardware is much cheaper (this is specially important when scaling)
– it means everyone can run it on its laptop
• Performance results suggest a substantial gain in overall TPS rate (1000+ TPS per instance)
• We can exploit the potential of 64 bits environments
– DataPower runs on 32 bits unless money is invested to acquire the new 64 DataPower XI52s
• The community of open source based solutions is much bigger
Jboss Architect Meetup | 05 December 2013 46
Are there any other advantages?
• it is Java based
– the skills required are much more common
• It is developer friendly
– DataPower development is painful
• One single Framework for all (instead of several in DataPower)
• RED HAT Customer support represents a quality leap coming from IBM's
– fast responses to customer cases
– skilled agents at the other end of the line
– proactive and collaborative reaction
– open to customer's point of view
– a user friendly Customer support Portal !!
Absolutely !!
Jboss Architect Meetup | 05 December 2013 47
What about the disadvantages?
• It's new
– we don't have OSGI experience
– we don't know yet the pitfalls
• New Component
– Red Hat has included Netty HTTP to satisfy our requirements
– we may find some integration defects
• New Framework
– will need some time to mature
– surely problems will be found
– migrated components may differ in behaviour from DataPower
Jboss Architect Meetup | 05 December 2013 48
The good and the bad
• The good
– Plenty has been said on previous slides
– The raw performance of the Camel framework and the Netty HTTP component
– The incredible number of available alternatives on how to do a particular task
– Very active product, with constant addition of features
– Customer support is first class
• The bad
– Not intuitive with the mix of available frameworks
– The available IDE to graphically define flows is not mature enough
– Having many options to do one task is seen as a positive, but in some cases is a negative
– Steep learning curve for less experienced developers
Jboss Architect Meetup | 05 December 2013 49
Questions and answers
Thank you
Jboss Architect Meetup | 5th December 2013