Using MRG and Infinispan for - JBoss Developer · 2019-02-22 · Using MRG and Infinispan for Large Scale Integration Prajod Vettiyattil 2. What this session is about ... file Adapte

Using MRG and Infinispanfor

Large Scale Integration

Prajod Vettiyattil

2

What this session is about

Challenges in Large Scaleintegration

Use cases for Large Scaleintegration

How to solve the challenges Solution Products to implement the

solution The Open Source difference

3

Challenges

Use Cases

Solutions

Key Phrases

4

Phrases: 1

• Large Scale Integration– Integration of 10s or 100s of systems, and

exchange GBs of messages in a day• Big Data

– A changing threshold– Data in the Terabytes, Petabytes, Exabytes…

• Asynchronous Messaging– Message oriented middleware

• Real time systems– Systems that are built to respond to requests in

real time, with predicable, consistent responsetimes

5

Phrases: 2

• Grid– A set of interconnected computers that work in

parallel to solve a computing problem• Cloud Computing

– Computing as a service– Client of the cloud is isolated from the details of

the implementation of the service

6

7

ChallengesChallenges

Use Cases

Solutions

The Key ChallengesLarge number of systems

8

The Key ChallengesComplexity of connection between these systems

9

The Key ChallengesConstraints on the systems and on the connections

10

soap /http

csv /ftp

Rest /http

csv /f ile

Adapter

Adapter

soap/ http

csv/ ftp

Rest/ http

csv/ file

Adapter

Rest /http

csv /f ile

Adapter

Text/tcp

soap /http

csv /ftp

soap /http

csv /ftp

Rest /http

csv /f ile

Adapter

csv/ file

Adapter

soap/ http

csv/ ftp

Rest/ http

csv/ file

Adapter

csv/ file

Adapter

Adapter

11

Use Cases and SolutionsChallenges

Use Cases

Solutions

ArchitectureWireless Telco BSS Integration

BSSApplications

Mediation Provisioning CRM WorkforceManagement

NumberInventory

Management

Interconnect

InfrastructureServices forMiddleware

TransportServices

CachingServices

Load BalancingServices

RecoveryServices

FailoverServices

ElementManagement

FaultManagement

RevenueManagement

Process Automation/ Business Process ManagementAdm

inistration and Monitoring Services

Security Services

Portals, Front End System, Partner Gateways

Telco Network Systems

Middleware Message Broker Enterprise ServiceBus

JEE Server

Billing

12

Telco scalabilitySome requirements

• 75+ million customers• Plan for Terabytes of CDRs and other

messages per day• Performance is critical to customer

experience and retention• CRM, Billing, Mediation, Middleware, Data

warehouse

13

High volume use case 1CRM to Billing Integration

MiddlewareMediationNetworkSwitch Billing

DataWarehouse

FraudManagement CRM

Other BSSApps

14

Point to Point Connection

Technical Requirements•Memory•Threads•Sockets•Sender performance•Receiver performance

High volume use case 1CRM to Billing Integration:

with Middleware Infrastructure

MiddlewareMediationNetworkSwitch Billing

DataWarehouse

FraudManagement

MiddlewareInfrastructure

• Caching• Load Balancing• Failover• Recovery

CRM

Other BSSApps

15

MiddlewareMediation

NetworkSwitch Billing

DataWarehouse

FraudManagement CRM

Other BSSApps

Middleware Infrastructure expanded

Mid d leware

Mediatio

n

Netwo rk

Switch

Billin g

DataWareh o u

se

Frau dMan ag em

en t

Mid d leware

In frastru ctu re

CRM

Oth erBSSAp p s

Middleware

Middleware Infrastructure

High Speed,Reliable

Massaging

ComputeNode Scaling

ResourceManagement

Work LoadManagement

Failover RecoveryDistributedCaching Cluster Toolkit

16

Middleware InfrastructureProducts

Mid d leware

Mediatio

n

Netwo rk

Switch

Billin g

DataWareh o u

se

Frau dMan ag em

en t

Mid d leware

In frastru ctu re

CRM

Oth erBSSAp p s Middleware

Middleware InfrastructureHigh Speed,

ReliableMassaging

ComputeNode Scaling

ResourceManagement

Work LoadManagement

Failover RecoveryDistributedCaching Cluster Toolkit

• Any ESB• Any JEE Server• Any Message Broker

MRG Messaging

MRG Grid

Infinispan

1

2 3 4 5

6

8

7

17

Middleware Infrastructure: ProductsMRG Messaging

Mid d leware

Mediatio

n

Netwo rk

Switch

Billin g

DataWareh o u

se

Frau dMan ag em

en t

Mid d leware

In frastru ctu re

CRM

Oth erBSSAp p s

Middleware Infrastructure

High Speed,Reliable

Massaging

Compute NodeScaling

ResourceManagement

Work LoadManagement

Failover RecoveryDistributedCaching

ClusterToolkit

MRG Messaging• AMQP support• Native RDMA,

Infiniband• Can use MRG Realtime• Large message

support(> GB)• Clustering and Failover• High speed, journal

based persistence• Java and C++ brokers• Based on Apache Qpid

18

Middleware Infrastructure: ProductsMRG Grid

Mid d leware

Mediatio

n

Netwo rk

Switch

Billin g

DataWareh o u

se

Frau dMan ag em

en t

Mid d leware

In frastru ctu re

CRM

Oth erBSSAp p s

Middleware Infrastructure

High Speed,Reliable

Massaging

ComputeNode

Scaling

ResourceManagement

Work LoadManagement

Failover RecoveryDistributedCaching Cluster Toolkit

MRG Grid• Scalable Grid Scheduler• Resource variety:

Desktop to Cloudschedulers

• Low latency results:Using MRG Messaging

• Dynamic provisioning• High Availability• Grid Federation• Is based on the Condor

Grid project

19

Middleware Infrastructure: ProductsInfinispan

Mid d leware

Mediatio

n

Netwo rk

Switch

Billin g

DataWareh o u

se

Frau dMan ag em

en t

Mid d leware

In frastru ctu re

CRM

Oth erBSSAp p s

Middleware Infrastructure

High Speed,Reliable

Massaging

Compute NodeScaling

ResourceManagement

Work LoadManagement

Failover RecoveryDistributedCaching

ClusterToolkit

Infinispan• In memory Data Grid• Distributed cache• Peer to peer

communicationbetween nodes

• Flexible persistence:JDBC, File, Amazon S3

• Map reduce: node localcomputing

• Implementation forperformance

20

High volume use case 2Post Trade Securities Processing

Processing Nodes

Aggregator

Node1 Node2

Node3 Node4

File Splitter+

LoadAllocator

Post TradeFiles

TradingApplications

TradingApplications

AccountingSolution

RiskManagement

Solution

Sources TargetsData Processing Solution

OutputChannels

21

Data Services

Ref Data Solution Customer MasterSrc 3 Src 4

Post Trade Securities ProcessingTechnical Requirements

22

Processing Nodes

Aggregator

Node1 Node2

Node3 Node4

FileSplitter +

LoadAllocator

InputChannels

OutputChannels

Data ServicesTechnical Requirements•File Streaming•Multiple Data views, Data sources•Data Aggregation•Reliable Messaging

Post Trade Securities ProcessingProducts

Processing Nodes

Aggregator

Node1 Node2

Node3 Node4

File Splitter+

LoadAllocator

MRG Messaging

MRG Grid

Infinispan

1

5

3

2

4

InputChannels

OutputChannels

6

7

23

Data Services JBoss Data Services

MRG Realtime• Consistent, predictable response• Websphere Realtime: RTSJ

24

Messages/microsecond

Res

pons

e Ti

me

Source:Red Hat

A Recap of the Solution

25

Challenges

Use Cases

Solutions

Challenges, Solutions and ProductsChallenge Solution Products

1 Small data elements,high volume

Distribution, load balancing andpartitioning

MRG Messaging, MRG Grid

2 Large data elements File splitting, distribution, inplace processing

MRG Messaging, MRG Grid

3 Data views, Many datasources

Data Services JBoss Data Services

3 Predictability Real time kernels, real timeJVMs

MRG Realtime, RTSJ

4 Availability Load balancing, clustering,failover

MRG Grid, Infinispan

5 Reliability File based caches, DBpersistence

MRG Messaging, Grid,Infinispan

6 Scalability Compute grids, Data grids,Asynchronous messaging

MRG Messaging, Grid,Infinispan

26

Solution Alternatives

27

The Map Reduce method• Split data, process in parallel, aggregate results

SplitData

Map Phase Reduce Phase

Task Tracker Task Tracker

Task Tracker Task Tracker

OutputData

Job Tracker

Data

NameNode

28

Client

InputData

GT3 and Condor

• Globus Toolkit– Open source toolkit for Compute Grids– Architecture, Service Model and Implementation– Job Tracking, Management, Monitoring, Resource

Management– Data Management: Movement, Location Registry

• Condor– Grid Framework from University of Wisconsin– Compute Node Scaling– Job Scheduling– Idle time utilization

29

Commercial Tools• IBM

– IBM Cloudburst– Websphere Virtual Enterprise– Websphere Realtime

• Oracle– Oracle Exalogic– Oracle Coherence– Oracle Grid Engine(Sun Grid Engine)

• Terracotta– Quartz– Big Memory

30

The Open Source difference

31

The Advantages

• Smaller adoption steps to reduce risk• Flexible Cost Model

– Subscription based pricing– Incident based pricing

• Cloud alignment– Elastic pricing model– Cloud Software platforms use open source

• Innovation from wider community• Custom enhancements

32

Conclusion

33

Key Points Discussed

• Large Scale Integration• Impact of Big Data on Integration• Use cases

– Telecom– Securities

• Solutions– MRG– Infinispan– Data Services

• Open source differentiators

34

Questions

35