RedHat MRG and Infinispan for Large Scale Integration

Using MRG and Infinispan

for

Large Scale Integration

Prajod Vettiyattil

2

What this session is about

Challenges in Large Scale integration

Use cases for Large Scale integration

How to solve the challenges Solution Products to implement the

solution

The Open Source difference

3

Challenges

Use Cases

Solutions

Key Phrases

4

Phrases: 1

• Large Scale Integration – Integration of 10s or 100s of systems, and

exchange GBs of messages in a day

• Big Data – A changing threshold – Data in the Terabytes, Petabytes, Exabytes…

• Asynchronous Messaging – Message oriented middleware

• Real time systems – Systems that are built to respond to requests in

real time, with predicable, consistent response times

5

Phrases: 2

• Grid – A set of interconnected computers that work in

parallel to solve a computing problem

• Cloud Computing – Computing as a service – Client of the cloud is isolated from the details of

the implementation of the service

6

7

Challenges

Challenges

Use Cases

Solutions

The Key Challenges Large number of systems

8

The Key Challenges Complexity of connection between these systems

9

The Key Challenges

Constraints on the systems and on the connections

10

soap /

http csv /

ftp

Rest /

http csv /

file Adapter

Adapter soap

/ http csv /

ftp

Rest

/ http csv /

file

Adap

ter

Rest /

http csv /

file Adapter

Text/tcp soap /

http csv /

ftp

soap /

http csv /

ftp

Rest /

http csv /

file Adapter

csv /

file

Adap

ter

soap

/ http csv /

ftp

Rest

/ http csv /

file

Adap

ter

csv /

file

Adap

ter Adap

ter

11

Use Cases and Solutions

Challenges

Use Cases

Solutions

Architecture

Wireless Telco BSS Integration

BSS

Applications

Mediation Provisioning CRM Workforce

Management

Number Inventory

Management

Interconnect

Infrastructure

Services for

Middleware

Transport

Services

Caching

Services

Load Balancing

Services

Recovery

Services

Failover

Services

Element

Management

Fault

Management

Revenue

Management

Process Automation/ Business Process Management

Ad

min

istratio

n a

nd

Mo

nito

ring S

ervices

Secu

rity S

ervices

Portals, Front End System, Partner Gateways

Telco Network Systems

Middleware Message Broker Enterprise Service Bus JEE Server

Billing

12

Telco scalability

Some requirements

• 75+ million customers • Plan for Terabytes of CDRs and other

messages per day • Performance is critical to customer

experience and retention • CRM, Billing, Mediation, Middleware, Data

warehouse

13

High volume use case 1

CRM to Billing Integration

Middleware Mediation Network

Switch Billing

Data

Warehouse

Fraud

Management CRM

Other BSS

Apps

14

Point to Point Connection

Technical Requirements • Memory

• Threads

• Sockets

• Sender performance

• Receiver performance

High volume use case 1

CRM to Billing Integration: with Middleware Infrastructure

Middleware Mediation Network

Switch Billing

Data

Warehouse

Fraud

Management

Middleware

Infrastructure

• Caching

• Load Balancing

• Failover

• Recovery

CRM

Other BSS

Apps

15

Middleware Mediatio

n

Networ

k

Switch

Billing

Data

Warehouse

Fraud

Management CRM

Other BSS

Apps

Middleware Infrastructure expanded

Middlewar

e

Medi

ation

Ne

tw

ork

Sw

itc

h

Billing

Data

Warehous

e

Fraud

Managem

ent

Middlewar

e

Infrastruct

ure

CRM

Other BSS

Apps

Middleware

Middleware Infrastructure

High Speed,

Reliable

Massaging

Compute Node

Scaling

Resource

Management

Work Load

Management

Failover Recovery Distributed

Caching Cluster Toolkit

16

Middleware Infrastructure

Products

Middlewar

e

Medi

ation

Ne

tw

ork

Sw

itc

h

Billing

Data

Warehous

e

Fraud

Managem

ent

Middlewar

e

Infrastruct

ure

CRM

Other BSS

Apps Middleware

Middleware Infrastructure High Speed,

Reliable

Massaging

Compute Node

Scaling

Resource

Management

Work Load

Management

Failover Recovery Distributed

Caching Cluster Toolkit

• Any ESB

• Any JEE Server

• Any Message Broker

MRG Messaging

MRG Grid

Infinispan

1

2 3 4 5

6

8

7

17

Middleware Infrastructure: Products

MRG Messaging

Middlewar

e

Medi

ation

Ne

tw

ork

Sw

itc

h

Billing

Data

Warehous

e

Fraud

Managem

ent

Middlewar

e

Infrastruct

ure

CRM

Other BSS

Apps

Middleware Infrastructure

High Speed,

Reliable

Massaging

Compute Node

Scaling

Resource

Management

Work Load

Management

Failover Recovery Distributed

Caching Cluster Toolkit

MRG Messaging

• AMQP support

• Native RDMA,

Infiniband

• Can use MRG Realtime

• Large message

support(> GB)

• Clustering and Failover

• High speed, journal

based persistence

• Java and C++ brokers

• Based on Apache Qpid

18

Middleware Infrastructure: Products

MRG Grid

Middlewar

e

Medi

ation

Ne

tw

ork

Sw

itc

h

Billing

Data

Warehous

e

Fraud

Managem

ent

Middlewar

e

Infrastruct

ure

CRM

Other BSS

Apps

Middleware Infrastructure

High Speed,

Reliable

Massaging

Compute

Node Scaling

Resource

Management

Work Load

Management

Failover Recovery Distributed

Caching Cluster Toolkit

MRG Grid

• Scalable Grid Scheduler

• Resource variety:

Desktop to Cloud

schedulers

• Low latency results:

Using MRG Messaging

• Dynamic provisioning

• High Availability

• Grid Federation

• Is based on the Condor

Grid project

19

Middleware Infrastructure: Products

Infinispan

Middlewar

e

Medi

ation

Ne

tw

ork

Sw

itc

h

Billing

Data

Warehous

e

Fraud

Managem

ent

Middlewar

e

Infrastruct

ure

CRM

Other BSS

Apps

Middleware Infrastructure

High Speed,

Reliable

Massaging

Compute Node

Scaling

Resource

Management

Work Load

Management

Failover Recovery Distributed

Caching

Cluster

Toolkit

Infinispan

• In memory Data Grid

• Distributed cache

• Peer to peer

communication between

nodes

• Flexible persistence:

JDBC, File, Amazon S3

• Map reduce: node local

computing

• Implementation for

performance

20

High volume use case 2

Post Trade Securities Processing

Processing Nodes

Aggregator

Node1 Node2

Node3 Node4

File Splitter +

Load

Allocator

Post Trade

Files

Trading

Applications

Trading

Applications

Accounting

Solution

Risk

Management

Solution

Sources Targets Data Processing Solution

Output

Channels

21

Data Services

Ref Data Solution Customer Master Src 3 Src 4

Post Trade Securities Processing

Technical Requirements

22

Processing Nodes

Aggregator

Node1 Node2

Node3 Node4

File Splitter

+

Load

Allocator

Input

Channels

Output

Channels

Data Services Technical Requirements • File Streaming

• Multiple Data views, Data sources

• Data Aggregation

• Reliable Messaging

Post Trade Securities Processing

Products

Processing Nodes

Aggregator

Node1 Node2

Node3 Node4

File Splitter

+

Load

Allocator

MRG Messaging

MRG Grid

Infinispan

1

5

3

2

4

Input

Channels

Output

Channels

6

7

23

Data Services JBoss Data Services

MRG Realtime

• Consistent, predictable response

• Websphere Realtime: RTSJ

24

Messages/microsecond

Re

sp

on

se

Tim

e

Source:

Red Hat

A Recap of the Solution

25

Challenges

Use Cases

Solutions

Challenges, Solutions and Products Challenge Solution Products

1 Small data elements,

high volume

Distribution, load balancing and

partitioning

MRG Messaging, MRG Grid

2 Large data elements File splitting, distribution, in

place processing

MRG Messaging, MRG Grid

3 Data views, Many data

sources

Data Services JBoss Data Services

3 Predictability Real time kernels, real time

JVMs

MRG Realtime, RTSJ

4 Availability Load balancing, clustering,

failover

MRG Grid, Infinispan

5 Reliability File based caches, DB

persistence

MRG Messaging, Grid,

Infinispan

6 Scalability Compute grids, Data grids,

Asynchronous messaging

MRG Messaging, Grid,

Infinispan

26

Solution Alternatives

27

The Map Reduce method

• Split data, process in parallel, aggregate results

Split

Data

Map Phase Reduce Phase

Task Tracker Task Tracker

Task Tracker Task Tracker

Output

Data

Job Tracker

Data

Name

Node

28

Client

Input

Data

GT3 and Condor

• Globus Toolkit

– Open source toolkit for Compute Grids

– Architecture, Service Model and Implementation

– Job Tracking, Management, Monitoring, Resource

Management

– Data Management: Movement, Location Registry

• Condor

– Grid Framework from University of Wisconsin

– Compute Node Scaling

– Job Scheduling

– Idle time utilization

29

Commercial Tools

• IBM

– IBM Cloudburst

– Websphere Virtual Enterprise

– Websphere Realtime

• Oracle

– Oracle Exalogic

– Oracle Coherence

– Oracle Grid Engine(Sun Grid Engine)

• Terracotta

– Quartz

– Big Memory

30

The Open Source difference

31

The Advantages

• Smaller adoption steps to reduce risk • Flexible Cost Model

– Subscription based pricing – Incident based pricing

• Cloud alignment – Elastic pricing model – Cloud Software platforms use open source

• Innovation from wider community • Custom enhancements

32

Conclusion

33

Key Points Discussed

• Large Scale Integration • Impact of Big Data on Integration • Use cases

– Telecom – Securities

• Solutions – MRG – Infinispan – Data Services

• Open source differentiators

34

Questions

35