Scaling the Data Distribution Service to Global Networks

Angelo Corsaro, Ph.D.PrismTech

[email protected]

Delivering Performance, Openness, and Freedom

OpenSplice DDS

Sara Tucci-Piergiovanni, Ph.D.University of Rome “La Sapienza”

[email protected]

Scaling the Data Distribution Service to Global Networks

Age

nda

State of the Art

Scaling DDS to Global Networks

Extensions to the Standard

Concluding Remarks

Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.

© 2009, PrismTech. All Rights Reserved

Addressing Data Distribution Challenges

The OMG DDS Standard‣ Introduced in 2004 to address the Data

Distribution challenges faced by a wide class of Defense and Aerospace Applications

‣ Key requirement for the standard were its ability to deliver very high performance while seamlessly scaling from embedded to ultra-large-scale deployments

‣ Today recommended by key administration worldwide and widely adopted across several different application domains, such as, Automated Trading, Simulations, SCADA, Telemetry, etc.

DDS is standard designed to address the data-distribution challenges across a wide class of Defense and Aerospace Applications

Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.

© 2009, PrismTech. All Rights Reserved

Addressing Data Distribution Challenges

‣ DDS is swiftly being adopted as the standard for providing System of Systems with ubiquitous and access to tactical data

‣ DDS deployments on WAN and sizable MAN have also proven it fitness with Metropolitan/Wide Area Networks

So... Why I am doing this presentation?

Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.

© 2009, PrismTech. All Rights Reserved

ChallengesDDS presents today some limitations with respect to addressing large scale geographical deployments, these are:

Discovery

‣ Standard Discovery Protocol Requires Multicast

‣ Scalability

‣ Limited Control over Topics Visibility, e.g., choosing what to expose to the external world

Communication

‣ Standard Communication Protocol Relies on UDP

‣ No Support for Compression

‣ Difficult NAT/Firewall traversal

‣ Sub-Optimal Data Distribution for WAN Scenario (limited exploitation of multicast)

‣ Scalability

Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.

© 2009, PrismTech. All Rights Reserved

Challenges

A

B

CD

E

F

m

J

K

Publisher

Subscriber

Publisher

Publisher

Subscriber

Wan

DDS-1

A

B

ZX

Y

F

m

J

K

Publisher

Subscriber

Publisher

Subscriber

Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.

© 2009, PrismTech. All Rights Reserved

Challenges

A

B

CD

E

F

m

J

K

Publisher

Subscriber

Publisher

Publisher

Subscriber

Wan

DDS-1

A

B

ZX

Y

F

m

J

K

Publisher

Subscriber

Publisher

Subscriber

AA

Age

nda

State of the Art

Scaling DDS to Global Networks

Extensions to the Standard

Concluding Remarks

Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.

© 2009, PrismTech. All Rights Reserved

Looking at Internet Apps.

When looking at a solutions for extending DDS applicability to Internet Scale Systems, we can’t ignore architectures adopted by very successful Internet Applications.

‣ P2P application like KaZaa have shown to scale very well and to support millions of users while effectively managing millions of subjects

‣ Application like Skype are connecting millions of people and managing in real-time they presence (discovery)

Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.

© 2009, PrismTech. All Rights Reserved

Architecture for Internet Scale DDS

‣ DDS-Level Routers will be introduced that establish a Peer-to-Peer overlay that relays DDS data over the WAN‣ The Router provides a single access point for

deciding what to expose to the external world

‣ The Router can perform Topic Transformation, etc.

‣ A Network of Super-Nodes is used for ensuring:‣ Scalable (Global) Discovery

‣ Subscription Management (when crossing the boundaries of a System)

‣ Support for establishing communication between Pub and Sub (perhaps via STUNT to circumvent NAT/FIREWALL issues)

N1,0

N1,1

N2,0

N2,1

N2,2 N

2,3

N2,4

N2,5

H1,0,0

H1,0,1

H1,0,k

H1,1,h

H1,1,0

H1,1,1

H2,0,i

H 2,1,j

H 2,1,k H2,1,h

H 2,1,n

H 2,1,m

R

R

R

R

R

R

R

R

SN

SN

SN

SN

SN

Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.

© 2009, PrismTech. All Rights Reserved

DDS Router‣ DDS Router communicates with the

local DDS Domain by Multicast

‣ DDS Router are configured with the set of topics to be exposed along with potential transformations

‣ Router2Router communication is carried on by RTPS over TCP (since TCP has been tuned over years to work fine in WAN

‣ If required the Router takes care of compressing Data

‣ DDS Router Runs a “variation” or DDSI/RTPS

Nk

R

Ni R

‣ Topic, Subscription and Publication are propagated to Super Nodes

‣ Information available into super nodes ensure that local reader/writer will be properly matched

Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.

© 2009, PrismTech. All Rights Reserved

DDS Super-Nodes (SN)

‣ Super-Nodes store all the “discovery information” ‣ List of Topics available in the system

‣Associations between Topics and Router role (e.g. Pub or Sub)

‣ A distributed protocol is used to replicated data, eventually, on all replicas

‣ Router keep a list of well-known Super-Nodes to bootstrap discovery

‣ NOTE: Super-Nodes can also be used to support the implementation of STUNT-like protocols to facilitate NAT/Firewall trespassing

R

R

R

SN

SN

SN

SN

R

Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.

© 2009, PrismTech. All Rights Reserved

How it Works

DDS

R3

DDSR4

DDS

R2

SN1 SN4

SN2 SN3

DDS R1

{P(A),S(B),P(C)}

{S(A),P(B),P(C)} {S(A),P(D),S(E)}

{S(B), S(D),P(E)}

Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.

© 2009, PrismTech. All Rights Reserved

How it Works

DDS

R3

DDSR4

DDS

R2

SN1 SN4

SN2 SN3

DDS R1

R1-{P(A),S(B),P(C)}

R2-{S(A),P(B),P(C)}

R3-{S(A),P(D),S(E)}

R4-{S(B), S(D),P(E)}

{P(A),S(B),P(C)}

{S(A),P(B),P(C)} {S(A),P(D),S(E)}

{S(B), S(D),P(E)}

Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.

© 2009, PrismTech. All Rights Reserved

How it Works

DDS

R3

DDSR4

DDS

R2

SN1 SN4

SN2 SN3

DDS R1

R1-{P(A),S(B),P(C)}R2-{S(A),P(B),P(C)}R3-{S(A),P(D),P(E)}R4-{S(B), S(D),S(E)}

{P(A),S(B),P(C)}

{S(A),P(B),P(C)} {S(A),P(D),S(E)}

{S(B), S(D),P(E)}

Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.

© 2009, PrismTech. All Rights Reserved

How it Works

DDS

R3

DDSR4

DDS

R2

SN1 SN4

SN2 SN3

DDS R1

R1-{P(A),S(B),P(C)}R2-{S(A),P(B),P(C)}R3-{S(A),P(D),P(E)}R4-{S(B), S(D),S(E)}

{P(A),S(B),P(C)}

{S(A),P(B),P(C)} {S(A),P(D),S(E)}

{S(B), S(D),P(E)}

{P(A)}

{P(A)}

{P(B)}

{P(B)}

{P(D)}

{P(E)}

Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.

© 2009, PrismTech. All Rights Reserved

How it Works

DDS

R3

DDSR4

DDS

R2

SN1 SN4

SN2 SN3

DDS R1

R1-{P(A),S(B),P(C)}R2-{S(A),P(B),P(C)}R3-{S(A),P(D),P(E)}R4-{S(B), S(D),S(E)}

{P(A),S(B),P(C)}

{S(A),P(B),P(C)} {S(A),P(D),S(E)}

{S(B), S(D),P(E)}

AA

AAA AAA

{P(A)}

{P(A)}

{P(B)}

{P(B)}

{P(D)}

{P(E)}

Age

nda

State of the Art

Scaling DDS to Global Networks

Extensions to the Standard

Concluding Remarks

Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.

© 2009, PrismTech. All Rights Reserved

Extension to the DDS Standard

‣What described in the previous slides can easily be implemented in an interoperable manner, exploiting the pre-designed extension points present in the DDS Standard

‣Areas of extension include:

‣Discovery

‣DDSI/RTPS

Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.

© 2009, PrismTech. All Rights Reserved

Discovery Extension

‣ A new Discovery Protocol should be added that allows to discover relevant information (Topics, Subscriptions, Publications) via the Super-Nodes

‣ A new protocol to be used between Super-Nodes should also be standardized!

R

R

R

SN

SN

SN

SN

R

1

2

1

2

Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.

© 2009, PrismTech. All Rights Reserved

DDSI/RTPS Protocol xTensions

DDSI/RTPS Should be extended to include:

‣Support for TCP/IP ‣Offloading reliability from DDSI to TCP

whenever possible

‣Compression

DDS

R3

DDSR4

DDS

R2

SN1 SN4

SN2 SN3

DDS R1

Age

nda

State of the Art

Scaling DDS to Global Networks

Extensions to the Standard

Concluding Remarks

Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.

© 2009, PrismTech. All Rights Reserved

Concluding Remarks‣ The DDS Standard includes today some known

limitation when trying to address Ultra-Large-Scale Systems

‣ These limitation mostly impact its scalability and efficiency

‣ The good news is that exploiting available extension points in the standard it is possible to extend the DDS Architectural Style to scale to Ultra-Large-Scale Systems

‣ The Router/Super-Nodes approach builds over years of experience in the domain of Internet Scale applications, such as, KaZaa, Skype, etc.

‣ Thus, PrismTech will work toward update of the DDS Spec to include these extension points.