Reactive Stream Processing Using DDS and Rx

Your systems. Working as one.

Reactive Stream Processing

using DDS and Rx

www.rti.com

Sumant Tambe, Ph.D.

Senior Software Research Engineer and Microsoft MVP

Real-Time Innovations, Inc.

@sutambe

Oct. 11, 2014

Outline

• Reactive Systems

• Stream Processing

• Overview of Reactive Extensions

• Overview of DDS

• Streaming Shapes Demo in C#

10/10/2014 2Real-Time Innovations, Inc.

Systems Everyone Wants to Build

• Event-Driven: Modular, pipelined, asynchronous

• Elasic: Scales easily up/down with load and cpu cores.

• Resilient: fault-tolerant

• Responsive: Reacts to events at the speed of environment

10/10/2014 Real-Time Innovations, Inc. 3

Networking Middleware

App

Stream Processing

• Stream Processing is the term used to describe an architectural style that operate on a continuous sequence of data.

10/10/2014 Real-Time Innovations, Inc. 4

Shape of an application

o/p

Where Once CombineLatest Select Scan Merge Raw Data

i/p

*nix command line (pipes and filter)

$ ls -1 | grep “search” | grep “research”

research

$

10/10/2014 Real-Time Innovations, Inc. 6

$ cat - | grep “Real”

R

Real

Real

[Ctrl+C]

$

Reactive Extensions• The Reactive Extensions (Rx) is a

library for composing asynchronous and event-based programs using observable sequences

• Rx = Observables + Composition + Schedulers

• Streams are first-class

• Filter, project, aggregate, compose and perform time-based operations on multiple streams

• Uses Functional Programming Style

• Rx.NET, RxJava, RxJS, RxCpp, RxRuby, RxPyton, and more…

10/10/2014 Real-Time Innovations, Inc. 7

10/10/2014 Real-Time Innovations, Inc. 8

Data Distribution Service (DDS)

A Reactive Middleware

DDS: Standards-based Integration Infrastructure for Critical Applications

© 2009 Real-Time Innovations, Inc.

Streaming

DataSensors Events

Real-Time

Applications

Enterprise

ApplicationsActuators

Systems that interact with the Real World

• Must adapt to changing environment• Cannot stop processing the information• Live within world-imposed timing

Beyond traditional interpretation of real-time

© 2010 Real-Time Innovations, Inc.

Real-Time Data Distribution

10/10/2014 © 2012 RTI • COMPANY 11

RPC over

DDS

2014

DDS

Security

2012

Family of Specifications

© 2009 Real-Time Innovations, Inc. COMPANY 12

DDSImplementation

Network / TCP / UDP / IP

App

DDSImplementation

App

DDSImplementation

DDS Spec

2004

DDS

Interoperablity

2006

UML Profile

for DDS

2008

DDS for

Lw CCM

2009

DDS

X-Types

2010 2010

DDS-STD-C++

DDS-JAVA5

App

Evolution of DataBus

Data-centricity basics

Everyday Example: Schedule Meeting via Emails

Alternative Process #1 (message-centric):

1. Email: “Meeting Monday at 10:00.”

2. Email: “Here’s dial-in info for meeting…”

3. Email: “Meeting moved to Tuesday”

4. You: “Where do I have to be? When?”

5. You: (sifting through email messages…)

14

Everyday Example: Schedule Meeting Using a Calendar

Alternative Process #2:

1. Calendar: (add meeting Monday at 10:00)

2. Calendar: (add dial-in info)

3. Calendar: (move meeting to Tuesday)

4. You: “Where do I have to be? When?”

5. You: (check calendar. Contains consolidated-state)

15

The difference is state!

The infrastructure consolidates changes and maintains it

10/10/2014 © 2012 RTI • COMPANY 16

DDS Communication Model

DDS for Distribution, Rx for Processing

10/10/2014 Real-Time Innovations, Inc. 17

DR

DR

DR

Observable Observer

DW

DW

Processing

Rx4DDS.NET = DDS + Rx

10/10/2014 © 2012 RTI • COMPANY 18

• DDS wrapper for Rx.NET

• In C#

• Anything that produces data is an Observable

– Topics, Discovery, Statuses, statuses, etc.

DDS and Rx: A Great Match

10/10/2014 © 2012 RTI • COMPANY 19

DDS Concept Rx Concept/Type/Operator

Topic of type T An object that implements IObservable<T>, which internally creates a DataReader<T>

Communication status, Discovery event streams

IObservable<SampleLostStatus>

IObservable<SubscriptionBuiltinTopicData>

Topic of type T with key type=Key

IObservable<IGroupedObservable<Key, T>>

Detect a new instance Notify Observers about a new IGroupedObservable<Key, T> with key==instance. Invoke IObserver<IGroupedObservable<Key, T>>.OnNext()

Dispose an instance Notify Observers through IObserver<IGroupedObservable<Key,T>>.OnCompleted()

Take an instance update of type T

Notify Observers about a new value of T using Iobserver<T>.OnNext()

Read with history=N IObservable<T>.Replay(N) (Produces a new IObservable<T>)

DDS and Rx: A Great Match

10/10/2014 © 2012 RTI • COMPANY 20

DDS Concept Rx Concept/Type/Operation

Query Conditions Iobservable<T>.Where(…) OR

Iobservable<T>.GroupBy(…)

SELECT in CFT expression IObservable<T>.Select(...)

FROM in CFT expression DDSObservable.FromTopic(“Topic1”)

DDSObservable.FromKeyedTopic(“Topic2”)

WHERE in CFT expression IObservable<T>.Where(...)

ORDER BY in CFT expression IObservable<T>.OrderBy(...)

MultiTopic (INNER JOIN) IObservable<T>.Join(...)

.Where(...)

.Select(...)

Join between DDS and non-DDS data

Join, CombineLatest, Zip

Living Demo in C#

10/10/2014 Real-Time Innovations, Inc. 21

Background Code

10/12/2014 Real-Time Innovations, Inc. 22

public class ShapeType : ICopyable<ShapeType>{

public string color;public int shapesize;public int x;public int y;

public ShapeType();}

public class ShapeTypeExtended : ShapeType{

public float angle;public ShapeFillKind fillKind;

public ShapeTypeExtended();}

private DDS.TypedDataWriter<ShapeTypeExtended> triangle_writer;

Flower Demo

10/12/2014 Real-Time Innovations, Inc. 23

IDisposable flower(DDS.DomainParticipant participant){

int a = 30, b = 30, c = 10;

returnObservable

.Interval(TimeSpan.FromMilliseconds(1), Scheduler.Immediate)

.Select((long x) =>{

int angle = (int)(x % 360);return new ShapeTypeExtended{

x = (int)(120 + (a + b) * Math.Cos(angle) + b * Math.Cos((a / b - c) * angle)),y = (int)(120 + (a + b) * Math.Sin(angle) + b * Math.Sin((a / b - c) * angle)),color = "GREEN",shapesize = 5

};}).Subscribe(triangle_writer);

}

Simple Square to Triangle Transformation

10/12/2014 Real-Time Innovations, Inc. 24

IDisposable forward_verbose(DDS.DomainParticipant participant){

var rx_reader = DDSObservable.FromTopic<ShapeTypeExtended>(participant, "Square");

IDisposable disposable =rx_reader.OnDataAvailable((ShapeTypeExtended shape) =>{

DDS.InstanceHandle_t handle = DDS.InstanceHandle_t.HANDLE_NIL;triangle_writer.write(shape, ref handle);

});

return disposable;}

IDisposable forward_shortest(DDS.DomainParticipant participant){

return DDSObservable.FromTopic<ShapeTypeExtended>(participant, "Square").OnDataAvailable(triangle_writer);

}

Swap Square’s x and y and Propagate Stream Dispose Event

10/12/2014 Real-Time Innovations, Inc. 25

IDisposable swap(DDS.DomainParticipant participant){

return DDSObservable.FromTopic<ShapeTypeExtended>(participant, "Square").Select(shape => new ShapeTypeExtended{

x = shape.y,y = shape.x,color = shape.color,shapesize = shape.shapesize

}).SubscribeAndDisposeOnCompleted(triangle_writer,

new ShapeTypeExtended { color = "BLUE" });}

Aggregator (Square + Circle = Triangle!)

10/12/2014 Real-Time Innovations, Inc. 26

IDisposable aggregator(DDS.DomainParticipant participant){

var rx_square_reader =DDSObservable.FromTopic<ShapeTypeExtended>(participant, "Square");

var rx_circle_reader =DDSObservable.FromTopic<ShapeTypeExtended>(participant, "Circle");

return new CompositeDisposable(new IDisposable[] {

rx_square_reader.Subscribe(triangle_writer),rx_circle_reader.Subscribe(triangle_writer)

});

}

Square/Circle Correlator using LINQ

10/12/2014 Real-Time Innovations, Inc. 27

IDisposable selectmany_correlator(DDS.DomainParticipant participant, bool useLinq){

var rx_circle_reader =DDSObservable.FromTopic<ShapeTypeExtended>(participant,

"Circle", Scheduler.Default);

var rx_square_reader =DDSObservable.FromTopic<ShapeTypeExtended>(participant,

"Square", Scheduler.Default);

var correlator =from square in rx_square_readerfrom circle in rx_circle_reader.Take(1)where square.color == circle.colorselect new ShapeTypeExtended{

x = square.x,y = square.y,color = square.color,shapesize = circle.x

};

return correlator.Subscribe(triangle_writer);}

Thank You!

• Rx4DDS.NET– https://github.com/rticommunity/rticonnextdds-reactive

• Data Distribution Service– http://portals.omg.org/dds

• Real-Time Innovations, Inc.– www.rti.com

10/12/2014 Real-Time Innovations, Inc. 28