Reactive Data System in Practice

Reactive Data System in Practice

Or how to solve Fast Data Problem and reacting to the World faster

By Triều Nguyễn at http://trieu.xyz http://www.rfxlab.com (Reactive Data System Lab)

λ(x)

2008: Java Developer, develop Social Trading Network for a startup (Yopco)2011: joined FPT Online, software engineer, worked in FPT ID project and VnExpress Mobile (Backend)2012: backend engineer at Greengar (Scaling server and real-time services)12/2012 to 05/2015: back to FPT Online, lead engineer at eClick Ad Platform06/2015 to now: lead engineer at iTVad, a startup project at FPT for OTT and TV Advertising Solutions

about me

1. Just some concepts 2. Ideas in practice3. What is RFX framework ?4. Why is RFX ?5. How to solve problems with RFX ?

a. Counting pageview of websiteb. Counting unique user of websitec. Querying data more reactive and adaptive

6. Final ideas

Contents of this talk

1 - Just some concepts

Fast Data vs Big Data

What is Reactive Data System ?

is a software design approach that uses the principles of event driven architecture for the design and implementation of data intensive applications.

Event Driven ArchitectureEDA is a framework that promotes production, detection, consumption and reaction to the events. Event based applications will run on RDS framework.

Why is Reactive Data SystemWe want to ask bigger questions faster

2 - Ideas in practice

Problem (User Story in Practice)

● User does login with FB Connect at Ecommerce website

● User does a click on an item, then system should do the following tasks:1. Find best items to recommend 2. Find a best item to send email marketing3. If there are more than 100 users, that click on a item,

run a marketing campaign on Facebook automatically.

An example from E-Commerce

3 - What is RFX framework ?

● A framework for big fast data problems● A collection of Open Source Tools from Netty to Spark● The mission of RFX

1. Build data product quickly with design patterns2. React to critical events in real-time

What is RFX or Reactive Function X ?

RFX is inspired by the Actor Model Theory

Paper: http://vialab.science.uoit.ca/textvis2011/papers/textvis%202011-rohrdantz.pdf

RFX is designed for startup business

I design RFX because I have to solve following problems both at FPT and at RFXLab.com● Marketing Automation● Product UX personalization● Real-time Advertising● System monitoring (frontend and backend)● Email marketing● Product Analytics● Customer Analytics

→ So I build a full-stack framework (a collection of tools for both backend and frontend)

Philosophy of RFX

Sub modules in RFX framework

4 - Why is RFX framework ?

Why Rfx ?

● Ideas since 2012 (from Actor model theory)● R&D and Deployed in Production since 2013

● Open Source: Apache License, Version 2.0 ● Full Stack: from Frontend to Backend● Agile for Data Analytics● Lightweight Lambda Architecture

● Really fast and near-real-time processing● Tested with 1.000.000 events / second● Simple development model for data engineer

Roadmap to the future

5 - How to solve problems with RFX ?

ProblemsUser story in Digital Marketing: 1. Counting pageview of website2. Counting unique user of website3. Real-time marketing

Find rush hour in your system

Which device user reach our content

Solution Architecture

Tracking(RFX-track)

Event Queue (Apache Kafka)

Event Processor(RFX-stream)

RedisMongoDB Synch Data Jobs(RFX-job)

Report(RFX-report) Event Actor

(RFX-actor)Ad Server

An example topology

6 - Final ideas

Summary

● Fast Data is cool● Fast data: The next step after big data● RFX is designed to solve common Fast Data

problems in Digital Marketing and Advertising

https://github.com/rfxlab/rfx

The end and thank youhttps://github.com/rfxlab

http://rfxlab.com

λ