ApacheCon 2015 - A Stock Prediction System Using OSS

A Stock Prediction System using open-source software

Fred Melo

[email protected]

@fredmelo_br

William Markito [email protected]

@william_markito

It's all about DATA

Data Sources Look for patterns

Prediction

Machine Learning is the answer

Neural Networks

Clustering Genetic Algorithms

Train with historical dataset

Apply model to the new input

Applying Machine Learning

Hard to add new data sources

Why?

Hard to scale

Why so hard?

Hard to make it real-time

HDFS

Data Lake

Store Analytics

Hard to change Labor intensive

Inefficient

No real-time information ETL based Data-source specific

Traditional models are reactive and static

HDFS Data Lake Expert System /

Machine Learning

In-Memory Real-Time Data

Continuous Learning Continuous Improvement

Continuous Adapting

Data Stream Pipeline

Multiple Data Sources Real-Time Processing Store Everything

Stream-based, real-time closed-loop analytics are needed

Info

Analysis

Look at past trends (for similar input)

Evaluate current input

Score / Predict

Neural Network

How can it be addressed?

Info

Analysis

Filter

[ json ]

Neural Network

How can it be addressed?

Info

Analysis

Filter Enrich Neural Network

How can it be addressed?

Info

Analysis

Neural Network Filter Enrich Transform

How can it be addressed?

Info

Analysis

Filter Enrich Transform

Neural Network

How can it be addressed?

Info

Analysis

Filter Enrich Transform

Transform

Neural Network

How can it be addressed?

Neural Network

In-Memory Data Grid Real-time scoring

How can it be addressed?

Train

Neural Network

In-Memory Data Grid

Front-end

Update Push

How can it be addressed?

Ingest Transform Sink SpringXD

Store / Analyze

Fast Data

Distributed Computing

Predict / Machine Learning

Other Sources and Destinations

JMS

Streaming real-time analytics architecture

Transform Sink

SpringXD

Extensible Open-Source Fault-Tolerant Horizontally Scalable

HTTP

Machine Learning

Fast Data

Filter

Predict Sink HTTP

Split

Dashboard

Push

Demo Architecture

SpringXD

shell - R

Transformer

geode-json client

geode-json client

http-client

http-server

obj-to-json

splitter

splitter

Simulator

tap

SpringXD

INGEST / SINK PROCESS ANALYZE

•  Little or no coding required

•  Dozens of built-in connectors

•  Seamless integration with Kafka, Sqoop

•  Create new connectors easily using Spring

•  Call Spark, Reactor or RxJava

•  Built-in configurable filtering, splitting and transformation

•  Out-of-box configurable jobs for batch processing

•  Import and invoke PMML jobs easily

•  Call Python, R, Madlib and other tools

•  Built-in configurable counters and gauges

Data Stream Pipelining

SpringXD

XD Nodes XD Nodes XD Nodes XD Nodes

Ingest

SpringXD

Split Filter Transform Sink

XD admin

XD Nodes

Ingest Split Filter Transform Sink

Stream Deployment

Messaging

Scale-Out and HA Architecture

Transform Sink

SpringXD

Extensible Open-Source Fault-Tolerant Horizontally Scalable

HTTP

Machine Learning

Fast Data

Filter

Predict Sink HTTP

Split

Dashboard

Push

Demo Architecture

Geode client-server architecture

Partitioned Regions

Event handling

Transform Sink

SpringXD

Extensible Open-Source Fault-Tolerant Horizontally Scalable

HTTP

Machine Learning

Fast Data

Filter

Predict Sink HTTP

Split

Dashboard

Push

Demo Architecture

Neural Networks

Neural Networks

medium avg (x+1)

relative strength (x)

medium avg (x)

price(x)

Neural Network

Neural Network

Transform Sink

SpringXD

Extensible Open-Source Fault-Tolerant Horizontally Scalable

HTTP

Machine Learning

Fast Data

Filter

Predict Sink HTTP

Split

Dashboard

Push

Demo Architecture

Demo Time

SpringXD

shell - R

Transformer

geode-json client

geode-json client

http-client

http-server

obj-to-json

splitter

splitter

Simulator

tap

SpringXD

http://projectgeode.org http://projects.spring.io/spring-xd http://www.r-project.org

Follow-up: In-Memory Unconference

"A place for all things in-memory: projects, people, ideas, roadmaps, discussions."Location: Hill Country A/B”

Weds 4:15pm - 6pm. (after this talk)

The demo code is on GitHub! @fredmelo_br @william_markito