Processing Twitter Events in Real-Time with Oracle Event Processing (OEP) 12c

2013 © Trivadis

BASEL BERN BRUGG LAUSANNE ZUERICH DUESSELDORF FRANKFURT A.M. FREIBURG I.BR. HAMBURG MUNICH STUTTGART VIENNA

2013 © Trivadis

Echtzeitverarbeitung von Twitter Streams mit Oracle Event Processing (OEP) 12c

Guido Schmutz

DOAG Konferenz 2014

19.11.2014 – 15:00 Raum Kiev

19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP)

INFOBOX – Read and delete •  A heading and an optional sub-heading

can be placed on the first slide. •  The title is written directly under the

name (Shift+Return) •  If multiple speakers are named, please

just write the names one underneath the other (there is normally no space for titles, etc.)

2013 © Trivadis

Guido Schmutz

•  Working for Trivadis for more than 17 years

•  Oracle ACE Director for Fusion Middleware and SOA •  Co-Author of different books •  Consultant, Trainer Software Architect for Java, Oracle, SOA and

Big Data / Fast Data •  Member of Trivadis Architecture Board •  Technology Manager @ Trivadis

•  More than 25 years of software development experience

•  Contact: guido.schmutz@trivadis.com •  Blog: http://guidoschmutz.wordpress.com •  Twitter: gschmutz

2013 © Trivadis

AGENDA

1.  Introduction

2.  Twitter Use Case

3.  Processing with Oracle Event Processing (OEP)

4.  Visualization with Oracle Business Activity Monitoring (BAM)

5.  Event Processing in Information Architecture

INFOBOX – Read and delete •  If the agenda is used as an interim

page, please highlight the relevant chapter in red font.

•  To allow optimum alignment of objects,

display the drawing guides (right-click on the page)

2013 © Trivadis

Big Data Definition (4 Vs)

+ Time to action ? – Big Data + Event Processing = Fast Data

Characteristics of Big Data: Its Volume, Velocity and Variety in combination

2013 © Trivadis

The world is changing …

The model of Generating/Consuming Data has changed ….

Old Model: few companies are generating data, all others are consuming data

New Model: all of use are generating data, and all of us are consuming data

2013 © Trivadis

Internet Of Things – Sensors are/will be everywhere

There are more devices tapping into the internet than people on earth

How do we prepare our systems/architecture for the future?

7 Source: Cisco Source: The Economist

2013 © Trivadis

AGENDA

1.  Introduction

2013 © Trivadis

Retrieve Tweets and Visualize

2013 © Trivadis

Access to Tweets

Quelle

Source Limitations Cost Twitter’s Search API 3200 / user

5000 / keyword 180 requests / 15 minutes

Twitter’s Streaming API 1%-40% of total volume free

DataSift none 0.15 -0.20$ /

unit Gnip none On request

2013 © Trivadis

AGENDA

1.  Introduction

2013 © Trivadis

Oracle Event Processing (OEP) - Engine

Lightweight Java Application Server

•  Full environment for running Java applications

•  Module Framework - OSGi

High Throughput

•  Hundreds of thousands of events/second

Event Processing Infrastructure

Easy-to-use development environment

•  Service Framework – Spring DM, POJO

Multiple-choice VM

•  JRockit or WebLogic RealTime

2013 © Trivadis

Oracle Event Processing Application

Adapter •  Provide a way for EPN to exchange data with components that are external to network •  Either an Event Source (inbound) or an Event Sink (outbound)

Processor •  Contain query code written in CQL and consumes event data from a preceding

channel •  May also generate new events to an output channel •  CQL is an extension of SQL

2013 © Trivadis

Oracle Event Processing Application

Bean Bean

•  Provide a place for logic written in Java •  Java code can receive events (Event Sink) and/or send events (Event Source)

•  Are implemented using the Spring Framework model or the OEP event bean model Channel

•  Two types of channel exist: Stream and Relation

CQL Processor •  Executes Continous Query Language (CQL) queries •  Used to declaratively describe logic (queries) to be applied on the event stream

2013 © Trivadis

Oracle Event Processing - CQL

CQL is an extension to Structured Query Language (SQL) •  Same keywords and syntax rules, but additional support for streaming data

event conceptually corresponds to a row in a database table, but •  one event is always before or after another in time

•  stream is potentially infinite and ever-changing

Data Query

Result Tuples Result Tuples

Traditional Database CEP Engine

QueryData

2013 © Trivadis

Oracle Event Processing – Visual Development Tools

2013 © Trivadis

Oracle Event Processing – Operation & Management

2013 © Trivadis

Implementation – complete picture

Mention Extractor

Twitter Adapter

CounterProcessor

Hashtag Extractor

Author Extractor

BAM Tweet

BAM Counter

Working on my OEP & BAM demo for the session

”Echtzeitverarbeitung von Twitter Feeds …” #doag2014

#oep12c #bam12c http://bit.ly/1xafUZJ

gschmutz

#doag2014 #oep12

#bam12c,2

gschmutz,1

@doag2014,5 #oep12c,1 JMS

JMS Twitter

range 30 seconds slide 30 seconds

Sentiment Enricher

Working on my OEP & BAM demo for the session ”Echtzeitverarbeitung von Twitter Feeds …” #doag2014 #oep12c

#bam12c http://bit.ly/1xafUZJ

#bam12c

2013 © Trivadis

1) Creating a Twitter Adapter

Twitter Adapter

Twitter

2013 © Trivadis

2) Send Tweets to BAM

Twitter Adapter

BAM Tweet

JMS Twitter Sentiment Enricher

2013 © Trivadis

3) Extract interesting information from Tweet

Mention Extractor

Twitter Adapter

Hashtag Extractor

Author Extractor

BAM Tweet

gschmutz

#doag2014 #oep12

JMS Twitter Sentiment Enricher

#bam12c

2013 © Trivadis

4) Count occurrences within period

Mention Extractor

Twitter Adapter

CounterProcessor

Hashtag Extractor

Author Extractor

BAM Tweet

BAM Counter

gschmutz

#doag2014 #oep12

#bam12c,2

gschmutz,1

@doag2014,5 #oep12c,1 JMS

JMS Twitter

Sentiment Enricher

#bam12c

2013 © Trivadis

Implementing in Oracle Event Processing

Mention Extractor

Twitter Adapter

CounterProcessor

Hashtag Extractor

Author Extractor

BAM Tweet

BAM Counter

JMS Twitter

Sentiment Enricher

2013 © Trivadis

1) Creating Twitter Adapter – Connecting to Twitter Stream

2013 © Trivadis

1) Creating Twitter Adapter – Tweet Event

2013 © Trivadis

1) Creating Twitter Adapter – Adapter Factory

2013 © Trivadis

1) Creating Twitter Adapter – Assembly

2013 © Trivadis

1) Creating Twitter Adapter – Export Adapter to server

2013 © Trivadis

1) Creating Twitter Adapter – Using Twitter Adapter

2013 © Trivadis

2) Sentiment Enrichment

2013 © Trivadis

3) Sending Tweets to BAM

Using Oracle BAM Enterprise Message Sources (JMS) interface

2013 © Trivadis

3) Sending Tweets to BAM – Convert event to JMS MapMessage

2013 © Trivadis

4) Extract information from Tweet – Extract Hashtags from TweetEvent

2013 © Trivadis

4) Extract information from Tweet – Extract Hashtags from TweetEvent

5) Count hashtag occurrences within period - Using CQL

Implementation – Complete Picture

AGENDA

1.  Introduction

Oracle BAM: Architected for Integration and Visualization

Processing Twitter Stream with Oracle Event Processing (OEP) 19.11.2014

Oracle BAM – Create a Data Object

Oracle BAM Enterprise Message Source Configuration

Oracle BAM Composer

AGENDA

1.  Introduction

Architectural Pattern: Standalone Event Stream Processing

Event Processing (ESP / CEP)

State Store / Event Store En

Internet of Things

Social Media Streams

Analytical Applications

Business Rule Management

System Rules

Event Processing

Result Store

Hadoop Big Data Infrastructure

Architectural Pattern: Event Stream Processing as part of Lambda Architecture

State Store / Event Store

Internet of Things

Event Processing

Map/Reduce HDFS Result

Result Store

Hadoop Big Data Infrastructure

Architectural Pattern: Event Stream Processing as part of “Kappa” Architecture

Internet of Things

DB Ente

Event Processing

Replay HDFS

Result Store

Event Processing in modern architecture

Enterprise Applications

External Cloud Service Providers

BPM and SOA Platform

Business Logic/Rules

Data Analytics

Internet of Things

Event Processing

Mobile Apps

Rich (Web) Client Apps

Visualization

Biz Logic Rules

Business RuleManagement

System Rules

Processes ACM

HumanWF

Oracle Event Processing for Java Embedded

Coherence

Oracle NoSQL Business Activity Monitoring (BAM)

Oracle RDBMS Se

SOA Suite

BPM Suite

Oracle Rules

Oracle Event Processing (OEP) Oracle NoSQL Oracle RDBMS

Business Activity Monitoring (BAM)

Oracle Real-Time Decisions (RTD)

Questions and answers ...

BASEL BERN BRUGG LAUSANNE ZUERICH DUESSELDORF FRANKFURT A.M. FREIBURG I.BR. HAMBURG MUNICH STUTTGART VIENNA

Guido Schmutz

Technology Manager

guido.schmutz@trivadis.com

INFOBOX – Read and delete •  There are two versions of the last slide

available, one for the contact details of a speaker, and one for two or more speakers.

•  Name, title and location always underneath one another in one row (Shift+Return)

•  This idea is that this is the last slide (also for questions and answers) and is on the screen for a long time at the end of the presentation, so the viewers have the chance to write down the contact data J

Processing Twitter Events in Real-Time with Oracle Event Processing (OEP) 12c

Technology

Happenzi OEP

IPenable OEP

Avlsi Oep Pid

Berkeley NEST Wireless OEP

1972 Oep Steminfla

Oep for cng_ventures_july_7_vfinal2

Best Practices for XQuery processing in Oracle Database 12c

Oracle GoldenGate 12c - oracledwh.de€¦ · 1 Oracle...

EasyRent.com - Assignment #7 - OEP

CrowdKeys oep venturelab keynote01

Oep Introduction 2010 Eng

el€¦ · Alice el onderland s | Piet Heinstraat 2 |0 6992...

Jumpto goal oep

CrowdKeys OEP Partnership

Opal oep guidelines

Oep interim report