2013 © Trivadis
BASEL BERN BRUGG LAUSANNE ZUERICH DUESSELDORF FRANKFURT A.M. FREIBURG I.BR. HAMBURG MUNICH STUTTGART VIENNA
2013 © Trivadis
Echtzeitverarbeitung von Twitter Streams mit Oracle Event Processing (OEP) 12c
Guido Schmutz
DOAG Konferenz 2014
19.11.2014 – 15:00 Raum Kiev
19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP)
1
INFOBOX – Read and delete • A heading and an optional sub-heading
can be placed on the first slide. • The title is written directly under the
name (Shift+Return) • If multiple speakers are named, please
just write the names one underneath the other (there is normally no space for titles, etc.)
2013 © Trivadis
Guido Schmutz
• Working for Trivadis for more than 17 years
• Oracle ACE Director for Fusion Middleware and SOA • Co-Author of different books • Consultant, Trainer Software Architect for Java, Oracle, SOA and
Big Data / Fast Data • Member of Trivadis Architecture Board • Technology Manager @ Trivadis
• More than 25 years of software development experience
• Contact: [email protected] • Blog: http://guidoschmutz.wordpress.com • Twitter: gschmutz
19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP)
2
2013 © Trivadis
AGENDA
1. Introduction
2. Twitter Use Case
3. Processing with Oracle Event Processing (OEP)
4. Visualization with Oracle Business Activity Monitoring (BAM)
5. Event Processing in Information Architecture
19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP)
3
INFOBOX – Read and delete • If the agenda is used as an interim
page, please highlight the relevant chapter in red font.
• To allow optimum alignment of objects,
display the drawing guides (right-click on the page)
2013 © Trivadis
Big Data Definition (4 Vs)
19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP)
4
+ Time to action ? – Big Data + Event Processing = Fast Data
Characteristics of Big Data: Its Volume, Velocity and Variety in combination
2013 © Trivadis
The world is changing …
The model of Generating/Consuming Data has changed ….
Old Model: few companies are generating data, all others are consuming data
New Model: all of use are generating data, and all of us are consuming data
19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP)
5
2013 © Trivadis
19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP)
6
2013 © Trivadis
Internet Of Things – Sensors are/will be everywhere
There are more devices tapping into the internet than people on earth
How do we prepare our systems/architecture for the future?
19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP)
7 Source: Cisco Source: The Economist
2013 © Trivadis
AGENDA
1. Introduction
2. Twitter Use Case
3. Processing with Oracle Event Processing (OEP)
4. Visualization with Oracle Business Activity Monitoring (BAM)
5. Event Processing in Information Architecture
19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP)
8
2013 © Trivadis
Retrieve Tweets and Visualize
19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP)
9
2013 © Trivadis
Access to Tweets
19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP)
10
Quelle
Source Limitations Cost Twitter’s Search API 3200 / user
5000 / keyword 180 requests / 15 minutes
free
Twitter’s Streaming API 1%-40% of total volume free
DataSift none 0.15 -0.20$ /
unit Gnip none On request
2013 © Trivadis
AGENDA
1. Introduction
2. Twitter Use Case
3. Processing with Oracle Event Processing (OEP)
4. Visualization with Oracle Business Activity Monitoring (BAM)
5. Event Processing in Information Architecture
19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP)
11
2013 © Trivadis
Oracle Event Processing (OEP) - Engine
Lightweight Java Application Server
• Full environment for running Java applications
• Module Framework - OSGi
High Throughput
• Hundreds of thousands of events/second
Event Processing Infrastructure
Easy-to-use development environment
• Service Framework – Spring DM, POJO
Multiple-choice VM
• JRockit or WebLogic RealTime
19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP)
12
2013 © Trivadis
Oracle Event Processing Application
Adapter • Provide a way for EPN to exchange data with components that are external to network • Either an Event Source (inbound) or an Event Sink (outbound)
Processor • Contain query code written in CQL and consumes event data from a preceding
channel • May also generate new events to an output channel • CQL is an extension of SQL
19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP)
13
2013 © Trivadis
Oracle Event Processing Application
Bean Bean
• Provide a place for logic written in Java • Java code can receive events (Event Sink) and/or send events (Event Source)
• Are implemented using the Spring Framework model or the OEP event bean model Channel
• Two types of channel exist: Stream and Relation
CQL Processor • Executes Continous Query Language (CQL) queries • Used to declaratively describe logic (queries) to be applied on the event stream
19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP)
14
2013 © Trivadis
Oracle Event Processing - CQL
CQL is an extension to Structured Query Language (SQL) • Same keywords and syntax rules, but additional support for streaming data
event conceptually corresponds to a row in a database table, but • one event is always before or after another in time
• stream is potentially infinite and ever-changing
19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP)
15
Data Query
Result Tuples Result Tuples
Traditional Database CEP Engine
QueryData
2013 © Trivadis
Oracle Event Processing – Visual Development Tools
19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP)
16
2013 © Trivadis
Oracle Event Processing – Operation & Management
19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP)
17
2013 © Trivadis
Implementation – complete picture
19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP)
18
Mention Extractor
Twitter Adapter
CounterProcessor
Hashtag Extractor
Author Extractor
BAM Tweet
BAM Counter
Working on my OEP & BAM demo for the session
”Echtzeitverarbeitung von Twitter Feeds …” #doag2014
#oep12c #bam12c http://bit.ly/1xafUZJ
gschmutz
#doag2014 #oep12
#bam12c,2
gschmutz,1
@doag2014,5 #oep12c,1 JMS
JMS Twitter
range 30 seconds slide 30 seconds
Sentiment Enricher
Working on my OEP & BAM demo for the session ”Echtzeitverarbeitung von Twitter Feeds …” #doag2014 #oep12c
#bam12c http://bit.ly/1xafUZJ
1
#bam12c
2013 © Trivadis
1) Creating a Twitter Adapter
19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP)
19
Twitter Adapter
Working on my OEP & BAM demo for the session
”Echtzeitverarbeitung von Twitter Feeds …” #doag2014
#oep12c #bam12c http://bit.ly/1xafUZJ
2013 © Trivadis
2) Send Tweets to BAM
19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP)
20
Twitter Adapter
BAM Tweet
Working on my OEP & BAM demo for the session
”Echtzeitverarbeitung von Twitter Feeds …” #doag2014
#oep12c #bam12c http://bit.ly/1xafUZJ
JMS Twitter Sentiment Enricher
Working on my OEP & BAM demo for the session ”Echtzeitverarbeitung von Twitter Feeds …” #doag2014 #oep12c
#bam12c http://bit.ly/1xafUZJ
1
2013 © Trivadis
3) Extract interesting information from Tweet
19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP)
21
Mention Extractor
Twitter Adapter
Hashtag Extractor
Author Extractor
BAM Tweet
Working on my OEP & BAM demo for the session
”Echtzeitverarbeitung von Twitter Feeds …” #doag2014
#oep12c #bam12c http://bit.ly/1xafUZJ
gschmutz
#doag2014 #oep12
JMS Twitter Sentiment Enricher
Working on my OEP & BAM demo for the session ”Echtzeitverarbeitung von Twitter Feeds …” #doag2014 #oep12c
#bam12c http://bit.ly/1xafUZJ
1
#bam12c
2013 © Trivadis
4) Count occurrences within period
19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP)
22
Mention Extractor
Twitter Adapter
CounterProcessor
Hashtag Extractor
Author Extractor
BAM Tweet
BAM Counter
Working on my OEP & BAM demo for the session
”Echtzeitverarbeitung von Twitter Feeds …” #doag2014
#oep12c #bam12c http://bit.ly/1xafUZJ
gschmutz
#doag2014 #oep12
#bam12c,2
gschmutz,1
@doag2014,5 #oep12c,1 JMS
JMS Twitter
range 30 seconds slide 30 seconds
Sentiment Enricher
Working on my OEP & BAM demo for the session ”Echtzeitverarbeitung von Twitter Feeds …” #doag2014 #oep12c
#bam12c http://bit.ly/1xafUZJ
1
#bam12c
2013 © Trivadis
Implementing in Oracle Event Processing
19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP)
23
Mention Extractor
Twitter Adapter
CounterProcessor
Hashtag Extractor
Author Extractor
BAM Tweet
BAM Counter
JMS
JMS Twitter
range 30 seconds slide 30 seconds
Sentiment Enricher
2013 © Trivadis
1) Creating Twitter Adapter – Connecting to Twitter Stream
19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP)
24
2013 © Trivadis
1) Creating Twitter Adapter – Tweet Event
19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP)
25
2013 © Trivadis
1) Creating Twitter Adapter – Adapter Factory
19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP)
26
2013 © Trivadis
1) Creating Twitter Adapter – Assembly
19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP)
27
2013 © Trivadis
1) Creating Twitter Adapter – Export Adapter to server
19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP)
28
2013 © Trivadis
1) Creating Twitter Adapter – Using Twitter Adapter
19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP)
29
2013 © Trivadis
2) Sentiment Enrichment
19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP)
30
2013 © Trivadis
3) Sending Tweets to BAM
Using Oracle BAM Enterprise Message Sources (JMS) interface
19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP)
31
2013 © Trivadis
19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP)
32
3) Sending Tweets to BAM – Convert event to JMS MapMessage
2013 © Trivadis
4) Extract information from Tweet – Extract Hashtags from TweetEvent
19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP)
33
2013 © Trivadis
4) Extract information from Tweet – Extract Hashtags from TweetEvent
19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP)
34
2013 © Trivadis
5) Count hashtag occurrences within period - Using CQL
19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP)
35
2013 © Trivadis
Implementation – Complete Picture
19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP)
36
2013 © Trivadis
AGENDA
1. Introduction
2. Twitter Use Case
3. Processing with Oracle Event Processing (OEP)
4. Visualization with Oracle Business Activity Monitoring (BAM)
5. Event Processing in Information Architecture
19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP)
37
2013 © Trivadis
Oracle BAM: Architected for Integration and Visualization
Processing Twitter Stream with Oracle Event Processing (OEP) 19.11.2014
38
2013 © Trivadis
Oracle BAM – Create a Data Object
19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP)
39
2013 © Trivadis
Oracle BAM Enterprise Message Source Configuration
19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP)
40
2013 © Trivadis
Oracle BAM Composer
19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP)
41
2013 © Trivadis
AGENDA
1. Introduction
2. Twitter Use Case
3. Processing with Oracle Event Processing (OEP)
4. Visualization with Oracle Business Activity Monitoring (BAM)
5. Event Processing in Information Architecture
19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP)
42
2013 © Trivadis
Architectural Pattern: Standalone Event Stream Processing
19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP)
43 43
Event Processing (ESP / CEP)
State Store / Event Store En
terp
rise
Even
t Bus
(In
gres
s)
Even
t Cl
oud
Internet of Things
Social Media Streams
Ente
rpris
e Ev
ent B
us
43
Analytical Applications
DB
Ente
rpris
e Se
rvic
e Bu
s
Business Rule Management
System Rules
Event Processing
Result Store
2013 © Trivadis
Hadoop Big Data Infrastructure
Architectural Pattern: Event Stream Processing as part of Lambda Architecture
19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP)
44 44
Event Processing (ESP / CEP)
State Store / Event Store
Ente
rpris
e Ev
ent B
us
(Ingr
ess)
Even
t Cl
oud
Internet of Things
Social Media Streams
Ente
rpris
e Ev
ent B
us
44
Analytical Applications
DB
Ente
rpris
e Se
rvic
e Bu
s
Event Processing
Map/Reduce HDFS Result
Store
Result Store
2013 © Trivadis
Hadoop Big Data Infrastructure
Architectural Pattern: Event Stream Processing as part of “Kappa” Architecture
19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP)
45 45
Event Processing (ESP / CEP)
State Store / Event Store
Ente
rpris
e Ev
ent B
us
(Ingr
ess)
Even
t Cl
oud
Internet of Things
Social Media Streams
45
Analytical Applications
DB Ente
rpris
e Se
rvic
e Bu
s
Event Processing
Replay HDFS
Result Store
2013 © Trivadis
Even
t Cl
oud
Event Processing in modern architecture
19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP)
46
Enterprise Applications
WS
REST
JMS
RDMBS
Loca
l ESB
External Cloud Service Providers
Ente
rpris
e Se
rvic
e Bu
s (E
SB)
EJB
Event Processing (ESP / CEP)
State Store / Event Store
BPM and SOA Platform
Event
REST
Business Logic/Rules
NoSQL
Analytical Applications
Data Analytics
Internet of Things
Event Processing
46
Mobile Apps
DB
Rich (Web) Client Apps
DB
Social Media Streams
Ente
rpris
e Ev
ent B
us
(Ingr
ess)
Ente
rpris
e Ev
ent B
us
Visualization
Biz Logic Rules
WS
Event
Business RuleManagement
System Rules
Processes ACM
HumanWF
Oracle Event Processing for Java Embedded
Coherence
Oracle NoSQL Business Activity Monitoring (BAM)
Oracle RDBMS Se
rvic
e Bu
s
SOA Suite
BPM Suite
Web
Logi
c JM
S
Web
Logi
c JM
S ED
N
Oracle Rules
Oracle Event Processing (OEP) Oracle NoSQL Oracle RDBMS
Business Activity Monitoring (BAM)
API G
atew
ay
EDN
Oracle Real-Time Decisions (RTD)
2013 © Trivadis
Questions and answers ...
2013 © Trivadis
BASEL BERN BRUGG LAUSANNE ZUERICH DUESSELDORF FRANKFURT A.M. FREIBURG I.BR. HAMBURG MUNICH STUTTGART VIENNA
Guido Schmutz
Technology Manager
19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP)
INFOBOX – Read and delete • There are two versions of the last slide
available, one for the contact details of a speaker, and one for two or more speakers.
• Name, title and location always underneath one another in one row (Shift+Return)
• This idea is that this is the last slide (also for questions and answers) and is on the screen for a long time at the end of the presentation, so the viewers have the chance to write down the contact data J
47