Soa12c launch 5 event processing shmakov eng cr
Post on 31-May-2015
96 Views
Preview:
DESCRIPTION
Transcript
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 1
SOA Suite 12.1.3 Field Enablement
Oracle Event Processing Enabling Fast Data and The Internet of Things Technical Presentation
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 2
Modern Computing Challenges
Solution Product Overview
Oracle Event Processing Applications
Continuous Query Language (CQL)
Internet of Things (IoT)
What is Fast Data?
12c Features
Agenda
Oracle Event Processing 12c Sales Enablement
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 3
The application evaluates if the roaming asset (device) is within a certain
distance of the business. If it is, the system will generate an event to send the
customer a promotional message.
This powerful example encompasses the integration of JMS, Real Time Spatial
Analysis and Big Data integration
Mobile Marketing - OEP in Action: DEMO
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 4
Modern Computing Challenges
Does the problem you want to solve have any one or more of the following
conditions?
Requires high-throughput and low-latency processing
Continuously streaming data (Log Files, converting Batch to Real Time)
Real-time correlation between multiple incoming data sources
Time-sensitive alerts, aggregations and calculations
Needs to look for patterns in the data stream (Spatial Analysis)
Identify when some event(s) should have happened and did not.
Data does not need to be stored, if there is nothing of interest in it
Problem is more easily solved by analyzing before storing in DB
Conditions for Event Processing Opportunities
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 5
Solution Product Overview
Solution involves one or more of the following:
High Volume
Continuous Streaming
Sub-Millisecond Latency
Disparate Sources
Time-Window Processing
Pattern Matching
Business Event Visualization
Oracle Event Processing
OEP
Streaming
Event Data
Alerts,
Actions
Filtering,
Pattern Matching,
Missing Events,
Aggregations,
Correlations,
Calculations,
Geo-Spatial
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 6
Solution Product Overview
Inverted Database
Data is ‘static’
Queries are ‘dynamic’
• Data (event) is ‘dynamic’
• Queries are ‘static’
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 7
Oracle Event Processing
POJO
App Frameworks
Spring Services
Stream Management
Core Event Infrastructure
Complex Event Processor
Real-Time Kernel
Data Cartridges
Extended Event Infrastructure
Cluster Management HTTP Pub/Sub Engine
Event Repository
Coherence
Foundation Services
Security Logging
Processes Streaming Data In-Memory & in
Real-Time (274 OSGi Bundles)
Event Analysis: Pattern Matching, Missing
Events, Aggregations, Correlations,
Calculations, Coherence Integration,
Spatial Functions, Big Data Integration
Performs Aggregations and Correlates
Multiple Streams of Data
Integrates easily with other Oracle
products: Big Data (Hadoop & NoSQL) –
FAST DATA, SOA (EDN), Coherence,
Spatial, RTD, etc.
Str
ea
min
g E
ve
nt D
ata
Alerts, Actions
Solution Overview: Oracle Event Processing (OEP)
OSGi
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 8
Solution Overview
Filtering, Correlation & Aggregation
New stream filtered for specific criteria, e.g. stock price > $22
Process events from different streams together
Scrolling, time-based window metrics, e.g. average # of stock trades in
the last hour
Detect Absence of Events and Missing Events
Event “A” NOT followed by Event “B” within 10 minutes
Event “A”, Event “B” should occur next, but Event “C” occurs instead
Oracle Event Processing
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 9
Perform Calculations in Real-Time
Perform Calculations on Data From Sensors:
Use last X seconds or minutes of data OR
Use last X data points
Smooth out “noise” in the data
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 10
Input
Adapter Channel
Input
Adapter Channel
Business
Logic (CQL)
Channel
Channel
Channel
Output
Adapter
Output
Adapter
Oracle Event Processing Application
DB
Input adapters connect to data sources Channels help control the flow of data and can be tuned for optimal performance Databases and Coherence caches can be referenced directly in CQL processors CQL processors contain filtering, correlation, aggregation and pattern matching business logic Output adapters send data and alerts to downstream systems and business processes
Coherence
Business
Logic (CQL)
Business
Logic (CQL)
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 11
High-Performance In-Memory Data Processing
Input
Adapter Data
Input
Adapter Data
Channel Business
Logic (CQL)
Channel
Data
Data
Analytics
Channel Business
Logic (CQL)
Enrich Output
Adapter Data Data Data Data
Analytics: Continuously Sliding Time Windows of Streaming Data, Filtering,
Correlations, Calculations, Aggregations, Pattern Matching, Missing Event Detection,
Spatial Analysis, etc.
Enrichment: Integrate with data from DB, Coherence, NoSQL, Hadoop etc.
Oracle Event Processing
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 12
Administration & Monitoring
View the data flow of the application
“Event Processing Network” (EPN)
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 13
Administration & Monitoring
Monitor throughput and latency between any two nodes in
the application
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 14
Oracle Event Processing & Coherence
Challenges Benefits
Handle and correlate events in real-
time, including support for multiple
patterns:
• Pre-processing (buffer OEP)
• Within OEP (to cache reference data)
• Post OEP (to expose processed events to
consuming apps)
• High throughput for storing data
• Aggregation and event querying
• Pattern implementation flexibility
combining two complementary
technologies
Data
Grid
OEP
Consolidated
& in-context
Data
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 15
Simple Filtering
SELECT *
FROM inputChannel [NOW]
WHERE eventValue > 10
Continuously calculate the last hour sales by store
SELECT SUM(amount) as salesTotal, storeID
FROM inputChannel [range 60 minutes]
GROUP BY storeID
Calculate the average of the last 2 stock ticks by stock symbol
SELECT AVG(stockPrice) as avgPrice, stockSymbol
FROM inputChannel [PARTITION BY stockSymbol ROWS 2]
GROUP BY stockSymbol
Sample CQL Queries
Filter for events meeting specific
threshold values
Running total of up- to-the-
moment sales by store
Average of the last 2 stock ticks
by symbol
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 16
CQL view that joins a cache containing
reference data to an individual streaming
event to provide additional context for further
processing.
Sample CQL Queries
Cache Join Query
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 17
Pattern Matching
Powerful concept that allows identification of complex event patterns
Defined as regular expressions
PATTERN (X+ Y+)
1 or more X events … … followed by 1 or
more Y events
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 18
CQL view filters events for use by a query.
CQL query that starts when an OFF event
occurs and waits 10 seconds to make sure
that it is not followed by an ON event before
sending the OFF event downstream.
Pattern Match Query
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 19
Find passengers stuck in security when their flight reaches “final boarding”.
SELECT
stuck.reservationLocator,
'STUCK' as state
FROM PassengerStateEventChannel MATCH_RECOGNIZE (
PARTITION BY
reservationLocator
MEASURES
Entered.reservationLocator AS reservationLocator
PATTERN (CheckIn Entered NotExited*? Final)
DEFINE
CheckIn AS state = 'CHECKIN',
Entered AS state = 'ENTERED',
NotExited AS state != 'EXITED',
Final AS state = 'FINAL'
) AS stuck
Sample CQL Queries
Find passengers who are stuck in
security when their flight is in the
“FINAL BOARDING” process.
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 20
Flight is in final boarding!
Find passengers stuck in
security.
Pattern Matching
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 21
Find users with no status update for 4 hours
SELECT
M.userWithNoStatus
FROM StatusEventChannel MATCH_RECOGNIZE (
PARTITION BY
user
MEASURES
user AS userwithNoStatus
INCLUDE TIMER EVENTS
PATTERN (A B)
DURATION 4 HOURS
DEFINE
A AS A.user is not null
) AS M
Sample CQL Queries
Missing Events: Finds users who report
a status, but fail to report another
status during the next 4 hours.
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 22
A market leader in security and also the innovation leader in
their industry that continuously offer to its customer’s new Smart
Home products and services.
An inventory and anti-jaming system connected through 3G
network to the alarm reception system, devices send keep Alive
(KA) signal in real-time providing Inventory system data and
manage device status (discovered, available, unavailable, error)
through the reception of the KA events. All this will be
connected to business applications and CRM.
Greater real-time insight into customer security assets and
immediate detection of competitive equipment tampering
ensuring customer retention and satisfaction.
Home security Alarm system equipment disruption monitoring
CO
MP
AN
Y
SO
LU
TIO
N
BE
NE
FIT
S
Deal:
• Oracle Event Processing
• Coherence
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 24
Real Time Tracking via
GPS and resource
evaluation in relation to
virtual geographical areas
Real Time Dynamic
definition of virtual
geographical complex
(polygon) areas
Real-Time Spatial Geo-Fencing Solutions
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 25
Track school buses on a map
SELECT bus.busId as busId, bus.seq as seq,
com.oracle.cep.cartridge.spatial.Geometry.createPoint(8307, bus.longitude,
bus.latitude) as geom
FROM BusPosStream as bus
Alert when the bus arrives at the bus stop
SELECT systimestamp() as incidentTime, bus.busId as busId, busstop.seq as stopSeq
FROM BusPosGeomStream[NOW] as bus, BusStopRelation as busstop
WHERE CONTAIN@spatial(busstop.geom, bus.geom, 100.0d) = true and bus.busId =
busstop.busId
Sample CQL Queries
Track a bus on a map
Determine when the bus is near
the bus stop
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 26
Point-in-Polygon Matching OEP detects entry into a pre-defined area
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 27
Nearest Neighbor OEP can find the nearest place of interest (e.g. WiFi Hotspot)
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 28
Dynamic Query Generation Business Users Modify Processing in Real-Time
Simple forms can be used to allow users to
add, modify or delete CQL queries at run-
time without stopping the application or
the server allowing existing processing to
continue running while business users
dynamically make modifications.
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 29
Dynamic Query Generation
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 30
Internet of Things (IoT)
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 31
Solution Overview: Oracle Event Processing (OEP)
Architecture designed and Optimized for Java
Enabled Devices/Gateways (197 OSGi Bundles)
High-speed real-time device data capture and
analysis helps mitigate the risk of down-time and
helps with continuous process improvement
Enables local real-time process monitoring to
detect degradation of performance, thus helping
with initiation of alerts or requests for operations
and maintenance actions
Significantly reduces noise to signal ratio of data
reaching the server, enabling savings in
bandwidth and scalability costs
Faster time to market for devices and easy over-
the-air updates post-deployment. OSGi
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 32
Oracle Event Processing for Java Embedded
Runs on devices capable of running
Java SE Embedded
Key component of Internet of Things
(IoT)
Smaller, lighter-weight version of the
same OEP product
No need to learn new skills to
program application for devices
OEP
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 33
OEP
OEP
OEP
OEP
OEP
OEP in Device to Data Center Deployments Real-time local data analysis for real-world event data
• Upstream nodes
perform basic filtering
and aggregation
• Larger servers
downstream perform
complex combining and
correlation across
multiple streams
• OEP in embedded
devices allows initial
processing to be
handled by less
powerful devices near
the origin/edge.
Event Flow
Edge devices Servers Gateways Enterprise Apps
OEP
OEP
OEP
OEP
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 34
Oracle Fast Data for Big Data
Fast Data Big Data Infrastructure
Oracle BI Foundation Suite
Oracle Real-Time Decisions
Endeca Information Discovery
Business Analytics
Oracle Event Processing Oracle Big Data
Connectors
Oracle Data Integrator
Oracle
Advanced
Analytics
Oracle
Database
Oracle
Spatial
& Graph
Apache Flume
Oracle GoldenGate
Oracle
NoSQL
Database
Cloudera
Hadoop
Oracle R
Distribution
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 35
1. Big data ≠ Infinite storage Yes, storage is cheap but it helps to have clean data, with context and less redundancy
2. Hadoop is batch-oriented and there is inherent latency "With the paths that go through Hadoop [at Yahoo!], the latency is about fifteen minutes […] it will never be true real-time. " * Raymie Stata, Yahoo! CTO (June 2011)
Some Challenges Working with Big Data
: http://www.theregister.co.uk/2011/06/30/yahoo_hadoop_and_realtime/
minutes
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 36
1. Filter out noise (ex: data ticks with no
change), add context (by correlating
multiple sources), increase relevance
1. Identify certain critical conditions as you
insert data into the warehouse
Move time-critical analysis to front of process
Get Ahead of the Curve
Filter out,
correlate
Use Event Processing Techniques
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 37
Fast Data Getting Ahead of the Curve
Big Data
minutes ms
Fast Data
His
toric
al d
ep
th: d
ee
p
His
toric
al d
ep
th:
sh
allo
w
Example:
analysis of traffic
patterns and
congestion times
for urban
planning
Example:
monitoring of traffic
cameras to ensure
given license plates
are not in use on
multiple vehicles
Add “depth” to your fast data by
merging output of MapReduce
to stream processing
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 38
COMPANY OVERVIEW
• Motorola Solutions connects people through technology • Motorola Solutions serves both enterprise and government
customers with core markets in public safety government agencies and commercial enterprises
• Global presence with 23,000 employees worldwide in 65 countries with sales in over 100 countries
CHALLENGES/OPPORTUNITIES
• Provide a Fast Data intelligence layer for Big data streaming video feeds • Event Processing analyzes Real Time Video meta-data for
distinct patterns of interest to Government agencies and law enforcement
• Centralized Exalogic implementation
PROJECT OBJECTIVES
• Projects worldwide using a reusable scalable
architecture supporting 2800 or more cameras
• Streaming video meta data interfaces with
IOmniscient processing for Face recognition and
License plate monitoring
• Event patterns for duplicate plates within
temporal period beyond distance capabilities
• Speeding analysis with driver recognition
RESULTS
• Ongoing partner relationship delivering
planned projects in Mexico, Oman
• Prototyping successfully completed with live
video feed data
• Foundational technology for Smart Cities
platform solutions
How Much Data Will Humans Create & Store
This Year (2011)? 1.8 zettabytes being created and replicated (Streaming Video)
57.5 billion 32 GB iPads, How much is that? About $34.4 trillion worth.
http://mashable.com/2011/06/28/data-infographic/
Motorola Smart IPVS : Vehicle Capacity & Flow Control
WARNING:
Suspect License
plate – CABO .
MEXICO CITY
WARNING: Driver
Criminal Record
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 39
JDeveloper IDE for OEP 12c
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 40
JDeveloper IDE for OEP 12c
• Create Event Processing Network • Graphically develop an OEP application
• Drag EPN components to canvas and configure
• Deploy to Configured OEP Servers • Define OEP Servers and Deployment Profiles
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 41
Improved Installer
• Installer and Domain Creation
Wizard Improvements
• Mac OS X is supported for
development.
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 42
New and Improved Adapters New File Adapter
• Doesn’t require the “Load
Generator Utility”.
• Easily configure all the required
parameters when defining the
adapter in Jdev.
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 43
CQL Enhancements Variable Duration For Pattern Match Queries
Duration for the pattern
match query can be
specific to each event.
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 44
CQL Pattern Templates Starts the Query Logic for the Developer
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 45
List of 12c Enhancements • CQL enhancements
• Spatial Improvements
• Application compiler
• Application and CQL testing
• New and Improved adapters
• Public Cartridge API
• Improved Installer
• Improved Coherence Cartridge
• Support for JDeveloper
• EDN integration
• Business Rules integration
• Performance optimizations
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 46
Request
Request
Event Data Event Data
SOA Composite
Instances
OEP Application
SOA
EPN
EDN Integration SOA Composite vs. Event Processing Network
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 47
Customer1
$600
Customer1
$600 Customer1
$700
> $1000 per
customer in 24 hours
Customer1
$700
EDN Integration
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 48
DEMO
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 49
Oracle Event Processing (OEP) Summary High-Volume Low-Latency Event Processing Infrastructure
Event Processing Network (EPN)
Light-weight Java Application Server (embeddable)
Easily Customizable
Integrate with existing infrastructure and other Oracle Products (e.g.
Coherence, Business Activity Monitoring, Database, Big Data
Appliance, Data Mining, Spatial, NoSQL Database etc.)
Time Management & Pattern Matching
Continuously Perform Calculations Over Time Windows
Partition Event Streams By Key Values
Perform Complex Pattern Matching, Detect Missing Events
Adjust Core Business Logic in Real-time without Redeploying
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 50
Fast Data Business Integrator and User Tooling
• Fast exploration of Real
Time Streams
• Fast definition of Real
Time Event Patterns
• Fast Testing and
Deployment of Projects
• Fast Export to
Development Environment
for more elaborate
prototyping
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 51
Summary: When To Use OEP
1 Business Logic Layer for Event-Driven/Coherence Applications
2 High-Volume Business Activity Monitoring Applications
3 Real-Time Spatial Applications
4 Fast Data: Real-Time Requirements with Big Data Infrastructure
5 High Volume Batch to Real-Time Conversion Projects
6 Internet of Things (IoT): Processing Data On and From Devices
7 Pattern Matching / Missing Events / Alerting / Fraud Detection
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 52
top related