8/11/2019 Odtug Soa Pt Session
1/44
Tuning Your SOA Infrastructure forPerformance and Scalability
Vikas Anand
Senior Principal Product Manager [email protected]
mailto:[email protected]:[email protected]8/11/2019 Odtug Soa Pt Session
2/44
The following is intended to outline our generalproduct direction. It is intended for information
purposes only, and may not be incorporated into anycontract. It is not a commitment to deliver anymaterial, code, or functionality, and should not berelied upon in making purchasing decisions.
The development, release, and timing of anyfeatures or functionality described for Oraclesproducts remains at the sole discretion of Oracle.
8/11/2019 Odtug Soa Pt Session
3/44
Oracle Fusion MiddlewareComplete, Open, Integrated, Best-in-Class
DevelopmentTools
User Engagement
SocialWeb Mobile
EnterpriseManagement
IdentityManagement
Content Management
Business Intelligence
Business ProcessManagement
Service Integration
Data Integration
Cloud Application Foundation
8/11/2019 Odtug Soa Pt Session
4/44
So what does performance mean
a dramatic or musical entertainment ; "they listened to ten differentperformances"; "the play ran for 100 performances"; "the frequentperformances of the symphony testify to its popularity"the act of presenting a play or a piece of music or other entertainment ;"we congratulated him on his performance at the rehearsal"; "an inspiredperformance of Mozart's C minor concerto"the act of performing; of doing something successfully ; using knowledgeas distinguished from merely possessing it; "they criticised his performance asmayor"; "experience generally improves performance"any recognized accomplishment ; "they admired his performance understress"; "when Roger Maris powered four home runs in one game hisperformance merits awe"operation: process or manner of functioning or operating ; "the power of itsengine determines its operation"; "the plane's operation in high winds"; "theycompared the cooking performance of each oven"; "the jet's performance
conformed to high standards"
8/11/2019 Odtug Soa Pt Session
5/44
Performance SOA Context
Reliable - End to end transactionality i.e. no order or message loss Recovery of messages as and when required Automated transaction recovery and rollback capabilities A view of the transaction
Scalable - : Vertical vs horizontal scale out H/W, OS, JDK
Sizing DB, AQ, MQ, Messaging infrastructure, MiddlewareThroughput - :
Transactions-per-second (TPS) End to end transaction time Messages completed in a given time frame Throughput vs Response Time
8/11/2019 Odtug Soa Pt Session
6/44
Technology..!!!
8/11/2019 Odtug Soa Pt Session
7/44
SHARED SERVICES PLATFORMLow latency, high throughput, high scalability, very high availability, Secure
COMPOSITE APPLICATIONSHigh Concurrency, Long lived business processes, Very high reliability, Secure
Specialized Needs For Different
Architectural Layers
MAINFRAME
EventManagement
Real time latency,very high
throughput
8/11/2019 Odtug Soa Pt Session
8/44
TPSmsg/s
1,000sservices
Scalable foundation
ERP MAINFRAME SERVICES EVENTSPARTNERSDB
SERVICE BUS
Messaging pattern Multi-dimensional scaling Service virtualization:
Universal connectivity Protocol translation Location transparency Automatic pooling
COMPLEXEVENTPROCESSING
Eventing pattern Unstructured data Real-time patterns
identification
DATA INTEGRATORGOLDENGATE
Data integration pattern E-L-T Batch operation Bulk transfers Change Data Capture
(CDC)
8/11/2019 Odtug Soa Pt Session
9/44
11gR1 architecture
Unified Runtime
Policy Manager
Optimizedbinding
HumanWorkflow
(+AMX, AG,Orgn)
BusinessRules
Oracle Service Bus
Common JCA-based connectivity infrastructure
Repository
EM console+BPMN Screens
BAM
B2B
Mediator
Process Analytics
Proc Cubes
Process Core
BPEL BPMN
2010 Oracle Corporatio n
Optimized SOA runt ime Running on Application
Grid Coherence WLS JRockit
8/11/2019 Odtug Soa Pt Session
10/44
Typical SOA Deployments
1200 TPS Synchronous /Asynchronous Processes Large Architecture, and Development Teams
Large Telco Customer
Scaling to 1000 TPS from 100 /200 TPS currently 400 CPUsRetail Customer
300 TPS Millions of lines of Code
Air TrafficManagement customer
1 Million Events/Sec @ 69 [s] avg (CEP) Stock price increases or decreases by more than 2% percent
between 2 ticksFront Office Trading
40 million msg/day msg size = 5-10KB
(Credit card transactions)
Credit Card Company
8/11/2019 Odtug Soa Pt Session
11/44
Performance Integration use case
8/11/2019 Odtug Soa Pt Session
12/44
Design it Right !!!
8/11/2019 Odtug Soa Pt Session
13/44
Performance Anti Patterns Some
ExamplesDesign of the application(s)
Development anti-patterns
Sizing and Capacity Over and under sizing of components
Deployment Architecture
Data Retention Policy and Maintenance
8/11/2019 Odtug Soa Pt Session
14/44
Anti-Pattern Design of Applications
Usage of synchronous processes to call asynchronous processes andwaits for a response
May lighten the load on the server but adds a transaction
abstraction that needs to be handledNo view of the transaction need a mechanism to providelocation of the ongoing transaction (tracking)No server session affinity during callbacks in a cluster environmentTimeout issues, hanging threads and thread deadlocks
CPU, Garbage Collection and server threads usage will be highException and error handling may require customizationsScalability may be an issue which can only be resolved with adesign change
8/11/2019 Odtug Soa Pt Session
15/44
Anti-Patterns Synchronous-
Asynchronous
BPEL
BPEL
BPEL
BPEL
Timeout scenarios
8/11/2019 Odtug Soa Pt Session
16/44
Anti-Patterns Synchronous Fault
Handling
BPEL
Fault
Rollback?
Rollback?
Fault
F a u
l t
Handle Fault?
8/11/2019 Odtug Soa Pt Session
17/44
Anti-Patterns Design of Applications
Usage of extensiveloops
Using very big loops toconduct iterativeprocessing does notscale with largenumbers
Can lead to big JVMobject heaps andpotential GC bottlenecks
Abuse of FlowN(Batch/event paradigm)
Very large number of FlowN slows down server, largenumber of threadsHow to deal with long runningBPEL processes
8/11/2019 Odtug Soa Pt Session
18/44
Anti-Patterns Design of Applications
Over usage of asynchronous processes(Mediator/BPEL)
Adds dehydration overheadDevelopers are not aware of transactional boundarieswhich can quickly complicate the message lifecycle
Over usage of callbacks (chatty system) in a
single BPEL processEvery callback is a dehydration point which will causeDB activityHaving too many callback activities can quickly impact
performance
8/11/2019 Odtug Soa Pt Session
19/44
Anti-Pattern Asynchronous
BPEL
Mediator
BPEL
BPEL
BPEL BPEL
BPEL
BPEL
BPEL
BPEL
BPEL
8/11/2019 Odtug Soa Pt Session
20/44
Anti-Patterns Sizing and Capacity
Over sizing some components while notadding capacity for other components
Most notably adding more FMW servers while notadding any more DB capacityNot sizing the dehydration DB correctly, or notable to add disk at the right time (real
performance issue)Transaction systems not able to sustain load ofincoming requests leads to lost messages,system down scenarios, overall system slow
down
8/11/2019 Odtug Soa Pt Session
21/44
Anti-Patterns Deployment Architecture
Trying to overextend a clusters capabilityOne set of processes consume more resources
which impact other set of processes for e.g.OrderConsumption processes vsOrderProcessing processesOrderProcessing takes much longer however
OrderConsumption takes away CPU processingpower impacting overall performance
8/11/2019 Odtug Soa Pt Session
22/44
Anti-Patterns Deployment Architecture
8/11/2019 Odtug Soa Pt Session
23/44
Anti-Patterns Deployment Architecture
8/11/2019 Odtug Soa Pt Session
24/44
Hmm.. how much Data do Ireally need!!!
8/11/2019 Odtug Soa Pt Session
25/44
Purging and DB retention policies
DB purging not being optimizedHas one of the biggest impacts on a SOA business applicationImpacts performance of runtime SQL queries
Impacts asynchronous routing rules and asynchronous BPELprocessesSlows down console access when querying for auditing dataVery large overhead when purging data longer cycles required,defragmentation of DB, maintenance window
If retention periods are long, larger disk space is required BLOBcolumn tables grow taking up bigger chunk of the DB
8/11/2019 Odtug Soa Pt Session
26/44
Anti-Patterns - Transactions
BPEL
Mediator
Anti-Patterns Purging and DBRetention Policies
Impacts DB performance
Slows down console access
Longer purging cycles
8/11/2019 Odtug Soa Pt Session
27/44
Performance Tunings SOAInfrastructure Why Purging?
Large volume of stale data
Choices Loop Purging Parallel Purging Partitioning
Purging Strategies Small Installations
8/11/2019 Odtug Soa Pt Session
28/44
Perfect Now lets Tune itfor performance
8/11/2019 Odtug Soa Pt Session
29/44
Performance Tunings Overview
BPEL/BPMN: Threadtuning, Audit-Trail,dehydration, messagepersistence, Stats
Mediator : Audit Level,Worker Threads
Adapters: Threads, batchsize
SOA Component Level
OS JVM Database Weblogic Application
Server
Generic
Audit Level Payload Validation Purging
SOA Infrastructure
++
8/11/2019 Odtug Soa Pt Session
30/44
Performance Tunings BPEL/BPMNBPEL/BPMN Engine Level Properties
8/11/2019 Odtug Soa Pt Session
31/44
Performance TuningsBPEL/BPMN Engine Level Properties
# of Threads to process invocation messages (dspInvokeThreads) # of Threads to process engine messages (dspEngineThreads) # of Threads to process system messages dspSystem Threads
Concurrency
Audit Trail logging levels (AuditLevel) Message Persistence (oneWayDeliveryPolicy)Memory / DB
XML Validations (validateXML) Statistics for most recently processed requests (statsLastN)CPU
8/11/2019 Odtug Soa Pt Session
32/44
Performance Tunings BPELBPEL Process Level Properties
8/11/2019 Odtug Soa Pt Session
33/44
Performance Tunings - BPELBPEL Process Level Properties
Parallel processing of invole in multiplebranches (nonBlockingInvoke)
Synchronous / Transient process design
ResponseTime
Dehydration needed or not(inMemoryOptimization)
Instance data storage in DB(completionPersistPolicy)
Database
Payload Validation (validateXML)CPU
8/11/2019 Odtug Soa Pt Session
34/44
Performance Tunings Mediator Mediator Engine Level Properties
8/11/2019 Odtug Soa Pt Session
35/44
Performance Tunings Mediator Mediator Level Properties
Parallel Routing # of parallel threads for message processing
Resequencer # of worker threads
Concurrency
Audit Level Parallel Routing
# of rows that are fetched by each thread per iteration Resequencer
# of Groups that are locked by each thread at a time
Database
Metrics Level Sleep intervals before next pollingCPU
8/11/2019 Odtug Soa Pt Session
36/44
Performance Tunings AdaptersDB Adapter Properties
8/11/2019 Odtug Soa Pt Session
37/44
Performance Tunings - Adapters
# of worker / Poller / Dequeue threadsConcurrency
# of rows to be processed per transaction (MaxTransactionSize) # of file records to be processed (MaxRaiseSize) Collect stats on relevant tables Large files
Database
Process design IO # Threads at the application server level
CPU
Adapter Level Properties
8/11/2019 Odtug Soa Pt Session
38/44
Performance Tunings SOA InfrastructureSOA Infra Properties
CA
8/11/2019 Odtug Soa Pt Session
39/44
Audit Level PurgingDatabase
Payload ValidationCPU
SOA InfrastructurePerformance Tunings SOA Infrastructure
8/11/2019 Odtug Soa Pt Session
40/44
Performance Tunings Generic
Production Mode Connection Pooling Logging Self tuning
Weblogic
Heap Size Nursery Size GC Algorithm Use Large pages 64 bit vs 32 bit
JVM
SOA & Application Schema tuning Tuning Parameters Redo Logs
Database
8/11/2019 Odtug Soa Pt Session
41/44
Performance Tools Monitoring, Profiling and Debugging
Choice oftool
dependson:
Which componentsbeing monitored,which metrics
Monitoring, Profiling,or Debugging
Profiling
JRockit MissionControl (JRA cpu
profiling, Latency Analyzer, MemoryLeak Detector)
Monitoring
Enterprise Manager,WebLogic Server
Admin Console,DMS Spy,WebLogicDiagnosticFramework
Debugging
Enterprise Manager(Audit Trailenabled), ServerLogs
Others
AWR, DB EnterpriseManager
8/11/2019 Odtug Soa Pt Session
42/44
Summary..
Understand the requirements and the UseCases
Choose the right TechnologySize it !!!Define the right design Patterns
Deploy the right way !!!Leverage the Knobs to Tune
And Thank You For attending our session
8/11/2019 Odtug Soa Pt Session
43/44
For More Information
search.oracle.com
or
oracle.com
8/11/2019 Odtug Soa Pt Session
44/44