Salvisberg AQ Design for Best Performance 02

Basel · Baden · Berne · Lausanne · Zurich · Düsseldorf · Frankfurt/M. · Freiburg i. Br. · Hamburg · Munich · Stuttgart · Vienna

AQ Design for Best Performance

Philipp SalvisbergTechnology ManagerApplication Performance [email protected]

DOAG ConferenceNürnberg, 1st December 2008

mailto:[email protected]

© 2008AQ Design for Best Performance 2

Who am I?

Technology Manager APM and partner at Trivadis [email protected]

Head of the

Main focus on database centric development with Oracle DBApplication Performance Management (APM)Application DevelopmentBusiness Intelligence

20 years experience in using Oracle products

mailto:[email protected]

http://www.trivadis.com/

© 2008

Application Performance Management Context

AQ Design for Best Performance 3

Middleware

Integration(ESB, EIB

Data Access)

Presentation

Resources

Business Logic

Frontend

Thin Client

Rich Client

Fat ClientBu

sine

ss L

ogic

Pres

enta

tion

Pres

enta

tion

Files

Databases

Services

Busi

ness

Log

ic

© 2008

Introduction

Queue Types

Case Study

Benchmarks

Recommendations

Conclusion

4AQ Design for Best Performance

Agenda

Bildmarkierung

TextmarkierungSchrift VerdanaGrösse 16Farbe weiss

Daten sindimmer im Spiel.Comprehensiveconsultancy isthe through ballto yourperformance.

© 2008

What is Advanced Queuing?


Simply put, it‘s enqueue…

… and dequeue

© 2008

Decoupling systems, components through asynchronous processing

Designing workflows as a sequence of services where each servicepasses data as messages via dedicated queues to subsequent services

Concurrent execution of subsequent services to increase overallthroughput

Exchange messages between other queuesContent basedLocal / remoteOracle AQ, IBM WebSphere MQ, TIBCO Rendezvous

Visualize process state and increase stabilityUsing Transactions (local or XA)Defined restart procedure after a system crash

What is Advanced Queuing good for?


© 2008

Examples:

Applications based on an ESB (SOA, BPEL, …)

Data Replication with Oracle Streams

Oracle Scheduler

Avaloq Banking System

Trivadis Output Management Service (TOMAS)

Where is Advanced Queuing used?


© 2008

Introduction

Queue Types

Case Study

Benchmarks

Recommendations

Conclusion


Agenda

Bildmarkierung


Daten sindimmer im Spiel.Comprehensiveunderstanding isthe through ballto yourperformance.

© 2008

Single Consumer Queue (Point-to-Point)


App A

Producer

Consumer

App B

Consumer

Producer

enqueue

enqueue

dequeue

dequeue

Request Queue

Response Queue

© 2008

Message States*


Ready

Processed

Expired Waiting

* = additional sub-states exist, e.g. for ready: “in memory”, “deferred”, “spilled”, “deferred spilled”

© 2008

Single Consumer Queue – Table Structure


Column Type PK Index 1 Index 2 CommentQ_NAME VARCHAR2(30) 1MSGID RAW(16) 1 Generated GUIDCORRID VARCHAR2(128) Correlation IDPRIORITY NUMBER 3 Message PrioritySTATE NUMBER 2DELAY TIMESTAMP(6)EXPIRATION NUMBERTIME_MANAGER_INFO TIMESTAMP(6) 1LOCAL_ORDER_NO NUMBER 7CHAIN_NO NUMBER 6CSCN NUMBERDSCN NUMBERENQ_TIME TIMESTAMP(6) 4ENQ_UID VARCHAR2(30)ENQ_TID VARCHAR2(30)DEQ_TIME TIMESTAMP(6)DEQ_UID VARCHAR2(30)DEQ_TID VARCHAR2(30)RETRY_COUNT NUMBEREXCEPTION_QSCHEMA VARCHAR2(30)EXCEPTION_QUEUE VARCHAR2(30)STEP_NO NUMBER 5RECIPIENT_KEY NUMBERDEQUEUE_MSGID RAW(16)SENDER_NAME VARCHAR2(30)SENDER_ADDRESS VARCHAR2(1024)SENDER_PROTOCOL NUMBERUSER_DATA SYS.AQ$_JMS_MESSAGE PayloadUSER_PROP SYS.ANYDATA

© 2008

Multi Consumer Queue (Publish-Subscribe)


App A

Producer

App B

Consumer

enqueue

dequeue

App C

Consumerdequeue

C C C

B B B

© 2008

AQ$_SAMPLE_MC_QT_G

*pfK MSGID*pfK SUBSCRIBER#*PK NAME*PK ADDRESS#

SIGNDBS_ SIGN

AQ$_SAMPLE_MC_QT_H

*pfK MSGID*pfK SUBSCRIBER#*PK NAME*PK ADDRESS#

DEQUEUE_TIMETRANSACTION_IDDEQUEUE_USERPROPAGATED_MSGIDRETRY_COUNTHINTSPARE

AQ$_SAMPLE_MC_QT_I

*pfK SUBSCRIBER#*PK NAME*PK QUEUE#*PK MSG_PRIORITY*PK MSG_ENQ_TIME*PK MSG_STEP_NO*PK MSG_CHAIN_NO*PK MSG_LOCAL_ORDER_NO*pfK MSGID

HINTSPARE

AQ$_SAMPLE_MC_QT_S

*PK SUBSCRIBER_ID* QUEUE_NAME

NAMEADDRESSPROTOCOLSUBSCRIBER_TYPERULE_NAMETRANS_NAMERULESET_NAMENEGATIVE_RULESET_NAME

AQ$_SAMPLE_MC_QT_T

*PK NEXT_DATE*PK TXN_ID*pfK MSGID

ACTION

SAMPLE_MC_QT

Q_NAME*PK MSGID

CORRIDPRIORITYSTATEDELAYEXPIRATIONTIME_MANAGER_INFOLOCAL_ORDER_NOCHAIN_ NOCSCNDSCNENQ_TIMEENQ_UIDENQ_TIDDEQ_TIMEDEQ_UIDDEQ_TIDRETRY_COUNTEXCEPTION_QSCHEMAEXCEPTION_QUEUESTEP_NORECIPIENT_KEYDEQUEUE_MSGIDSENDER_NAMESENDER_ADDRESSSENDER_PROTOCOLUSER_DATAUSER_PROP

0..* 1

0..* 1

0..* 1

0..*1

0..*1

0..*1

0..* 1

Multi Consumer Queue – Table Structure


© 2008

in Memory(SGA)

Buffered Queues

Officially since 10gR2

Stores Queues in MemorySGAstream_pool_size“faster”

Same API with newDelivery-Mode „Buffered“ forenqueue and dequeue


© 2008

Buffered Queues – Table Structure


AQ$_SAMPLE_MC_QT_S

*PK SUBSCRIBER_ID* QUEUE_NAME

NAMEADDRESSPROTOCOLSUBSCRIBER_TYPERULE_NAMETRANS_NAMERULESET_NAMENEGATIVE_RULESET_NAME

AQ$_SAMPLE_MC_QT_D

*PK QID*PK MSGNUM*PK SUB

MSGIDSEQNUMRSUBS

AQ$_SAMPLE_MC_QT_P

Q_NAME*PK MSGID

CORRIDPRIORITYSTATEDELAYEXPIRATIONTIME_MANAGER_INFOLOCAL_ORDER_ NOCHAIN_NOCSCNDSCNENQ_TIMEENQ_UIDENQ_TIDDEQ_TIMEDEQ_UIDDEQ_TIDRETRY_COUNTEXCEPTION_QSCHEMAEXCEPTION_QUEUESTEP_NORECIPIENT_KEYDEQUEUE_MSGIDSENDER_NAMESENDER_ADDRESSSENDER_PROTOCOLUSER_DATAUSER_PROP

SYS.QT<...>_BUFFER

ADDRINDXINST_IDOBJNO

*pfK QUEUE_ ID FK MSGID

CORRIDSEQUENCE_NUM

*pfK MSG_NUMSTATEPRIORITYEXPIRATIONENQ_TIMEENQ_UIDENQ_USER_NAMERETRY_COUNTSENDER_NAMESENDER_ADDRESSSENDER_PROTOCOLDEQUEUE_MSGIDSRCSEQUENCE_NUM

*pfK SUBSCRIBER_IDEXCEPTIONQ_SCHEMAEXCEPTIONQ_NAMEDBMS_AQ_BQVIEW.GET_ADT_PAYLOAD

0..* 1

0..*1

0..1 1

© 2008

Cannot participate in transactionsAutonomous transactions only!

Possible Data loss after instance shutdownIn-memory storage only!

Data consistency becomes a responsibility of the application

ORA-25293: Lob attributes must be Null for buffered operationsJMS payloads cannot be used for large texts and binary dataNo restriction for large object type like SYS.ANYDATA

ORA-25307: Enqueue rate too high, flow control enabledProducer is blocked when more than 5000 messages are not consumedPrevents application from flooding the shared poolTo increase the flow control limit see Metalink Note 551516.1

Buffered Queues – Downside



Main Types of Advanced Queues - RecapB

uffe

red

Single Consumer

Buffered, Multi Consumer QueuePrimarily in-memory-queue (SGA)Flow control limits queue lengthAutonomous transactions onlyData loss on instance shutdownLOBs not supportedFull propagation support except overmessaging gateway

Buffered, Single Consumer QueuePrimarily in-memory-queue (SGA)Flow control limits queue lengthAutonomous transactions onlyData loss on instance shutdownLOBs not supportedPropagation support for destination onlybut not over messaging gateway

Persistent, Multi Consumer QueueQueue tables stored on diskParticipating or autonomous transactionsLOBs supportedFull propagation support

Persistent, Single Consumer QueueQueue tables stored on diskParticipating or autonomous transactionsLOBs supportedPropagation support for destination only

Multi Consumer

Per

sist

ent

© 2008

Introduction

Queue Types

Case Study

Benchmarks

Recommendations

Conclusion


Agenda

Bildmarkierung


Daten sindimmer im Spiel.Comprehensiveunderstanding isthe through ballto yourperformance.

© 2008

Project Context – High Volume Printing

KPIs Today:

B/W Printing500 - 1000 pages per minute

Full-Color PrintingUp to 300 page per minute

“Tuning”Double-sided printing on A3Get 2 A4 pages by cuttingQuadruple Throughput~ 4000 pages per minute~ 67 pages per second



Trivadis BC Architecture Blueprint

© 2008

TOMAS – Transactional Behavior


1st Transaction

2nd Transaction

3rd Transaction (per Document)

4th Transaction5th Transaction

6th Transaction

© 2008

TOMAS – Number of Messages


Example:

5 Mio Docs

Control Flow500 Input Files2’000 Enq.2’000 Deq.

Document Flow10’000 Docs/File5 Mio Enq.5 Mio Deq.

© 2008

Introduction

Queue Types

Case Study

Benchmarks

Recommendations

Conclusion


Agenda

Bildmarkierung


Daten sindimmer im Spiel.It's about time forsome results

© 2008

Measuring What Counts?


OperationEnq, Deq

ConsumerSingle, Multi

Bulk?No, Yes (10)

Block Size8K, 2K Processing

AS, DB

Payload Size100 (~330)

DOP1, 2, 4, 8, 16

Volume16000, 4800

Message TypePersistent, Buffered Recipients

1

Payload Typeaq$_jms_message

Commit Interval1000

Other Parameter“fixed values” Queue Usage

Shared, Private

MetricsResponse Time,

DB Time,Logical I/O

msg/sec

© 2008

Test Environment

Application Server

Intel XEON E5345

2 Quad-Cores CPUat 2.33 GHz

16 GB RAM

SLES9, 64-bit

JDK 1.5.0_14

Database Server

IBM PowerPC_POWER5

4 Dual-Core CPUat 1.5 GHz

27 GB RAM

AIX 5.3 64-bit

Oracle RDBMS 10.2.0.3


© 2008

0

500

1'000

1'500

2'000

2'500

3'000

3'500

4'000

1 2 4 8 16

msg

/sec

DOP

Blocksize - Bulk? - Queue Usage

2 - No - Private

2 - No - Shared

2 - Yes - Private

2 - Yes - Shared

8 - No - Private

8 - No - Shared

8 - Yes - Private

8 - Yes - Shared

Single Consumer Enqueue – Overview


MESSAGE_TYPE PersistentVOLUME 16000CONSUMER SingleOPERATION EnqueuePAYLOAD_SIZE 100MESSAGE_PROCESSING AS

© 2008

Single Consumer Enqueue – No Bulk


MESSAGE_TYPE PersistentVOLUME 16000CONSUMER SingleOPERATION EnqueuePAYLOAD_SIZE 100MESSAGE_PROCESSING ASBLOCKSIZE 8BULK_PROCESSING NoQUEUE_USAGE Private

411845

1'632

2'892

3'425

0

20'000

40'000

60'000

80'000

100'000

120'000

140'000

160'000

0

2'000

4'000

6'000

8'000

10'000

12'000

1 2 4 8 16

Logi

cal I

/O

msg

/sec

DOP

msg/sec (Response Time)

msg/sec (DB Time)

LIO

© 2008

Single Consumer Enqueue – Bulk 10 Msgs


MESSAGE_TYPE PersistentVOLUME 16000CONSUMER SingleOPERATION EnqueuePAYLOAD_SIZE 100MESSAGE_PROCESSING ASBLOCKSIZE 8BULK_PROCESSING YesQUEUE_USAGE Private

547

779

1'374

1'864

2'591

0

20'000

40'000

60'000

80'000

100'000

120'000

140'000

160'000

180'000

0

500

1'000

1'500

2'000

2'500

3'000

3'500

1 2 4 8 16

Logi

cal I

/O

msg

/sec

DOP

msg/sec (Response Time)

msg/sec (DB Time)

LIO

© 2008

0

500

1'000

1'500

2'000

2'500

3'000

1 2 4 8 16

msg

/sec

DOP

Blocksize - Bulk? - Queue Usage

2 - No - Private

2 - No - Shared

2 - Yes - Private

2 - Yes - Shared

8 - No - Private

8 - No - Shared

8 - Yes - Private

8 - Yes - Shared

Single Consumer Dequeue – Overview


MESSAGE_TYPE PersistentVOLUME 16000CONSUMER SingleOPERATION DequeuePAYLOAD_SIZE 100MESSAGE_PROCESSING AS

© 2008

339

661

1'122

1'926

2'814

167'112

0

500'000

1'000'000

1'500'000

2'000'000

2'500'000

3'000'000

3'500'000

4'000'000

0

500

1'000

1'500

2'000

2'500

3'000

1 2 4 8 16

Logi

cal I

/O

msg

/sec

DOP

Private - msg/sec (Response Time)

Shared - msg/sec (Response Time)

Private - LIO

Shared - LIO

Single Cons. Dequeue – Private vs. Shared Queue


MESSAGE_TYPE PersistentVOLUME 16000CONSUMER SingleOPERATION DequeuePAYLOAD_SIZE 100MESSAGE_PROCESSING ASBLOCKSIZE 8BULK_PROCESSING No

© 2008

Single Cons. Enqueue – Private vs. Shared Queue


411

845

1'632

2'892

3'425

143'345

0

20'000

40'000

60'000

80'000

100'000

120'000

140'000

160'000

180'000

200'000

0

500

1'000

1'500

2'000

2'500

3'000

3'500

4'000

1 2 4 8 16

Logi

cal I

/O

msg

/sec

DOP

Private - msg/sec (Response Time)

Shared - msg/sec (Response Time)

Private - LIO

Shared - LIO

MESSAGE_TYPE PersistentVOLUME 16000CONSUMER SingleOPERATION EnqueuePAYLOAD_SIZE 100MESSAGE_PROCESSING ASBLOCKSIZE 8BULK_PROCESSING No

© 2008

Summary AQ Throughput (Single Consumer)

5 Mio EnqueuesShared Queue,3349 Mgs/sec, 1493 secPrivate Queue,3425 Mgs/sec, 1460 sec

5 Mio DequeuesShared Queue,1239 Mgs/sec, 4035 secPrivate Queue,3425 Mgs/sec, 1765 sec

Messaging Overhead30 to 54 Minutes


0

500

1000

1500

2000

2500

3000

35003'349 3'425

1'239

2'833

Mes

sage

s/Se

cond

Enqueue, shared Queue

Enqueue, private Queue

Dequeue, shared Queue

Dequeue, private Queue

© 2008

Introduction

Queue Types

Case Study

Benchmarks

Recommendations

Conclusion


Agenda

Bildmarkierung



© 2008

The easiest way to reduce the messaging overhead is to reducethe number of messages

In this case a message could contain an array of document IDsinstead a reference to a single document

An array size of e.g. 10 could reduce the messaging overhead byfactor 10

Changing the size of an array payload is a matter of configuration

Reduce Number of Messages


© 2008

Parallelize enqueue operations

Parallelize dequeue operations

Be aware that enqueue operations scale two or three times betterthan dequeue operations (shared queues)

Consider different degrees of parallelism for enqueue anddequeue operations

Parallelize to Increase Message Throughput


© 2008

Enqueueing Application expects to write to a single queue

Dequeueing Application expects to read from a single queue

A transparent solution is wanted

Reduce Concurrent Dequeue Operation (1)


AQProxy

Implements required JMSInterfacesSupports variousDistribution Strategies, e.g.

RandomSequentialReversePrioritySequential

Maps logical queue names tophysical queues, e.g.

queue[1-4] to queue1, …queue[2 4 8] to queue2, …

„Logical“ Queue (Multiplexer)

© 2008

AQProxy is just one option for a “logical” queue

Using Multi Consumer instead of Single Consumer Queues isanother option (assigning consumers to dequeue processes)

The downside of using “logical” queuesThe processing sequence might be affected, which e.g. requires newconcepts for high priority messagesMore database objects to be managed

Partitioning is not an optionIt’s not supported by AQFinding an appropriate key for partition elimination would have been aprecondition anyway

Reduce Concurrent Dequeue Operation (2)


© 2008

Minimize creation of QueueSession, TopicSessionCostly because of SQL statements executed for queue initializationNeed to be recreated whenever a JDBC connection is closed (e.g.through transaction managers)

JMS wrapper may have significant performance implications whenqueue related objects are recreated automatically:

E.g. Spring Framework’s JMS template

Use local instead of distributed transactions whenever possible

For better control usePlain JMSPlain JDBC via PL/SQL interfacePlain JDBC via new 11g JDBC AQ classes, seehttp://download.oracle.com/docs/cd/B28359_01/java.111/b31224/streamsaq.htm#BABBIAAC

Java Related Performance Recommendations


http://download.oracle.com/docs/cd/B28359_01/java.111/b31224/streamsaq.htm#BABBIAAC

© 2008

Create a dedicated queue table for each queue

Use NEXT_MESSAGE whenever possible

Schedule queue maintenance tasksAlter table … coalesceAlter table … shrink spaceSee Metalink Notes 271855.1, 284692.1, 421474.1 for details

Check out further documentationsPerformance Tuning Advanced Queuing Database and Applications

Metalink Note 102926.1 contains an overview of AQ performance relatedthemes and links to related Oracle documentation chapters

Generic AQ informationMetalink Note 267933.1 lists AQ related Metalink documents

General Performance Recommendations


© 2008

Introduction

Queue Types

Case Study

Benchmarks

Recommendations

Conclusion


Agenda

Bildmarkierung



© 2008

Consider using Buffered Queues when the reliability and transactionsupport of persistent tables is not required

Single and Multi Consumer Queues have similar throughput, but mayrequire different configuration (e.g. block size)

Increase the throughput by increasing the degree of parallelism,however, enqueue scales better than dequeue

Increase the scalability by assigning queues to enqueue and dequeueprocesses or consumers to dequeue processes (private queues)

Make the granularity of a message a part of your design considerations,that’s the key to improve overall messaging performance

Conclusion


Basel · Baden · Berne · Lausanne · Zurich · Düsseldorf · Frankfurt/M. · Freiburg i. Br. · Hamburg · Munich · Stuttgart · Vienna

Thank you!

?

Salvisberg AQ Design for Best Performance 02

Documents