A9: WebSphere MQ for z/OS The Inside Story Damon Cross, Advisory Software Engineer [email protected]
Jul 12, 2015
A9: WebSphere MQ for z/OS The Inside Story
Damon Cross, Advisory Software [email protected]
© 2014 IBM Corporation © 2014 IBM Corporation
Please NoteIBM’s statements regarding its plans, directions, and intent are subject to change or withdrawal without notice at IBM’s sole discretion. Information regarding potential future products is intended to outline our general product direction and it should not be relied on in making a purchasing decision.
The information mentioned regarding potential future products is not a commitment, promise, or legal obligation to deliver any material, code or functionality. Information about potential future products may not be incorporated into any contract. The development, release, and timing of any future features or functionality described for our products remains at our sole discretion
Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the amount of multiprogramming in the user’s job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here.
© 2014 IBM Corporation
Agenda
• Transactions• Queue managers• Resource managers• Scenario walk-through• Summary
© 2014 IBM Corporation
Transactions
An attempt to change the state− Either fully completes or is fully restored to the original state− Like a contract− Atomicity, Consistency, Isolation, Durability
Business is all about transactions− e.g. Banking, Order processing, Inventory management
Transactional messaging− Once and only once delivery of messages
Coordination is essential for Extended Transactions− Transactions that update multiple resources
© 2014 IBM Corporation
Agenda
• Transactions• Queue managers• Resource managers• Scenario walk-through• Summary
© 2014 IBM Corporation
The queue manager – delivering messages
APPLICATION
EXEC SQL QUERY
EXEC SQL INSERT
DATABASE
Read
Write
APPLICATION
MQGET
MQPUT
Read / Write
Write
MQ
DB/2 IMSCICS
z/OS
CLIENT
CLIENT
CLIENTMQI
QUEUE
© 2014 IBM Corporation
Core technology – logging and buffering
...C3 A1B4
Conveyor Belt Log
A1
C3
B4
Pigeon Hole Page set
112
2
3
45
67
8
9
10
11
C3 A1B4
?
© 2014 IBM Corporation
Accessing the MQI - stubs and adapters
APPLICATION
MQI STUB
RMI TRUE
CICS REGION
APPLICATION
MQI STUB
ADAPTER
BATCH / TSO
PROCESS
MQI STUB
ADAPTER
CHANNEL INITIATOR
APPLICATION
MQI STUB
IMS REGION (BMP/FP/MPP)
CONTROL
ESAF EXITS
QUEUE MANAGER
PROGRAM CALLS (QRPL)
© 2014 IBM Corporation
Agenda
• Transactions• Queue managers• Resource managers• Scenario walk-through• Summary
© 2014 IBM Corporation
Building blocks – resource managers
CONNECTION
MANAGER
RECOVERY
MANAGER
LOG
MANAGER
MESSAGE
MANAGER
DATA
MANAGER
BUFFER
MANAGER
LOCK
MANAGER
CF
MANAGER
COMMIT / BACKOUT UR
MQI
CKPT
CKPT
REDO / UNDO
URRELEASE
MQOPEN
© 2014 IBM Corporation
Handling applications – Connection Manager
APPLICATION
MQCONN
MQOPEN
MQPUT
…
TRANSACTION COORDINATOR
THREAD
EOT
EOMPHASE 1
PHASE 2
CONTROL BLOCKS
CONNECTION MANAGER
ADDRESS SPACE BOUNDARY
OTHER RESOURCE MANAGERS
DATA
LOGS
CONTROL BLOCKS
CONTROL BLOCKS
© 2014 IBM Corporation
Supporting the MQI & MQSC – Message Manager
MESSAGE MANAGER
APPLICATIONCOMMAND, RUNTIME & GROUP SERVERS
MQSC CONSOLE
QSG
MQOPEN
MQCLOSE
MQGET
MQPUT
MQSET
MQINQ
MQSUB
OBJECTS
SYSTEM.CLUSTER.*
MQOPEN
BASE
VALIDATION, CONSISTENCY CHECKING, NAME RESOLUTION, TRIGGERING, GET-WAIT PROCESSING, PUT-TO-WAITING-GETTER
© 2014 IBM Corporation
Managing messages and objects – Data Manager
112
23
4567
8910
11
SCAVENGER
LOG
REDO
UNDO
MESSAGES
OBJECTS
QUEUE ATTRIBUTES
SUB-QUEUE MESSAGE POINTERS
SPACE GROUP
SPACE GROUP
SPACE GROUP
PAGE SET 0
PAGE SETS 1--99
© 2014 IBM Corporation
Improving performance – Buffer Manager
PAGE SET
BUFFER
PAGES
READ PAGE SET
DEFINE BUFFPOOL(bpid) BUFFERS(nnnn)
DEFINE PSID(psid) BUFFPOOL(bpid)
STEAL LRU
DIRTY
DIRTY
DIRTY
NO FORCECHECKPOINT,
STEAL, WRITE AHEAD
OLDESTWRITE PAGESET
FORCE DIRTY PAGES > 2 CHECKPOINTSCHECKPOINT
© 2014 IBM Corporation
Local message queue storage
HEAD
START SCAN
TAIL
SUB-QUEUE (x 20)
PAGE SET 0
MSG 1 (DELETED)
MSG 2 (≤ 4 KB) MQMD + DATA
MSG 3 (≤ 4 MB)
MQMD + DATA NEXT DATA
MSG 4 (≤ 100 MB)
MQMD + DATA NEXT DATA NEXT DATA NEXT DATA
PAGE SET n
© 2014 IBM Corporation
Local message queue storage
• Pages are generally deallocated as messages removed from queues.
•• Asynchronous deallocation is performed by the scavenger task
– Acquires exclusive locks on pages for all messages to be deleted.
•• Pure text pages don't suffer contention and can always be
deleted during MQGET processing.•• Page allocation for small (<4KB) messages changed in V6.•• Objects are stored much like messages on pageset 0.
© 2014 IBM Corporation
Handling transactions – Recovery Manager
TRANSACTION STATES
112
2
3
4567
8
9
1011 112
2
3
4567
8
9
1011 112
2
3
4567
8
9
1011 112
2
3
4567
8
9
1011 112
2
3
4567
8
9
1011
1-PHASE COMMIT
1-PHASE BACKOUT
2-PHASE COMMIT
BEGIN
BEGIN
BEGIN
IN FLIGHT
IN FLIGHT
IN FLIGHT
IN COMMIT
IN BACKOUT
IN DOUBT
COMMITTED
BACKED OUT
IN COMMIT COMMITTED
LOGLOGLOAD (zPARM)
CHECKPOINT
LOG
PAGESET
RESTART
RECOVERY
/
© 2014 IBM Corporation
Providing logging interfaces – Log Manager
• Log read and write functions
• Log shunting
• Multiple active log data sets and archive
• Archive inventory management
• Duplexed for reliability
• “Bootstrap” file – End of log location– Archive inventory– CSQJU004
• Various Utilities – CSQ1LOGP, CSQ4LOGS
© 2014 IBM Corporation
Examining transactions – a log print 00000000D569 URID(00000000D569) RM(RECOVERY) TYPE( START UR ) **** 00640024 00200001 03000000 0000D569 00000000 D545 0000 00240000 0000D000 00000000 00000700 00000000 00000000 00000000 0000D6C4 0020 D6E6C4C1 4040B5B4 8FA08793 02864040 40404040 4040C2C1 E3C3C840 4040D6C4 0040 D6E6C4C1 40400000 00000000 0000
00000000D5CD URID(00000000D569) RM(DATA) LRID(00000000.00000E01) TYPE( UNDO REDO ) SUBTYPE( DECREMENT BY ) **** 002A0064 0600000F C9000000 0000D569 00000000 D569 0000 00000000 00000E01 00040326 00000001 00000001
00000000D5F7 URID(00000000D569) RM(DATA) LRID(00000001.00000201) TYPE( UNDO REDO ) SUBTYPE( DELETE ) **** 0026002A 06000008 C9000000 0000D569 00000000 D5CD 0000 00000001 00000201 00000000 00000E01
00000000D61D URID(00000000D569) RM(RECOVERY) TYPE( START COMMIT1 ) **** 007C0026 00200002 03000000 0000D569 00000000 D5F7 0000 00240000 0000D000 00000000 00000700 00000000 00000000 00000000 00004040 0020 40404040 40400000 00000000 00000000 00000000 00000000 00000000 00000000 0040 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 0060 00000000 0000
00000000D699 URID(00000000D569) RM(RECOVERY) TYPE( PHASE 1 TO 2 ) **** 0024007C 0020000C 03000000 0000D569 00000000 D61D 0000 00240000 0000D000 00000000 0000
00000000D6BD URID(00000000D569) RM(RECOVERY) TYPE( END COMMIT2 ) **** 00240024 00200010 03000000 0000D569 00000000 D699 0000 00240000 0000D000 00000000 0000
00000000D569 URID(00000000D569)
TYPE( END COMMIT2 )
RM(RECOVERY)
RM(DATA)SUBTYPE( DECREMENT BY )
SUBTYPE( DELETE )
LRID(00000000.00000E01)
LRID(00000001.00000201)
TYPE( START UR )
TYPE( UNDO REDO )
00000000D699 URID(00000000D569) RM(RECOVERY) TYPE( PHASE 1 TO 2 ) **** 0024007C 0020000C 03000000 0000D569 00000000 D61D0000 00240000 0000D000 00000000 0000
© 2014 IBM Corporation
Concurrency and isolation – Lock Manager
MQOO_INPUT_SHARED
MQOO_INPUT_EXCLUSIVE
A
B
C
A B C
PAGE (shared)
MESSAGE (exclusive)
MESSAGE (exclusive)
112
2
3
4567
8
9
1011 112
2
3
4567
8
9
1011 112
2
3
4567
8
9
1011 112
2
3
4567
8
9
1011
REQUEST
COMMIT
ALLOCATION
MQOPEN
vs
DELETE QLOCAL
API Request
Transaction
Lifetime
© 2014 IBM Corporation
Managing shared messages – CF Manager
QM1
CF MANAGER
LIST STRUCTURE
QM2
CF MANAGER
QM3
CF MANAGER
IXL*
DEFINE QLOCAL(...)
QSGDISP(SHARED)
CFSTRUCT(...)
© 2014 IBM Corporation
Shared message queue storage
SQ1 LH
SQ2 LH
SQM1 LH
LIST STRUCTURE
COMMITTED PUTS
UNCOMMITTED GETS
UNCOMMITTED PUTS QUEUE DETAILS
EXPIRED MESSAGES
MQCMIT
MQCMIT
MQGET
(MQGET)
MQPUTMQGET
KEYS
DATA
State Priority Time Queue Manager
MsgId / CorrelationId
READ
WRITE
MOVE
DELETE
MONITOR
© 2014 IBM Corporation
Agenda
• Transactions• Queue managers• Resource managers• Scenario walk-through• Summary
© 2014 IBM Corporation
Scenario: MQPUT to a triggered queueAPPLICATION MESSAGE MGR DATA MGR BUFFER MGR RECOVERY MGR LOG MGR LOCK MGR
MQOPEN
ACQUIRE LOCK
LOCATE QUEUE (HASH TABLE)
SECURITY CHECK
RESOLVE NAME
GET HANDLE
MQPUT
USE HANDLE
LOCATE NEW PAGE (FOR MESSAGE)
BUFFER PAGE
START UR LOG RECORDS
LOG RECORDS
CHECK TRIGGER RULES
MQCMIT
FORCE LOG
RELEASE LOCKS
© 2014 IBM Corporation
Scenario: MQGET from a queueAPPLICATION MESSAGE MGR DATA MGR BUFFER MGR RECOVERY MGR LOG MGR LOCK MGR
MQOPEN
ACQUIRE LOCK
LOCATE QUEUE (HASH TABLE)
SECURITY CHECK
RESOLVE NAME
GET HANDLE
MQGET
USE HANDLE
FIND MESSAGE (INDEX / NEXT)
BUFFER PAGE
START UR LOG RECORDS
LOG RECORDS
MQCMIT
FORCE LOG
RELEASE LOCKS
© 2014 IBM Corporation
Summary• Delivers transactional messaging
– Enables robust business applications• Complex, but well organised
– Adapters– Address spaces– Resource managers
• Designed for throughput, availability and scalability– Logging– Buffering– Locking– Communications
Questions?
A9: WebSphere MQ for z/OS The Inside Story
© 2014 IBM Corporation
For Additional Information
IBM Training http://www.ibm.com/training
IBM WebSphere http://www.ibm.com/software/websphere/ http://www.ibm.com/software/products/ibm-mq
IBM developerWorks http://www.ibm.com/developerworks/websphere https://www.ibm.com/developerworks/community/blogs/messaging
WebSphere forums and community http://www.ibm.com/developerworks/websphere/community/
© 2014 IBM Corporation
IBM MQ Sessions this week
10:30 - 12:00 13:15 - 14:15 14:30 - 15:30 16:00 - 17:00 17:15 - 18:15
Tuesday
Opening General Session- IBM Digital Experience and WebSphere Technical University
Session A31: IBM MQ CHLAUTH rules – with MQ V8 updates
Speaker: Morag HughsonRoom 02
Session A4: WebSphere MQ for z/OS: Performance and Accounting
Speaker: Alexander RossRoom 8
Session I26: DataPower-MQ Connectivity Deep Dive (Theory)
Speaker: Robin WileyRoom 27
Session Z1: WebSphere MQ for z/OS V8: Latest Features Deep Dive
Speaker: Damon CrossRoom 6
9:00 - 10:00 10:30 - 11:30 11:45 - 12:45 14:00 - 15:00 15:15 - 16:15 16:45 - 17:45
Wednesday
Session Z5: WebSphere MQ for z/OS: Security
Speaker: Damon CrossRoom 02
Session A21: What's New in IBM Messaging
Speaker: Morag HughsonRoom 8
Session C7: Messaging in the Cloud with IBM MQ Light and IBM Bluemix
Speaker: Rob NicholsonRoom 27
Session A17: Managing work-loads, scaling and availability with IBM MQ clusters
Speaker: David WareRoom 6
Lab IL5: DataPower-MQ Connectivity Deep Dive (Hands-On)
Speaker: Robin WileyRoom 7b
Session A9: WebSphere MQ for z/OS: The Inside Story
Speaker: Damon CrossRoom 6
Thursday
Session A35: How to Develop Responsive Applications with IBM MQ Light
Speaker: Rob NicholsonRoom 27
Session A22: New IBM MQ V8 Security Features
Speaker: Morag HughsonRoom 01
Session A3: WebSphere MQ for z/OS: Shared Queues
Speaker: Alex RossRoom 6
Session A18: Using Publish /Subscribe with IBM MQ
Speaker: David WareRoom 27
Friday
Lab AL6: Developing a First Application with IBM WebSphere MQ Light
Speakers: Robert Nicholson, Alex RossRoom 7b
Session A16: Using IBM MQ Pub/Sub in an MQ network
Speaker: David WareRoom 6
A9: WebSphere MQ for z/OS The Inside Story
Thank you!