OPS-28: A New Spin on Some Old Latches Richard Banville Fellow
Feb 02, 2016
OPS-28:A New Spin on Some Old Latches
Richard BanvilleFellow
© 2008 Progress Software Corporation2 OPS-28: A New Spin on Some Old Latches
Agenda
Monitoring Latch usage Performance
© 2008 Progress Software Corporation3 OPS-28: A New Spin on Some Old Latches
Some Definitions First
What is a “latch”• Synchronization mechanism
• Latches vs “locks”
What is a critical section• Code access to shared resource
• Types of resources
How is a latch obtained• test, test-and-set, spin loop
– Performed on “shared” value
Single threading access
© 2008 Progress Software Corporation5 OPS-28: A New Spin on Some Old Latches
Latch Info
VST access to latch information
_Latch-Type Latch type
_Latch-Name Name
_Latch-Lock # times locked
_Latch-Wait Nap count
_Latch-HoldUser # of last holder
_Latch-QholdUser # of last holder of queue latch
for each _latch:
© 2008 Progress Software Corporation6 OPS-28: A New Spin on Some Old Latches
_Latch-Type: MT_LT_QUEUE
Queue Latch governor
LatchSemaphore
• Multi level latch
• Wakeup order guaranteed
• -spin 0 forces all latches
Test, test-and-set spin loop
© 2008 Progress Software Corporation7 OPS-28: A New Spin on Some Old Latches
_Latch-Type: MT_LT_SPIN
Queue Latch governor
LatchSemaphore
• Multi level latch
• Wakeup order guaranteed
• -spin 0 forces all latches
Spin Latch Latch
Short Nap
• Single level latch
• Most performant
“backoff”
Test, test-and-set spin loop
Test, test-and-set spin loop
-nap to -napmax
© 2008 Progress Software Corporation8 OPS-28: A New Spin on Some Old Latches
Latch Implementations
Spin or queue latch acts as governor of “mux” latch table
governor
Resources protectedMux latch table (128)
• Multi level latch with one governor
• Associates critical section w/resource
• Underlying one to many relationshipTurns 1 latch into 128 latches
“Mux” latch
request
© 2008 Progress Software Corporation9 OPS-28: A New Spin on Some Old Latches
Latch Implementations
Spin or queue latch acts as governor of object latchResources protected
• Multi level latch with one governor
• Associates critical section w/resource
• Underlying one to one relationshipTurns 1 latch into many latches
governor
• Enhanced mux latch
Object latch - type 1
request
© 2008 Progress Software Corporation10 OPS-28: A New Spin on Some Old Latches
Latch Implementations
Spin or queue latch maintained within the resource itselfResources protected
• Single level latch. No governor
• Associates critical section w/resource
• One to one relationship
• Greatly improves concurrency
No single access funneling
• Statistics maintained as if 1 latch exists
Object latch - type 2
request
© 2008 Progress Software Corporation11 OPS-28: A New Spin on Some Old Latches
Latch Implementations
Table of spin or queue latchsResources protectedLatch table
• Single level latch with no governor
• Associates critical section w/resource
• Underlying one to many relationship
Latch families
request
• Statistics typically make this look like one latch
• Modified “mux” latch mechanism
© 2008 Progress Software Corporation12 OPS-28: A New Spin on Some Old Latches
Latch Holder
_Latch-QHolder• Last owner of queue type latch
• Not last queued user
_Latch-Holder• Last owner of spin latch
• Not honored for “true” object latches– Type 2
Who am I waiting on?
© 2008 Progress Software Corporation13 OPS-28: A New Spin on Some Old Latches
Latch Counts
_Latch-Lock• # times latch locked/acquired
• High numbers are OK
_Latch-Wait• # times user had to wait (after spin)
• High numbers may not be good– …but is not wrong– Performance issue potential
• Balance with –spin value (see tuning)
Activity statistics (or lack there of)
© 2008 Progress Software Corporation14 OPS-28: A New Spin on Some Old Latches
Latch Names
_Latch-Name
LKP LKF LHT LHT2 LHT3 LHT4 SEQ PWQ AIW BIW
BFP BHT BF1 BF2 BF3 BF4 BF5 BF6 BF7 CPQ
LRU LR2 LR3 LR4 L27 L28 L28 L30 L31
MTX USR OM AIB BIB SCC GST TXT TXQ
© 2008 Progress Software Corporation15 OPS-28: A New Spin on Some Old Latches
Agenda
Monitoring Latch usage Performance
© 2008 Progress Software Corporation16 OPS-28: A New Spin on Some Old Latches
Miscellaneous Latches
USR*• Protects login (user) control structures
SCC*• Protects schema locking operation
GST• Protects shared memory allocation
*Indicates queue latch
© 2008 Progress Software Corporation17 OPS-28: A New Spin on Some Old Latches
Obsolete Latches
Removed• AIW, LR2, LR3, LR4
Not in use• BIW
Renamed & reserved for future use• BF5 L27
• BF6 L28
• BF7 L29
• BF8 L30
Latches removed or reserved
© 2008 Progress Software Corporation18 OPS-28: A New Spin on Some Old Latches
Latches Protecting
Database Update Actions
© 2008 Progress Software Corporation19 OPS-28: A New Spin on Some Old Latches
Update Latches - MTX
DB Update ActionDB Update Action Record BI noteRecord BI note Record AI noteRecord AI note Perform ActionPerform Action
DB Update ActionDB Update Action Record BI noteRecord BI note Record AI noteRecord AI note Perform ActionPerform Action
DB Update ActionDB Update Action Record BI noteRecord BI note Record AI noteRecord AI note Perform ActionPerform Action
Top of the food chain – busiest/longest update latch
• Protects txn allocation & bi/ai note order
• Quiet point maintenance
• Online backup startup
Latches protecting recovery logging procedure
• High activity
• High nap rate
MTX
Not allowed
User 1
User 2
User 3
© 2008 Progress Software Corporation21 OPS-28: A New Spin on Some Old Latches
BIB and AIB latch usage
-bibufs 10
Free(a)
Free(b)
Free(c)
Free(d)
Free(e)
32 31
30
29
Modified QueueFree List
BI
Current Output Buffer
New Notes (Actions)
Forward Processing
Bi (ai) buffer life cycle
B I W
Very little contention
until BIW active
OLTP latch order:• MTX, BIB, AIB
Page writer latch order:• BIB or AIB
BIB
BIB
BIB
User
© 2008 Progress Software Corporation22 OPS-28: A New Spin on Some Old Latches
Latches – LRU & PWQ
3232 20482048 10241024
-B least recently used (LRU) chain maintenance
. . .. . .81928192 10561056LRU MRU
High activityHigh nap rate
LRU
© 2008 Progress Software Corporation23 OPS-28: A New Spin on Some Old Latches
Latches – LRU & PWQ
3232 20482048 10241024
-B least recently used (LRU) chain maintenance
. . .. . .81928192 10561056LRU MRU
10881088
20802080
32003200
Page writer queue (PWQ)
myDBmyDB
PageWriter
PageWriter
Dirty buffer
Very little contentionunless too many APWs
LRU
PWQ
PWQ
LRU
UserUser
© 2008 Progress Software Corporation24 OPS-28: A New Spin on Some Old Latches
Latches – LRU & CPQ
3232 20482048 10241024
-B least recently used (LRU) chain maintenance
. . .. . .81928192 10561056LRU MRU
10881088
20802080
32003200
Page writer queue (PWQ)
myDBmyDB
PageWriter
PageWriter
10241024
71047104
60806080
Check point queue (CPQ)
Check-point
Check-point
Very little contention
CPQ
CPQ
© 2008 Progress Software Corporation25 OPS-28: A New Spin on Some Old Latches
Online backup queue - BFP
Online “point in time” backup
User 1
BI
Area 6 Area 7 Area 8 …
Online Backup
.bak
Backupqueue
Modify requestModify request
• Very little contention
• Only used w/online backup
Back it up and mark it
BFP
TXEMTX
BFP
© 2008 Progress Software Corporation26 OPS-28: A New Spin on Some Old Latches
Update Latches
TXT• Protects transaction table modifications
TXQ• Protects acquire/release of transaction end
lock (TXE)
SEQ• Protects sequence control structures
Moderate activity Little contention
Latches for write operation structures
© 2008 Progress Software Corporation27 OPS-28: A New Spin on Some Old Latches
Latches Protecting
Data Access Actions
and Housekeeping
© 2008 Progress Software Corporation28 OPS-28: A New Spin on Some Old Latches
Disk Storage Files
Storage Areas
Data Area
Ext
ent
Ext
ent
Ext
ent
Ext
ent
Ext
ent
Ext
ent
Ext
ent
Ext
ent
Ext
ent
Ta
ble
A
Ta
ble
B
Cu
st
Tb
l
Ta
ble
D
Ind
ex
A-1
Ind
ex
B-1
Na
me
Id
x
Cu
st#
Id
x
SR
ep
Id
x
Ind
ex
D-1
Ind
ex
D-2
Logical
Physical
© 2008 Progress Software Corporation29 OPS-28: A New Spin on Some Old Latches
Disk Storage Files
Storage Areas
Schema Area Area 7 Area 8
Ext
ent
Ext
ent
Ext
ent
Ext
ent
Ext
ent
Ext
ent
Ext
ent
Ext
ent
Ext
ent
Ta
ble
A
Ta
ble
B
Cu
st
Tb
l
Ta
ble
D
Ind
ex
A-1
Ind
ex
B-1
Na
me
Id
x
Cu
st#
Id
x
SR
ep
Id
x
Ind
ex
D-1
Ind
ex
D-2
Logical
Physical
Lan
gu
ages
Sto
rage E
ng
ine
© 2008 Progress Software Corporation30 OPS-28: A New Spin on Some Old Latches
Disk Storage Files
Storage Areas and Object Mapping
Schema Area Area 7 Area 8
Ext
ent
Ext
ent
Ext
ent
Ext
ent
Ext
ent
Ext
ent
Ext
ent
Ext
ent
Ext
ent
Ta
ble
A
Ta
ble
B
Cu
st
Tb
l
Ta
ble
D
Ind
ex
A-1
Ind
ex
B-1
Na
me
Id
x
Cu
st#
Id
x
SR
ep
Id
x
Ind
ex
D-1
Ind
ex
D-2
Logical
Physical
Mapping
Lan
gu
ages
Sto
rage E
ng
ine
© 2008 Progress Software Corporation31 OPS-28: A New Spin on Some Old Latches
Object Cache – OM Latch
Ta
ble
A
Ta
ble
B
Cu
st
Tb
l
Ta
ble
D
Ind
ex
A-1
Ind
ex
B-1
Na
me
Id
x
Cu
st#
Id
x
SR
ep
Id
x
Ind
ex
D-1
Ind
ex
D-2
Logical
Mapping
Lan
gu
ages
-omsize
OM cache loaded on demand
Very high activityHigh nap rate
OM latch needed for
paging maintenance
Get _StorageObject record
Schema
OM
© 2008 Progress Software Corporation32 OPS-28: A New Spin on Some Old Latches
Object Cache - OM Latch
Ta
ble
A
Ta
ble
B
Cu
st
Tb
l
Ta
ble
D
Ind
ex
A-1
Ind
ex
B-1
Na
me
Id
x
Cu
st#
Id
x
SR
ep
Id
x
Ind
ex
D-1
Ind
ex
D-2
Logical
Mapping
Lan
gu
ages
OM cache loaded at startup
No latching, no paging
Zero contention! Primary CachePrimary Cache
Little contention 2nd cache for over flow &
OM latch needed for
paging maintenance
Get _StorageObject record
Schema
Secondary CacheSecondary Cache new objects
OM
Schema
-omsize
© 2008 Progress Software Corporation33 OPS-28: A New Spin on Some Old Latches
Buffer Pool Hash Table - BHT
Hash Table (-hash)
Shared memory block lookup
List of (–B) buffer pool entries(unordered)
Buffer pool location lookup single threaded
User 1
User 2
User 3
User 4
Block ID
Block ID
Block ID
Block ID
BHT
High activity, few naps
Ptr to buffer
© 2008 Progress Software Corporation34 OPS-28: A New Spin on Some Old Latches
Buffer Pool Hash Table - BHT
Hash Table (-hash)
BHT now latch family of 256
List of (–B) buffer pool entries(unordered)
Buffer pool location lookup multi-threaded
Block IDUser 1
User 2
User 3
User 4
Block ID
Block ID
Block ID
High activity, few naps
Ptr to buffer
BHT
BHT
BHT
. . .
BHT
BHT
© 2008 Progress Software Corporation35 OPS-28: A New Spin on Some Old Latches
-B buffer pool entry info – BF[1-4]
Database access latches
List of (–B) buffer pool entries(unordered)
Ptr to buffer
Hash Table (-hash)
BHT
BHT
BHT
. . .
BHT
BHT
© 2008 Progress Software Corporation36 OPS-28: A New Spin on Some Old Latches
-B buffer pool entry info – BF[1-4]
List of (–B) buffer pool entries(unordered)
Buffer pool info structure supports
Hash Table (-hash)
Database access latches
max of 4 concurrent threads.
BF1
BF2
BF3
BF4
Latch family of 4Ptr to buffer
BHT
BHT
BHT
. . .
BHT
BHT
© 2008 Progress Software Corporation37 OPS-28: A New Spin on Some Old Latches
-B buffer pool entry info – BF[1-4]
List of (–B) buffer pool entries(unordered)
Block IDUser 1
User 2
User 3
User 4
Block ID
Block ID
Block ID
BF latch family of 4 Object latch type 2Ptr to buffer
Hash Table (-hash)
BHT
BHT
BHT
. . .
BHT
BHT
© 2008 Progress Software Corporation38 OPS-28: A New Spin on Some Old Latches
-B buffer pool entry info – BF[1-4]
Hash Table (-hash)List of (–B) buffer pool entries(unordered)
Block IDUser 1
User 2
User 3
User 4
Block ID
Block ID
Block ID
• -B info structure supports –B threads
• 4 latch slots used for statistics only
Ptr to bufferBF latch family of 4 Object latch type 2
BHT
BHT
BHT
. . .
BHT
BHT
© 2008 Progress Software Corporation39 OPS-28: A New Spin on Some Old Latches
Lock Table Latches (LKF, LKP & LHT)
Free Chain (-L entries)
Hash Chain Anchor Table-lkhash entries
Acquire a lock
LKF
Plain latchSingle threaded accessHigh activityLow nap rate
© 2008 Progress Software Corporation40 OPS-28: A New Spin on Some Old Latches
Lock Table Latches (LKF, LKP & LHT)
Free Chain (-L entries)
Hash Chain Anchor Table-lkhash entries
Protected by LHT (LKT)
• Record locks• Record “get” locks
• Table locks• Purged record locks
and LKP latches
Type 1 object latchSingle threaded latch access
Multi threaded chain access
LHT
LKP
LHT
LKP
• High activity/waits
• Low activity/waits
© 2008 Progress Software Corporation41 OPS-28: A New Spin on Some Old Latches
LHT1
LHT2
LHT3
LHT4
Lock Table Latches (LKF, LKP & LHT)
Free Chain (-L entries)
Hash Chain Anchor Table-lkhash entries
Protected by LHT[1-4]
• Record locks
• Record “get” locks
Type 1 object latch family
Multi threaded latch access
Multi threaded chain access
- Added 4 governors
Enhanced concurrency
• Low contention/waits
LKP
• High activity, few waits
© 2008 Progress Software Corporation42 OPS-28: A New Spin on Some Old Latches
Agenda
Monitoring Latch usage Performance
© 2008 Progress Software Corporation43 OPS-28: A New Spin on Some Old Latches
Tuning parameters
-spin• # retries before nap
– 0 or 1 or 10,000 or 6,000 * # cpus or …• Can be changed online
-mux• Use multiplexing/governor (1)
-nap• Initial amount to nap (10 ms)
-napmax*• Max amount to nap (5 sec)
-napinc and -napstep• Both obsolete and ignored
Nothing new here
© 2008 Progress Software Corporation44 OPS-28: A New Spin on Some Old Latches
Foresight and insight
CPU usage• -spin ~6,000 * # CPUs
• CPU thrashes, decrease –spin
• > 16 CPUs, avoid cache line ping pong
• Activity matters
Activity• # latches acquired
• # latch waits
• Application throughput
What to monitor
© 2008 Progress Software Corporation45 OPS-28: A New Spin on Some Old Latches
Performance – 10.1C vs 10.1B
0
200,000
400,000
600,000
800,000
1,000,000
OM
LHT
LKF
LRU
BH
T
BF1
BF2
BF3
BF4
Readprobe Latch Results
10.1c10.1b
House Keeping Data Access
© 2008 Progress Software Corporation46 OPS-28: A New Spin on Some Old Latches
Performance – 10.1C vs 10.1B
0
100,000
200,000
300,000
400,000
500,000
OM
LHT
LKF
LRU
BH
T
BF1
BF2
BF3
BF4
Readprobe Latch Results
10.1c10.1b
Housekeeping Data Access
Latches/sec
LRU 31% Buf 43%
© 2008 Progress Software Corporation47 OPS-28: A New Spin on Some Old Latches
Performance – wait for it
0
10,000
20,000
30,000
40,000
50,000
Readprobe Latch Waits (per sec*)
10.1c10.1b
10.1c 0 0 0 45,023 10 1 1 1 2
10.1b 298 31,141 8,699 7,790 969 75 100 172 88
OM LHT LKF LRU BHT BF1 BF2 BF3 BF4
4,294 (~9%) fewer latch waits/sec in 10.1c
© 2008 Progress Software Corporation48 OPS-28: A New Spin on Some Old Latches
Readprobe Data Access Results
50,000
100,000
150,000
200,000
250,000
300,000
1 6 11 16 21 26 31 36 41 46 51 56 61 66 71 76 81 86 91 96 101
Rec
ord
s R
ead
10.1C
10.1B
Performance – you got it!
~40%~56%
0 4 8 12 16 24 32 256 512 768 992# Users
© 2008 Progress Software Corporation49 OPS-28: A New Spin on Some Old Latches
In Summary
Better insight We’ve done some things You can too
• Tuning
• Move to 10.1C
© 2008 Progress Software Corporation50 OPS-28: A New Spin on Some Old Latches
Questions?
© 2008 Progress Software Corporation51 OPS-28: A New Spin on Some Old Latches
Thank You
© 2008 Progress Software Corporation52 OPS-28: A New Spin on Some Old Latches