Blocking Lock Killer - CoHEsion Nashville 2013

Presented by: Shawn Lee, University of Illinois

Monday, November 4, 2013Session #149

Blocking Lock Killer

co•he•sion noun \kō-ˈhē-zhən\ 1 : the act or state of sticking together tightly; especially: unity

2

TopicsDatabase transactions and locking mechanisms

Blocking locks in Banner

Monitoring & triaging blocking locks

University of Illinois’ blocking lock killer tool

CoHEsion Summit

CoHEsion Summit3

Audience Participation Test!

Overview of database transactions and locking mechanisms

Concepts

5

What is a transaction?A single execution within the database

A series of reads and writes of database objects

Final action – commit or abort

Committed transactions must be durable

Aborted transactions must be undone

CoHEsion Summit

6

ACIDTransactions have four essential properties:

1) atomicity

2) consistency

3) isolation

4) durability

CoHEsion Summit

7

Motivation for ConcurrencyTransactions can be executed in serial, but…that’s ineffiecient

DBMS improves performance by interleaving the actions of different transactions so they can run concurrently

Only serializable schedules are generated

CoHEsion Summit

8

Issues with ConcurrencyThree problems that can arise:

Dirty Reads (WR conflict) – a transaction reads a value written by another uncommitted transaction

Unrepeatable Reads (RW conflict) – a transaction reads the value of the same object twice; another transaction modifies that value in between the two reads

CoHEsion Summit

9

Issues with Concurrency (cont.)Lost Updates (WW conflict) – Two transactions update the value of the same object, before either of them have committed

CoHEsion Summit

10

Data Locking MechanismsShared Locks – acquired when reading data without making modifications; shared with other transactions that only need to read, not write

Exclusive Locks – acquired when modifying data, prevents any other transaction from accessing the data

The amount of data locked can vary

CoHEsion Summit

11

More on lockingTwo-Phase Locking (2PL) – common protocol used to make it easier for DBMS to schedule serializable transactions

# of locks acquired by transaction grows (phase 1) then shrinks (phase 2)

Deadlocks – when two transactions are stuck waiting on each other; Oracle detects and handles these automatically

CoHEsion Summit

Database Locks in BannerCauses of blocking locks and their

effects

13

The effect of locks in BannerDatabase locks are commonLocks can last for seconds, minutes, or hours depending on what the root blocker is doingThe “blocker” and “blockee” could be a batch job, or a user.If the blockee is a user, an extensive lock can result in a poor experience

CoHEsion Summit

14

Common CausesUsers blocking themselves

SOAHOLD (placing student holds)PEALEAV (employee leave balances/accruals)FZAREQN (custom form for multiyear POs)

Peak processing times (start of term activities, grade roll, payroll deduction runs)

Scheduled jobs (Appworx, a.k.a. UC4)

CoHEsion Summit

CoHEsion Summit15

Poll

Triaging Blocking LocksHow we handle them

17

It starts with active monitoringHP SiteScopeselect count(*) from v$session a, v$session b where 10 <= round(a.seconds_in_wait/60,1) and a.blocking_session_status = 'VALID’ and a.wait_class = 'Application’ and a.blocking_session=b.sid and b.username not like 'APPWORX%'

CoHEsion Summit

18

Our triage processMagic number = 10 minutesIdentify the root blockerDuring business hours, call user firstIf can’t make phone contact, kill sessionExtra care for jobsub processesSelf-service sessions killed immediatelyOutside business hours, kill immediately then communicate immediately

CoHEsion Summit

19

After the killThe database rolls back the transaction

User may lose work

Forms users screen may go blank and/or they will get network errors

Impact to self-service users will be less due to web session persistence

Communication and follow-up are keyCoHEsion Summit

The Tool The old and the new

21 CoHEsion Summit Session ID# 149

The Old Tool

22 CoHEsion Summit Session ID# 149

The Old Tool (cont.)

23 CoHEsion Summit Session ID# 149

The New Tool

24 CoHEsion Summit Session ID# 149

Session Table

25 CoHEsion Summit Session ID# 149

Contact Info

26 CoHEsion Summit Session ID# 149

Details and Kill Now options

27 CoHEsion Summit Session ID# 149

Session Details

28 CoHEsion Summit Session ID# 149

After the Kill

29 CoHEsion Summit Session ID# 149

Tool Permissions

WebTailor user roles

System Admin – Can Kill Sessions

System Monitor – “Kill” option not visible

30

Database procedures-- FILE NAME..: pkg_banner_monitoring_tools.pkg-- OBJECT NAME: UILLINOIS.pkg_banner_monitoring_tools-- USAGE......: Banner monitoring tools, including new blocking lock utility to replace the blocking lock utility from the-- Triage site that was written by Mike Cornell

procedure P_ShowSessionDetails (sid_in IN v$session.sid%TYPE, serial_in IN v$session.serial#%TYPE);

procedure P_ShowSessionsAndBlocks (dispinfo IN VARCHAR2 DEFAULT 'DEFAULT');

procedure P_KillSession (sid_in IN v$session.sid%TYPE, serial_in IN v$session.serial#%TYPE, username_in IN v$session.username%TYPE);

CoHEsion Summit

31

Noteworthy components

CoHEsion Summit

Database cursor – blocker and blockee information in each rowV$LOCK, V$SESSION, V$PROCESSContact info retrieved from GOBEACC, SPRIDEN, SPRTELE, SPRADDR, GOREMAL

32

Noteworthy components

CoHEsion Summit

Row coloring:CASE WHEN rblock.blockee_minutes_in_wait > 10 THEN rowcolor:='#FF0000'; --red WHEN rblock.blockee_minutes_in_wait > 3 THEN rowcolor:='#F4FA58'; --yellow ELSE rowcolor:='#80FF00'; --greenEND CASE;htp.tableRowOpen(cattributes=>'bgcolor="' || rowcolor || '"');

Control display of “Kill” link:if twbkwbis.f_validlink('pkg_banner_monitoring_tools.P_KillSession') then

33

Noteworthy components

CoHEsion Summit

Page refresh:twbkwbis.P_OpenDoc ('pkg_banner_monitoring_tools.P_ShowSessionsAndBlocks', refresh_delay => 30, refresh_url => OWA_UTIL.get_cgi_env ('REQUEST_PROTOCOL') || '://' || OWA_UTIL.get_cgi_env ('HTTP_HOST') || ':' || OWA_UTIL.get_cgi_env ('SCRIPT_NAME') || '/pkg_banner_monitoring_tools.P_ShowSessionsAndBlocks' );

Result = https://ui2web1.apps.uillinois.edu/BANPROD1/pkg_banner_monitoring_tools.P_ShowSessionsAndBlocks

34

Noteworthy components

CoHEsion Summit

Kill session: oracle.killsession(sid_in,serial_in,username_in,dbname,result); -- check the result CASE WHEN result LIKE '%auditname parameter not provided%' THEN P_ShowSessionsAndBlocks('NOKILL_NODB'); WHEN result LIKE '%This is a privileged ID%' THEN P_ShowSessionsAndBlocks('NOKILL_PRIVID'); WHEN result LIKE '%did not match with username%' THEN P_ShowSessionsAndBlocks('NOKILL_USRMTCH'); WHEN result LIKE '%alter system kill session%' THEN P_ShowSessionsAndBlocks('KILLED'); END CASE;

CoHEsion Summit35

Poll

36

Thank You!

Shawn Lee, University of [email protected]

Please complete the session evaluation form

Session ID #149

CoHEsion Summit