SQL Police

1

SQL Police GSE Conference 2010

Marcus DavageDB2 Database Administrator

Lloyds Banking Group

2

The Agenda

The History The Need The Requirements The Solution The Result

3

The History

“there is nothing new under the sun”What has been is what will be,

and what has been done is what will be done,and there is nothing new under the sun.

Is there a thing of which it is said,“See, this is new”?

It has been alreadyin the ages before us.

(Ecclesiastes 1:9-10 ESV)

The Incarnations of SQL Police Marcus Davage at British Steel (c. 1994) Robert Lenzie at British Telecom (c. 1994) Marcus Davage, with a little bit of help from Robert Lenzie, both now at

Lloyds TSB (2002-2006)

4

The Kit“Put on the red light”

Heritage Lloyds Systems (not including HBOS/SW)

Z9 Architecture (z/OS 1.9) DB2 V7 going to V8 CICS TS 3.1 (and higher)

Production 4 LPAR sysplex 4 DB2 subsystems (non-DS)

DB2J 761 GB DB2P 1.1 TB DB2B 353 GB DB2L 2.4 TB (largest table – 350 GB)

194 CICS regions Sample transaction per day – 71,406,887 Max transactions per second – 2,111

Development 17 DB2 subsystems (non-DS) 1200+ CICS regions

5

The Kit“Giant steps are what you take”

Integrated LBG (Strategic Lloyds TSB + HBOS)

Z10 Architecture (z/OS 1.11) DB2 V8 CICS TS 4.1

Development 4 LPAR sysplex 17 DB2s (3 3-way data sharing groups)

Acceptance Testing 8 LPARs 12 DB2s (3 4-way data sharing groups)

Production As Acceptance Testing. Probably. plus Heritage Lloyds

System Testing (system programmers’ play-pen) Critical Business Testing (clone of Production)

6

The Development Environment

CA Endevor Software Change Manager R12

Languages Static SQL

COBOL (mostly) PL/I Assembler C

Dynamic SQL Rexx

DB2 Data Elements Migration of DB2 static meta-data (e.g. business rules) from source

development environment through other environments, schemas, subsystems in the development lifecycle using CA Endevor, Rexx routines and DB2 Stored Procedures

Another presentation for another time! ;-)

7

The ChangesIn Production

Subsystem PackagesTotal Changes

per week

Average Statements per

package

DB2J 15,753 141 19

DB2P 13,462 90 18

DB2B 11,357 194 21

DB2L 60,016 459 20

100,588 884 19.5

8

The ChangesIn Development

Subsystem PackagesChanges per

weekStatements per

package

Statements checked per

week

DB2T 207090 542 14 7588

DB2V 218901 352 12 4224

NDVR 2394 879 12 10548

Development SubsystemDevelopment Subsystem

System Test/UAT SubsystemSystem Test/UAT Subsystem

Endevor StatisticsEndevor Statistics

9

The NeedThe Programmers

Constant high-volume of CICS/COBOL/DB2 changes per week

10


Constant high-volume of CICS/COBOL/DB2 changes per week Aggressive, ambitious timescales

11


Constant high-volume of CICS/COBOL/DB2 changes per week Aggressive, ambitious timescales Many big, highly-visible projects

12


Constant high-volume of CICS/COBOL/DB2 changes per week Aggressive, ambitious timescales Many big, highly-visible projects 100s of developers, both on- and off-shore

13


Constant high-volume of CICS/COBOL/DB2 changes per week Aggressive, ambitious timescales Many big, highly-visible projects 100s of developers, both on- and off-shore SQL-based Production Performance Problems due to

Lack of adherence to SQL Coding Standards Lack of SQL Coding Standards full stop! Lack of SQL Coding expertise Lack of DB2 performance experience Lack of experience with Explain Tools

Can’t force programmer to use them Even if you could (and believe me, I tried), there was no time or budget for

training General lack of Clue™

14


Constant high-volume of CICS/COBOL/DB2 changes per week Aggressive, ambitious timescales Many big, highly-visible projects 100s of developers, both on- and off-shore SQL-based Production Performance Problems due to

Lack of adherence to SQL Coding Standards Lack of SQL Coding Standards full stop! Lack of SQL Coding expertise Lack of DB2 performance experience Lack of experience with Explain Tools

Can’t force programmer to use them Even if you could (and believe me, I tried), there was no time or budget for

training General lack of Clue™

Can’t buy a bigger/faster machine

15

The NeedThe Database Administrators

Small heritage Lloyds TSB DBA team 2 System DBA/Sysprogs, 4 Application DBAs, 2 Contractors

16



DBAs sore from the effects of poor database design and inefficiently coded SQL:

Batch windows over-run Online transactions hit with bad responses Knock-on effects

Deadlocks/Timeouts Logging CPU hogging I/O thrashing

– You know the score!

17



DBAs sore from the effects of poor database design and inefficiently coded SQL:

Batch windows over-run Online transactions hit with bad responses Knock-on effects

Deadlocks/Timeouts Logging CPU hogging I/O thrashing

– You know the score! DBAs already too busy with application DBA work, systems

maintenance and production support to analyse each SQL statement individually

18

The Requirements“Every breath you take”

Thorough SQL Checking and Explain analysis required

19

The Requirements“And every move you make”

Thorough SQL Checking and Explain analysis required Automatic

20

The Requirements“Every bond you break”

Thorough SQL Checking and Explain analysis required Automatic Must be flexible and adaptable

Rules-based Able to specify and modify individual rules, dynamically Package-level, Statement-level, Collection-level, Schema-level, Table-

level

21

The Requirements“Every step you take”



level Must uniquely identify each discrete SQL statement from one

compile to the next Statement number may change even if statement itself is unchanged

22

The Requirements“I'll be watching you”





Log violations

23

The Requirements“Every single day”





Log violations Allow certain statements through

24

The Requirements“And every word you say”





Log violations Allow certain statements through Must prevent bad code from hitting Production

25

The Requirements“Every game you play, every night you stay”





Log violations Allow certain statements through Must prevent bad code from hitting Production Must support multiple development environments across multiple

schemas

26

The Requirements“I'll be watching you”






schemas Must be vendor independent

CA/BMC/Compuware accounts – here today, gone tomorrow, but the problem remains!

27

The ScenarioWorst Case

1. Programmer writes code2. SQL is poo3. Code gets put into Production4. Production grinds to a halt5. Manager invokes DBA6. DBA uses tools to analyse performance7. Manager lurks over DBA’s shoulder8. DBA identifies poo in SQL

New index needed, or New predicates needed

Forgot that leading index column again?

9. DBA creates index or Programmer changes code10.SQL is no longer poo11.Code gets put into Production (again)12.Production does not grind to a halt13.All’s well that ends well…this time

28

The ScenarioBest Case

1. Programmer writes code2. SQL is poo3. Programmer invokes DBA to analyse SQL performance4. DBA uses tools to analyse performance5. Programmer lurks over DBA’s shoulder6. DBA identifies poo in SQL



7. DBA creates index or Programmer changes code8. SQL is no longer poo9. Code gets put into Production10.Production does not grind to a halt11.All’s well that ends well

29

The ScenarioBest Case

1. Programmer writes code2. SQL is poo3. Programmer invokes DBA to analyse SQL performance4. DBA uses tools to analyse performance5. Programmer lurks over DBA’s shoulder6. DBA identifies poo in SQL



7. DBA creates index or Programmer changes code8. SQL is no longer poo9. Code gets put into Production10.Production does not grind to a halt11.All’s well that ends well

Can you afford to do that 10,548 times a week?

30

The Vendors (c. 2004)(N.B. My own personal opinions, blah, blah, blah…)

Candle DB!Explain Nice look and feel Easy to use Batch and Online

BMC SQL Explorer Adequate look and feel Adequate ease of use Batch and Online Much better now, though. LBG standard.

CA Plan Analyser Scary User Interface Never used it

None of them cut the mustard…

(Vendors, please don’t approach me at the end!)

31

The RequirementsA recap






schemas Must be vendor independent

To cater for the whims of the accounts department

32

The Solution“De do do do! De da da da!”

Enter SQL Police, courtesy of MDI Tools™!

A suite of Rexx, DB2 and ISPF programs to aid and abet the DBA in persecuting and preventing the perpetrators of poo SQL!

MDI Explain The heart of SQL Police Rules-based Statement Analysis and Costing Routine for DB2 packages

SQL Rules An ISPF application to maintain the Rules data base

“Fuzzmuck” An ISPF application to modify the SQL Police list of violations

33

The Endevor Process

On an ADD of an element (program) into Endevor Pre-compile Compile Link Bind MDI Explain

Reads PLAN_TABLE Reads SQL Police data base Rules table Analyses access path output in PLAN_TABLE Applies rules to each statement

If rules are broken, check to see if it’s an allowed violation If not, write a row to the violations table

If any rules have failed, fail the step If Explain step fails, fail the job and prevent further progression of

program through Endevor Programmer then checks output for statement violations

34

The MDI Explain Process Flow

Initialise stuff Perform statement-level rules checking: For each Statement in SYSPACKSTMT or SYSSTMT

Extract SQL into an explainable statement Prepare the statement to get the TIMERON cost Perform PLAN_TABLE row-level SQLRULES checking: For each row in PLAN_TABLE for that statement

For each qualifying row in SQLRULES Intepret conditions If statement fails,

Execute Rexx Check SQLPOLICE for corresponding row If a row exists, and it has been bypassed, set condition code to 4, otherwise set the

condition code to the corresponding value from the rule and insert a row into SQLPOLICE.

Perform SQLRULES check against statement (as above) Perform package-level SQLRULES checking Summarise reports Set corresponding condition code

35

The Clever Bit“Every little thing she does is magic”

SQL statement unchanged, yet COBOL lines inserted Statement number changes

SQL statement changed, no COBOL lines inserted Statement number doesn’t change

Remember the requirements? Must uniquely identify each discrete SQL statement from one compile

to the next Statement number may change even if statement itself is unchanged

Solution? Generate a CRC number for each statement It only changes if the statement changes If the statement changes, the statement needs re-evaluating anyway!

36

The TablesSQLPOLICE

Column Name Data Type Length Description

LOCATION CHAR 16 DB2 server where the package was bound – normally blanks.

COLLID CHAR 18 Name of the package collection. Normally of the format EndevorSystem_EndevorType.

PACKAGE CHAR 8 Name of the package.

VERSION VARCHAR 64 Version of the package – normally blanks.

DBRM CHAR 8 The DBRM name, if packages are not used.

STMT SMALLINT Statement number.

CRC CHAR 8 Hexadecimal CRC number for the SQL statement.

COST DECIMAL 15,1 Statement cost in Timerons.

TYPE CHAR 1 The type of failure recorded:I Informational Message (condition code 0)W Warning Message (condition code 2)E Error Message (condition code 8)S Severe Error Message (condition code 12)C Curious (unexpected error while parsing – condition code 0)

BYPASS CHAR 1 Values:N The statement has failed the rules.Y The statement has failed the rules, but has been bypassed and

returns a condition code of 4.

UPDATED TIMESTMP 10 The timestamp the record was bypassed.

UPDATOR CHAR 8 The user id used to bypass the record.

INSERTED TIMESTMP 10 The timestamp the record was added.

INSERTOR CHAR 8 The user id that inserted the record.

REASON VARCHAR 80 The reason the record was bypassed.

37

The TablesSQLRULES

Column Name Data Type Length Description

RULE_KEY CHAR 6 A unique identifier for the rule.

RULE_ACTIVE CHAR 1 Values:Y The rule is active and will be applied to the SQL. N The rule is inactive and will not be applied to the SQL.

RULE_LEVEL CHAR 1 Values:P The rule will be applied once, to the entire package. S The rule will be applied to each SQL statement in the package.R The rule will be applied to each row of the plan table – possibly several times per SQL statement.

RULE_TYPE CHAR 1 The type of diagnostic message that this rule returns to MDI Explain for the statement in question. It can have the following values: I Informational Message (condition code 0) W Warning Message (condition code 2) E Error Message (condition code 8) S Severe Error Message (condition code 12)O Package-level Override (condition code 0) Use the O option with care, as it sets the final condition code to zero, bypassing any previous statement failures.

RULE_DESC VARCHAR 50 A 50 byte description that will appear in the report against the statement (if it fails the test).

RULE_REXX VARCHAR 254 A free-format, 254 byte field that contains the condition, in Rexx, appearing between an IF and a THEN statement to be applied in MDINDVRX against each statement.

RULE_RECOMMEND VARCHAR 254 A free-format, 254-byte field that may contain information or a recommendation for further action.

RULE_EXECUTE VARCHAR 254 A free-format, 254-byte field that may contain Rexx code that is to be executed when the condition is met.

38

The RulesKeywords and Variables

Any Rexx variable or function can be used

Any PLAN_TABLE column name can be used Including the following that have been shortened: ACCDEG AT CFE COLLID COST INDEX IO JOIN KC LOCK MC

METHOD MIXOPSEQ NPAGES NUMTABS PACKAGE PLANNO PMODE PREFETCH QBLOCKNO SCG SCJ SCO SCU SNG SNJ SNO SNU STMT TABLE TABNO

Column RULE_REXX contains the Rexx code that contains the rule to be applied

Column RULE_EXECUTE contains the optional Rexx code that is to be executed when the rule is violated

39

The RulesThe ISPF Panel

File Help ------------------------------------------------------------------------------- SQLRULES Command ===> Scroll ===> CSR S Key Active Type Level Description - ------ ------ ---- ----- -------------------------------------------------- BIT000 N O P Exclude Non-BIT/CLP/FIG packages CLP000 N O P Exclude Non-CLP/FIG packages FIG001 N S S Cost > 10000 timerons FIG002 N E R Tablespace scan > 1000 pages. FIG003 N W R Less than 1/3rd of the key columns used. FIG004 N W R Only one matching column of many used FIG005 N W R Non-matching index scan > 1000 timerons FIG006 N W R Exclusive lock FIG009 N I R List prefetch FIG013 Y W S SELECT * found FIG014 N I P Test rule FIG016 Y W S COUNT found FIG017 Y I R Sequential prefetch LRB752 Y O P ALLOW DEVELOPMENT IN FIGE3 MDI001 Y S S Cost > 10000 timerons MDI002 Y E R Tablespace scan > 1000 pages MDI003 Y W R Less than 1/3rd of the key columns used

40

The RulesExample – Severe

Rule Key : MDI001 : (Unique key for each rule) Active : Y : (Y Yes, N No) Rule Type : S : (I Information, W Warning, E Error, S Severe, O Override) Rule Level : S : (P Package level, S Statement level, R Plan table Row level)Description : Cost > 10000 timerons : Rexx Code : cost > 10000 :Recommendation: Change your predicates to use an index, or define a new index. :Rexx to Execute: :

Rule Key : MDI001 : (Unique key for each rule) Active : Y : (Y Yes, N No) Rule Type : S : (I Information, W Warning, E Error, S Severe, O Override) Rule Level : S : (P Package level, S Statement level, R Plan table Row level)Description : Cost > 10000 timerons : Rexx Code : cost > 10000 :Recommendation: Change your predicates to use an index, or define a new index. :Rexx to Execute: :

This rule returns a Severe error and condition code 12 for any statement whose cost is greater than 1000 timerons. There is no further Rexx to execute for this rule.

This rule returns a Severe error and condition code 12 for any statement whose cost is greater than 1000 timerons. There is no further Rexx to execute for this rule.

41

The RulesExample – Error

Rule Key : MDI002 : (Unique key for each rule) Active : Y : (Y Yes, N No) Rule Type : E : (I Information, W Warning, E Error, S Severe, O Override) Rule Level : R : (P Package level, S Statement level, R Plan table Row level)Description : Tablespace scan > 1000 pages : Rexx Code : at="R" & npages > 1000 :Recommendation: Try using an index :Rexx to Execute: :

Rule Key : MDI002 : (Unique key for each rule) Active : Y : (Y Yes, N No) Rule Type : E : (I Information, W Warning, E Error, S Severe, O Override) Rule Level : R : (P Package level, S Statement level, R Plan table Row level)Description : Tablespace scan > 1000 pages : Rexx Code : at="R" & npages > 1000 :Recommendation: Try using an index :Rexx to Execute: :

This rule returns an Error (condition code 8) for any statement which sequentially scans a tablespace with more than 1000 pages. This rule returns an Error (condition code 8) for any statement which sequentially scans a tablespace with more than 1000 pages.

42

The RulesExample – Package Override

Rule Key : TEST01 : (Unique key for each rule) Active : N : (Y Yes, N No) Rule Type : O : (I Information, W Warning, E Error, S Severe, O Override) Rule Level : P : (P Package level, S Statement level, R Plan table Row level)Description : Exclude Non-Figaro packages : Rexx Code : Left(COLLID,3)¬="FIG" :Recommendation: Non-Figaro packages are not currently implemented. :Rexx to Execute: Signal Disconnecting :

Rule Key : TEST01 : (Unique key for each rule) Active : N : (Y Yes, N No) Rule Type : O : (I Information, W Warning, E Error, S Severe, O Override) Rule Level : P : (P Package level, S Statement level, R Plan table Row level)Description : Exclude Non-Figaro packages : Rexx Code : Left(COLLID,3)¬="FIG" :Recommendation: Non-Figaro packages are not currently implemented. :Rexx to Execute: Signal Disconnecting :

This rule is package-based and is not active. If the collection of the package being explained does not begin with “FIG”, then the Override flag is set, forcing a condition code of 0, and the statement “Signal Disconnecting” is executed, forcing the program flow to jump immediately to the label “Disconnecting:” whereupon the process disconnects from DB2, cleans up and terminates.

This rule is package-based and is not active. If the collection of the package being explained does not begin with “FIG”, then the Override flag is set, forcing a condition code of 0, and the statement “Signal Disconnecting” is executed, forcing the program flow to jump immediately to the label “Disconnecting:” whereupon the process disconnects from DB2, cleans up and terminates.

43

The RulesExample – Information

Rule Key : MDI013 : (Unique key for each rule) Active : Y : (Y Yes, N No) Rule Type : I : (I Information, W Warning, E Error, S Severe, O Override) Rule Level : R : (P Package level, S Statement level, R Plan table Row level)Description : BFM table is being accessed : Rexx Code : Left(table,3)="BFM" :Recommendation: Have you informed Support? :Rexx to Execute: Call EMLSUP :

Rule Key : MDI013 : (Unique key for each rule) Active : Y : (Y Yes, N No) Rule Type : I : (I Information, W Warning, E Error, S Severe, O Override) Rule Level : R : (P Package level, S Statement level, R Plan table Row level)Description : BFM table is being accessed : Rexx Code : Left(table,3)="BFM" :Recommendation: Have you informed Support? :Rexx to Execute: Call EMLSUP :

This is a statement-based rule which returns a condition code of 0, but issues an informational message if the statement accesses a table beginning with “BFM”. It also calls an external routine to email the Support group.

This is a statement-based rule which returns a condition code of 0, but issues an informational message if the statement accesses a table beginning with “BFM”. It also calls an external routine to email the Support group.

44

The RulesExample – Warning

Rule Key : MDI003 : (Unique key for each rule) Active : Y : (Y Yes, N No) Rule Type : W : (I Information, W Warning, E Error, S Severe, O Override) Rule Level : R : (P Package level, S Statement level, R Plan table Row level) Description : Less than 1/3rd of the key columns used : Rexx Code : mc<kc/33 : Recommendation: Try matching on more key columns :Rexx to Execute: :

Rule Key : MDI003 : (Unique key for each rule) Active : Y : (Y Yes, N No) Rule Type : W : (I Information, W Warning, E Error, S Severe, O Override) Rule Level : R : (P Package level, S Statement level, R Plan table Row level) Description : Less than 1/3rd of the key columns used : Rexx Code : mc<kc/33 : Recommendation: Try matching on more key columns :Rexx to Execute: :

This issues a warning message and a condition code of 2 if any index accessed by the statement uses less than a third of available key columns.This issues a warning message and a condition code of 2 if any index accessed by the statement uses less than a third of available key columns.

45

The InvocationThe Statement

MDINDVRX

SSID(DB2T) DIAG(diagnostics keywords)

DBRM(dbrm)

VERSION(version)

LOCATION(locn)

COLLID(collection) PACKAGE(package)

STMT(stmtno)

BIND OWNER(owner) QUAL(qualifier)

LIBRARY(dbrmlib)

46

The InvocationThe JCL

//PACK1 EXEC PGM=IKJEFT01,REGION=32M //STEPLIB DD DISP=SHR,DSN=G370250.GEN.LOAD //DBRMLIB DD DISP=SHR,DSN=NDVR.FIG0.STAGEA.DBRMLIB//SYSTSPRT DD SYSOUT=*//SYSTSIN DD * MDINDVRX BIND DIAG(EXPLAIN SQLCA) – COLLID(FIGARO_COBOLFIG) PACKAGE(NDB020) OWNER(NDVR) – QUAL(FIG£1) /*

//PACK1 EXEC PGM=IKJEFT01,REGION=32M //STEPLIB DD DISP=SHR,DSN=G370250.GEN.LOAD//SYSTSPRT DD SYSOUT=* //SYSTSIN DD * MDINDVRX BIND DIAG (EXPLAIN SQLCA) - COLLID(FIGARO_COBOLFIG) PACKAGE(NDB020) OWNER(NDVR) - QUAL(FIG£1) LIBRARY(NDVR.FIG0.STAGEA.DBRMLIB) /*

//PACK1 EXEC PGM=IKJEFT01,REGION=32M //STEPLIB DD DISP=SHR,DSN=G370250.GEN.LOAD//SYSTSPRT DD SYSOUT=* //SYSTSIN DD * MDINDVRX BIND DIAG (EXPLAIN SQLCA) - COLLID(FIGARO_COBOLFIG) PACKAGE(NDB020) OWNER(NDVR) - QUAL(FIG£1) LIBRARY(NDVR.FIG0.STAGEA.DBRMLIB) /*

With DBRMLIB DDWith DBRMLIB DD

Without DBRMLIB DDWithout DBRMLIB DD

Binds package into a temporary collection to read off its PLAN_TABLE rows

Binds package into a temporary collection to read off its PLAN_TABLE rows

47

The InvocationDiagnostic Keywords

Keyword Description and Action

EXTRA or DIAG Displays extra diagnostic messages describing the current action of the program.

WIDE or SCREEN Display the output in 133-byte mode. The default output is 80 byte for online invocation, and 133 for batch.

EXPLAIN Display the rows of the PLAN_TABLE for each statement.

RULES Display the RULE_KEY and IF qualifier for each rule.

FOOZWAK Switch TRACE ?I on in procedure FOOZWAK: where an explainable SQL statement is extracted from SYSPACKSTMT. (Occasionally, problems occur while parsing.)

ERROR or SQLCA Switches on Call On Error Name SQLCA that displays the SQLCA for every negative SQLCODE returned.

TRACE Switch full TRACE ?I on from the top.Warning: Causes loads of output. And I mean LOADS.

PROBLEM or OFF If either keyword is specified, MDINDVRX will terminate with condition code 1. This is useful to switch off MDINDVRX while maintenance is being performed.

48

The InvocationDiagnostics Data Set

Menu Functions Utilities Help ------------------------------------------------------------------------------ VIEW NDVR.MDINDVRX Row 00001 of 00002 Command ===> Scroll ===> CSR Name Prompt Size Created Changed ID . EXPLAIN 1 2006/04/21 2006/04/21 09:59:58 G202013 . TRACE 1 2008/03/10 2008/03/10 16:03:32 G202013 **End**

Menu Functions Utilities Help ------------------------------------------------------------------------------ VIEW NDVR.MDINDVRX Row 00001 of 00002 Command ===> Scroll ===> CSR Name Prompt Size Created Changed ID . EXPLAIN 1 2006/04/21 2006/04/21 09:59:58 G202013 . TRACE 1 2008/03/10 2008/03/10 16:03:32 G202013 **End**

Program checks for a diagnostics data set on start-up Data set members (if any) have the name of the diagnostic

keywords that you want to switch on

49

The ExecutionExample 1

17 Mar 2006 14:24:53 MDI Explain =========== Rules-based Statement Analysis and Costing Routine for DB2 packages under CA/Endevor control (c)2002 Marcus Davage Incorporated. mailto: [email protected] Last Compiled: 2006-02-14-12.17.48.365977 MARCUS.DAVAGE is running MDINDVRX under G205013(TSU27090) in FOREground mode on NG00. Arguments passed: SSID(DB2T) BIND COLLID(G370250) LIBRARY(NDVR.FIG0.STAGEP.DBRMLIB) PACKAGE(HYE011) OWNER(G370250) QUAL(FIGE0)Diagnostics Data Set NDVR.MDINDVRX does not exist. DBRMLIB not allocated. Statement breakdown for G370250.HYE011.() in DB2T... Processing statement: 4953: SELECT COMPANY, CLIENT, NAMESEQ, CLTTRNNO FROM NA Cost: 35.4 Timerons Processing statement: 4990: SELECT COMPANY, CLIENT, NAMESEQ, CLTTRNNO, OFFSET Cost: 35.5 Timerons Information: Statement 4990 met Rule MDI009 (List prefetch). Recommendation: List prefetch is used. Full SQL Statement: SELECT COMPANY, CLIENT, NAMESEQ, CLTTRNNO, OFFSETNO FROM NAME WHERE COMPANY = ? AND UCSURNAME = ? AND UCFSTNAME BETWEEN ? AND ? AND INITIALS BETWEEN ? AND ? AND OFFSETNO > 0 AND ( ( CLIENT = ? AND NAMESEQ = ? AND CLTTRNNO > ? ) OR ( CLIENT = ? AND NAMESEQ > ? ) OR ( CLIENT > ? ) ) ORDER BY COMPANY, CLIENT, NAMESEQ, CLTTRNNO

50


Processing statement: 5756: SELECT A.COMPANY, A.CLIENT, A.NAMESEQ, A.CLTTRNNO Cost: 16854.2 Timerons Severe: Statement 5756 met Rule MDI001 (Cost > 10000 timerons). Recommendation: Change your predicates to use an index, or define a new index. Information: Statement 5756 met Rule MDI009 (List prefetch). Recommendation: List prefetch is used. Full SQL Statement: SELECT A.COMPANY, A.CLIENT, A.NAMESEQ, A.CLTTRNNO, A.OFFSETNO, B.ADDRREF, B.CLTTRNNO, B.OFFSETNO FROM NAME A, ADDRREF B WHERE A.COMPANY = ? AND UCSURNAME BETWEEN ? AND ? AND B.POSTCODE BETWEEN ? AND ? AND A.OFFSETNO > 0 AND A.COMPANY = B.COMPANY AND A.CLIENT = B.CLIENT AND B.OFFSETNO> 0 AND ( ( A.CLIENT = ? AND A.NAMESEQ = ? AND A.CLTTRNNO = ? AND B.ADDRREF = ?AND B.CLTTRNNO > ? ) OR ( A.CLIENT = ? AND A.NAMESEQ = ? AND A.CLTTRNNO = ? ANDB.ADDRREF > ? ) OR ( A.CLIENT = ? AND A.NAMESEQ = ? AND A.CLTTRNNO > ? ) OR ( A.CLIENT = ? AND A.NAMESEQ > ? ) OR ( A.CLIENT > ? ) ) ORDER BY A.COMPANY, A.CLIENT, A.NAMESEQ, A.CLTTRNNO, B.ADDRREF, B.CLTTRNNO

51


Processing statement: 5970: SELECT A.COMPANY, A.CLIENT, A.NAMESEQ, A.CLTTRNNO Cost: 32031.2 Timerons Severe: Statement 5970 met Rule MDI001 (Cost > 10000 timerons). Recommendation: Change your predicates to use an index, or define a new index. Full SQL Statement: SELECT A.COMPANY, A.CLIENT, A.NAMESEQ, A.CLTTRNNO, A.OFFSETNO, B.ADDRREF, B.CLTTRNNO, B.OFFSETNO FROM NAME A, ADDRREF B WHERE A.COMPANY = ? AND UCSURNAME BETWEEN ? AND ? AND B.UCCITY BETWEEN ? AND ? AND A.OFFSETNO > 0 AND A.COMPANY = B.COMPANY AND A.CLIENT = B.CLIENT AND B.OFFSETNO> 0 AND ( ( A.CLIENT = ? AND A.NAMESEQ = ? AND A.CLTTRNNO = ? AND B.ADDRREF = ?AND B.CLTTRNNO > ? ) OR ( A.CLIENT = ? AND A.NAMESEQ = ? AND A.CLTTRNNO = ? ANDB.ADDRREF > ? ) OR ( A.CLIENT = ? AND A.NAMESEQ = ? AND A.CLTTRNNO > ? ) OR ( A.CLIENT = ? AND A.NAMESEQ > ? ) OR ( A.CLIENT > ? ) ) ORDER BY A.COMPANY, A.CLIENT, A.NAMESEQ, A.CLTTRNNO, B.ADDRREF, B.CLTTRNNO Processing statement: 7221: SELECT COMPANY, CLIENT, NAMESEQ, CLTTRNNO, OFFSET Cost: 30.8 Timerons Warning: Statement 7221 met Rule MDI004 (Only one matching column of many used). Recommendation: Try providing more indexable predicates Full SQL Statement: SELECT COMPANY, CLIENT, NAMESEQ, CLTTRNNO, OFFSETNO FROM NAME WHERE COMPANY = ? AND FZSURNAME = ? AND UCFSTNAME BETWEEN ? AND ? AND INITIALS BETWEEN ? AND ? AND OFFSETNO > 0 AND ( ( CLIENT = ? AND NAMESEQ = ? AND CLTTRNNO > ? ) OR ( CLIENT = ? AND NAMESEQ > ? ) OR ( CLIENT > ? ) ) ORDERBY COMPANY, CLIENT, NAMESEQ, CLTTRNNO

52


Breakdown Summary: ------------------ 5 Information messages returned a condition code of 1 and passed. 2 Warning messages returned a condition code of 2 and passed. 6 Severe error messages returned a condition code of 12 and failed. If you think further action is necessary, please contact your Systems Administrator.Otherwise, use an Explain Tool to help you correct the problem.

Package-level Rules: -------------------- None apply. MDI Explain will now exit with condition code 12. Thank you.

53

The ResultExample 2 – Before

RULE T PACKAGE STMT X COST DETAILS ------- - -------- ------ - ---------- -------------------------------------------------MDI001S S HDB550 5100 N 23347.2 Cost > 10000 timerons; Statement cost = 23347.2 Breakdown Summary: ------------------ 1 Severe messages returned a condition code of 12 and failed. If you think further action is necessary, please contact your Database Administrator. MDX998I MDI Explain will now exit with condition code 12. MDX999I Thank you.

54

The Result“Message in a bottle”

When a package/DBRM statement fails a rule a row is inserted into the Violations table MDI Explain step fails Code prevented from going any further in Endevor

Programmer investigates error and output If he/she knows what’s good for him/her Explain tools used to diagnose problem BMC SQL Explorer used in LBG

Much better GUI nowadays Comprehensive/Rich functionality Really funky “what if” functionality

Tweak statements to see what happens Clone/ and tweak indexes to see what happens

If nothing can be done, statement violation is overridden E.g. Known batch jobs that tablespace-scan

55

The SQL PoliceFuzzMuck first panel

File ToolBox Edit Edit_Settings Menu Utilities Compilers Test Help+---------------------------------- FuzzMuck ----------------------------------+| || Select an object (PF04 to prompt) || || DB2 Subsystem : DB2T : || || Location : : || Collection : : || Package Name : : || Version : : || || or || || DBRM : : |+------------------------------------------------------------------------------+-------------------------------------------------------------+ | Welcome to FuzzMuck. Tread carefully. Think twice. Be sure. | +-------------------------------------------------------------+

56

The SQL PolicePrompt for collection id

File ToolBox Edit Edit_Settings Menu Utilities Compilers Test Help+---------------------------------- FuzzMuck ----------------------------------+| || Select an object (PF04 to prompt) || || DB2 Subsyst +------- Select a COLLID --------+ || | Row 1 to 8 of 8 | || Location : | Command> | || Collection | | || Package Nam | . CANDLE_TE_SYSIBM | || Version : _ | S FIGARO_COBOLFIG | ___________________________ : || | . FIGE0 | || | . FIGE1 | || | . FIGE2 | || DBRM : ____ | . FIGE3 | |+------------ | . FIGE4 | ------------------------------+ | . G370250 | | ******* Bottom of data ******* | | | +--------------------------------+

57

The SQL PolicePrompt for package id

File ToolBox Edit Edit_Settings Menu Utilities Compilers Test Help+---------------------------------- FuzzMuck ----------------------------------+| || Select an object (PF04 to prompt) || || DB2 Subsyst +------- Select a PACKAGE -------+ || | Row 1 to 2 of 2 | || Location : | Command> | || Collection | | || Package Nam | S HDB550 | || Version : _ | . NDB020 | ___________________________ : || | ******* Bottom of data ******* | || | | || | | || DBRM : ____ | | |+------------ | | ------------------------------+ | | | | | | +--------------------------------+

58

The SQL PoliceMain panel

File Explain Help ------------------------------------------------------------------------------- SQL Police Exclude List Command ===> Scroll ===> CSR Options: Select, EXclude, Reset, Delete, Test Object: .FIGARO_COBOLFIG.HDB550.() (Package) S Statemnt CRC Cost (Timerons) Exc Typ Reason - -------- -------- --------------- --- --- ----------------------------------- 5100 31DFB98E 18988.2 N S ******************************* Bottom of data ********************************

59

The SQL PoliceExcluding a statement

File Explain Help ------------------------------------------------------------------------------- SQL Police Exclude List Command ===> Scroll ===> CSR Options: Se +------------- Statement 5100 --------------+ | Why are you allowing this statement? | Object: .F | On Gordon Sumner’s approval | | | S Statemnt +-------------------------------------------+ - -------- -------- --------------- --- --- ----------------------------------- X 5100 31DFB98E 18988.2 N S ******************************* Bottom of data ********************************

60

The SQL PoliceExcluded statement

File Explain Help ------------------------------------------------------------------------------- SQL Police Exclude List Command ===> Scroll ===> CSR Options: Select, EXclude, Reset, Delete, Test Object: .FIGARO_COBOLFIG.HDB550.() (Package) S Statemnt CRC Cost (Timerons) Exc Typ Reason - -------- -------- --------------- --- --- ----------------------------------- 5100 31DFB98E 18988.2 Y S On Gordon Sumner's approval ******************************* Bottom of data ********************************

61

The ResultExample 2 – After

RULE T PACKAGE STMT X COST DETAILS

------- - -------- ------ - ---------- ----------------------------------------------------------MDI001S S HDB550 5100 Y 23347.2 Cost > 10000 timerons; Statement cost = 23347.2

Breakdown Summary:

------------------

1 message has been issued against allowed statements that have returned a condition code of 4

and passed.

If you think further action is necessary, please contact your Database Administrator.

MDX998I MDI Explain will now exit with condition code 4.

MDX999I Thank you.

62

The Result“Don’t stand so close to me”

All tables are logged using DB2 triggers to audit table Insert Before After Delete

All Overrides must be accompanied by Project Manager approval A statement override once brought production to a halt The DBA who overrode it was fingered Once bitten, twice shy

63

The ResultNow with added scrutiny

SubsystemTotal Number

of Packages

Total Changes per week

Average Statements

per package

Statements checked

per weekStatements

bypassed

Statements bypassed per week

DB2J 15753 141 19 2679

DB2P 13462 90 18 1620

DB2B 11357 194 21 4074

DB2L 60016 459 20 9180

Total in Production 100588 884 19.5 17553

DB2T 207090 542 14 7588

DB2V 218901 352 12 4224

NDVR 2394 879 12 10548 894 4

64

The ResultThe Savings

Existing Bad SQL Identified & Exterminated

Indexes Created or Dropped

Production CPU bill cut Actual savings realised £2m p.a. for one batch job alone!

DBA Effort reduced Between ½ hour to ½ day per statement – saved

Everyone’s happy!

65

The EndThe Future

Access path changes What happens if you drop an index, but don’t change a previously excluded

statement? Statement becomes an even-poorer-performer.

New Explain tables Plan tables to be aliases of DB2OSC.* tables PLAN_TABLE DSN_DETCOST_TABLE DSN_FILTER_TABLE DSN_FUNCTION_TABLE DSN_PGRANGE_TABLE DSN_PGROUP_TABLE DSN_PREDICAT_TABLE DSN_PTASK_TABLE DSN_QUERY_AUX DSN_QUERY_TABLE DSN_SORTKEY_TABLE DSN_SORT_TABLE DSN_STATEMENT_CACHE_AUX DSN_STATEMENT_CACHE_TABLE DSN_STATEMNT_TABLE DSN_STRUCT_TABLE DSN_VIEWREF_TABLE DSN_VIRTUAL_INDEXES

66

The EndThe End

“There has to be an invisible sunIt gives its heat to everyone”

Questions?

How many Police references did you see?

67

The EndNo, really I mean it this time

Session: CHUK GSE 2010

Marcus Davage MBCS CEng CITP

DB2 Database AdministratorLloyds Banking Group

Email: [email protected]: http://twitter.com/spufidooLinkedIn: http://uk.linkedin.com/in/marcusdavageTel: 07725427737

mailto:[email protected]

http://twitter.com/spufidoo

http://uk.linkedin.com/in/marcusdavage

SQL Police

Technology

coded sql

cicscoboldb2 changes

lack of adherence

week aggressive

db2 subsystems

programmers constant

week statements

db2 v7