DB2 for z/OS Stored Procedures “In the Mainstream” Robert Catterall, President Michigan DB2 Users Group December 6, 2007 © 2007 Catterall Consulting
Jan 21, 2015
DB2 for z/OS Stored Procedures“In the Mainstream”
Robert Catterall, President
Michigan DB2 Users GroupDecember 6, 2007
© 2007 Catterall Consulting
Agenda
• Stored procedures – the future of server-side SQL• What happens when a stored procedure is called• Creating and coding stored procedures• Deploying stored procedures• Some stored procedure odds and ends
2
Acknowledgement
• Peggy Zagelow, Senior Technical Staff Member at the IBM Silicon Valley Lab and development lead for DB2 for z/OS stored procedures, has answered many questions I’ve sent her way.
3
Stored Procedures – The Future of Server-Side SQL
Stored Procedures – The Future of Server-Side SQL
4
Traditional z/OS Server-Side SQL (1)
• Online: embedded in CICS or IMS/TM (IMS Transaction Manager) transaction programs◦ Why?
1. A great many DB2 for z/OS sites also have CICS or IMS/TM
2. It works: fast, reliable, CPU-efficient
3. DB2 for z/OS stored procedure functionality wasn’t even available until Version 4 (mid-1990s)
4. DB2 stored procedures initially lacked key features: no ability to fetch rows from a cursor in a stored procedure (addressed with V5) and no DDL support (provided with V6)
5. Many mainframers have an early 1990s mind set when it comes to anything involving DDF (“Too much overhead,” “Can’t effectively monitor it,” etc.)
5
Traditional z/OS Server-Side SQL (2)
• Batch: SQL embedded in programs that run in JES initiator address spaces◦ Often used to process large files received by organizations
from clients and/or suppliers◦ Often thought of as the only feasible way to process bulk
DB2 database updates, especially when deadlines are tight◦ “JES comes with z/OS – why use anything else for batch
work?”
6
Stored Procedures’ Time Has Come (1)
• Off-mainframe client-side application programmers love ‘em◦ They are already familiar with the stored procedure
programming model (use of stored procedures is a best practice in SQL Server and Oracle environments)
◦ Through the DB2 client and DB2 Connect, they can invoke DB2 for z/OS stored procedures using database interfaces with which they are familiar, including:
ADO.NET JDBC (Java Database Connectivity)
◦ They’d rather deal with query result sets as such, rather than deal with something like a CICS COMMAREA
7
Stored Procedures’ Time Has Come (2)
• For the skeptics on the z-side:◦ It is possible to run a high-volume online workload (1000+
transactions per second) using DB2 stored procedures invoked by off-mainframe programs (organizations are already doing so)
◦ Stored procedures provide the CPU efficiency and data security benefits of static SQL (authids require only the EXECUTE privilege on the stored procedure program and package)
◦ Stored procedure programs can be written in COBOL They are a great way to leverage both COBOL (on-mainframe) and
Java (or C# or whatever is used on the app server) programming skills for one application project
8
Stored Procedures’ Time Has Come (3)
• More for the z-folks:◦ In-depth stored procedure monitoring capability is here now,
with several products available from several major vendors◦ Package-level DB2 accounting information was enhanced with
DB2 V8 to include SQL, buffer pool, and locking counters (important when stored procedures invoked via DDF – no plan)
◦ WLM-managed stored procedure address spaces provide CICS AOR-like benefits (WLM = Workload Manager, part of z/OS):
You can have several of them (useful for isolation by application, and/or to temporarily isolate new stored procedures)
You get 2-phase commit capability, just as you do with CICS DB2 attach (coordinated commit and rollback of, for example, DB2 table and MQ queue changes made in a single unit of work)
9
What About Batch?
• Of course you can call a DB2 stored procedure from a batch program (any program that can issue an SQL statement can call a stored procedure)
• Here’s an idea: think bigger – think about replacing traditional batch processing of files with a combination of MQ and DB2 stored procedures◦ First step: a program reads an input file and places individual
records on an MQ queue◦ Message hitting queue triggers stored procedure call, with
message as input to stored procedure◦ Stored procedure makes requisite database changes,
provides requisite output (if needed)
10
More on “Batch Transactionalization”
• Throughput delivered via parallelization (multiple concurrently executing stored procedure programs)
• Requires no changes on the part of organizations that send files to you or receive files from you◦ But it lets you to talk to a file sender about sending a record at a
time to a queue that you externalize via a Web service (file sender might really like that)
◦ You could similarly send an output record at a time to a company that currently receives files from you
• Implementation details beyond the scope of this session◦ But you can get a lot of good info from Chapter 33 of the DB2
for z/OS V8 Application Programming and SQL Guide
11
What Happens When a Stored Procedure is Called
What Happens When a Stored Procedure is Called
12
Stored Procedure Processing Flow (1)
• First, you need a DB2 thread◦ Local applications (on-mainframe): created when first SQL
statement issued by program (first statement could be a CALL)◦ Remote clients (off-mainframe): created when CONNECT
statement issued (application gets a DDF thread – aka a DBAT)
• Next, CALL statement is issued by program◦ Format: CALL MYPROC (:invar1, :invar2, :outvar1, :outvar2)◦ Stored procedure name, input/output parameters passed to
DB2 Remote clients: via DB2’s Distributed Data Facility (DDF) Local applications: via same Program Call (PC) used for other SQL
statements (PC is a cross-memory address space-switching function provided by z/OS)
13
Stored Procedure Processing Flow (2)
• Upon receiving the CALL, DB2 looks in SYSROUTINES catalog table for information about the stored procedure, including:◦ Load module name (EXTERNAL_NAME)◦ Name of WLM application environment to be used in executing
the stored procedure program (WLM_ENVIRONMENT)◦ Once retrieved from the catalog, this information is placed in a
cache for quick access when needed again
• DB2 tells the WLM-managed address space to execute the stored procedure program
14
Stored Procedure Processing Flow (3)
• WLM-managed address space uses LE (Language Environment, another part of z/OS) to load and execute the stored procedure◦ A Java stored procedure will be loaded by way of Unix System
Services, but it will be executed via LE (there is a JVM under LE)
• Control is passed to the stored procedure program, along with associated input and output parameters◦ The calling program’s task (TCB or SRB) is suspended while the
stored procedure’s TCB uses the calling program’s DB2 thread◦ Stored procedure will PC (Program Call) into DB2 address space
for SQL statement execution (same as any mainframe program)◦ Before terminating, stored procedure assigns values to output
parameters
15
Stored Procedure Processing Flow (4)
• Control passes back to DB2◦ DB2 copies output parameters received from stored procedure
to client application parameter area
• Control passes back to client application◦ If the stored procedure program declares a cursor with the
option WITH RETURN, and OPENs the cursor (and doesn’t close it), the rows in the query result set will be returned to the calling program when the stored procedure program ends
16
Creating and Coding Stored Procedures
Creating and Coding Stored Procedures
17
Get the Definition Right
• A stored procedure is defined to DB2 via the CREATE PROCEDURE statement
• There are a lot of options that can be specified in a CREATE PROCEDURE statement – I’ll cover a few of these
18
CREATE PROCEDURE Options (1)
• ASUTIME◦ Specifies the total amount of processor time, in CPU service
units, that can be consumed by an invocation of the stored procedure
◦ It’s a good idea to specify a non-zero value for ASUTIME (the default value is zero)
Otherwise, if a stored procedure program started running really long, the maximum TCBs specified for a stored procedure address space could be reached, and WLM could fire up more instances of the address space
Recent z/OS enhancements will ensure that the system doesn’t choke on stored procedures as a result, but it would still be good if the system were to kill extra-long-running stored procedures
19
CREATE PROCEDURE Options (2)
• STAY RESIDENT◦ Indicates whether or not a stored procedure load module is to
remain resident in memory when the stored procedure ends◦ Specify STAY RESIDENT YES to improve execution
performance (though this is ignored for a Java stored procedure) There was a time when people didn’t want to do this, due to concern
about the disruptive effect of refreshing the memory of a stored procedure address space when a new version of a load module needed to be substituted for the old one
That was (as I recall) a legitimate concern some years ago; now, refreshing the memory of a WLM-managed stored procedure address space is a non-disruptive operation if WLM is operating in goal mode
20
CREATE PROCEDURE Options (3)
• PROGRAM TYPE◦ Indicates whether a stored procedure will run as a main routine
or as a subroutine◦ Specify PROGRAM TYPE SUB to boost CPU efficiency
Recall that a program written to run as a subroutine has to do some things that Language Environment would do for a main program, including initialization and clean-up work
◦ A nice DB2 V8 assist for COBOL stored procedures defined with PROGRAM TYPE SUB and STAY RESIDENT YES that use stored procedure parameters as host variables:
Set the SQL-INIT-FLAG variable (generated by the DB2 precompiler) to 0
Result: the SQLDA will be updated with the current addresses
21
CREATE PROCEDURE Options (4)
• COMMIT ON RETURN◦ Indicates whether or not DB2 will commit a transaction
immediately following the end of stored procedure execution◦ Personally, I prefer COMMIT ON RETURN NO (let the calling
application control the commit scope)◦ I’d be pretty careful about using COMMIT ON RETURN YES:
Could lead to premature committing of an application’s data changes
Requires the use of the WITH HOLD option for cursors that will be used to return result sets to calling program
22
Coding Stored Procedures
• Language choices for external stored procedures: COBOL, C, C++, REXX, Assembler, PL/I, and Java◦ Keep in mind that there is no need for the language of a stored
procedure program to match the language of a calling program
• An increasingly interesting stored procedure development language option: SQL◦ A SQL procedure contains the program source within the CREATE
PROCEDURE statement◦ Statements that can be used in the body of a SQL procedure
include GOTO, IF, ITERATE, LOOP, REPEAT, and WHILE So, program logic can be fairly sophisticated
23
More on SQL Procedures
• DB2 V8 (and prior releases): DB2 generates a C program from the CREATE PROCEDURE statement when LANGUAGE SQL is specified◦ The SQL procedure executes pretty much as does an external
DB2 stored procedure
• DB2 V9 New Function Mode introduces a big change:◦ A C program is not generated from the CREATE PROCEDURE
statement when LANGUAGE SQL is specified Instead, the executable form of the SQL procedure is stored in the
DB2 Directory, and the program – when called – runs in the DB2 Database Services Address Space
24
The zIIP Factor
• DB2 for z/OS Version 9 in New Function Mode:◦ A native SQL procedure – if called from an off-mainframe
requester via DDF – can execute on a zIIP engine◦ Result could be a nice reduction in TCO, as zIIP engines –
while not free – do not factor into mainframe software pricing
• My advice: start looking into SQL procedures now, if you haven’t already done so
25
A Question of Calls
• OK, back to external stored procedures• Suppose you have a stored procedure program ABC,
written in COBOL• Suppose ABC will call program XYZ, also written in COBOL• Should program ABC:
A. CALL program XYZ as a nested stored procedure?
B. Call program ABC as a subroutine via COBOL CALL?
• If you want peak CPU efficiency, the answer is B◦ That way, you have just program ABC’s TCB, versus a TCB for the
top-level stored procedure and each nested stored procedure
26
An OPTIMIZE FOR “Gotcha?” (1)
• You’re probably familiar with the OPTIMIZE FOR n ROWS clause for a DB2 SELECT statement◦ What it does: tells the DB2 optimizer to choose the data access
path that will return the first “n” rows of the result set as quickly as possible
◦ Often, people code OPTIMIZE FOR 1 ROW to get DB2 to use an index in favor of doing a sort operation (e.g., for ORDER BY)
n=1 makes the point to DB2, so why make n>1?
• Potential problem with OPTIMIZE FOR 1 ROW in a stored procedure (see next slide)
27
An OPTIMIZE FOR “Gotcha?” (2)
• Scenario:◦ DB2 V8 Client at FixPack 15 (this is the code that runs where
DB2-bound SQL statements originate on off-mainframe servers)◦ DB2 for z/OS Version 8 in Compatibility Mode
• If OPTIMIZE FOR n ROWS specified for a cursor-based SELECT in a stored procedure, DB2 will send “n” rows in each network transmission to the client◦ “Old” behavior: DB2 would send “n” rows in first transmission to
client; after that, rows sent in 32K blocks◦ Recommendation: leave out OPTIMIZE FOR n ROWS when
coding DECLARE CURSOR in a stored procedure program DB2 will then use block fetch for network efficiency
28
Deploying Stored ProceduresDeploying Stored Procedures
29
WLM Application Environments
• Created by way of a WLM panel• One of the things that can be set on this panel is NUMTCB
◦ Specifies the maximum number of concurrently executing stored procedures for an address space associated with the WLM execution environment
◦ DB2 V7 and prior: when NUMTCB reached, WLM could dynamically fire up another instance of the address space to handle demand
◦ DB2 V8 and beyond: NUMTCB is a ceiling for an address space, but WLM manages tasks and can start additional instances of an address space without NUMTCB being reached
• Reasonable value for NUMTCB: something in the range of 40 to 50
30
A Little More on WLM Environments
• For PROGRAM TYPE SUB stored procedures, written in COBOL, that have different run-time options (specified in CREATE PROCEDURE, passed to LE)◦ Recommendation: separate the stored procedures with
different run-time options by WLM application environment
31
Some Stored Procedure Odds & Ends
Some Stored Procedure Odds & Ends
32
Stored Procedure CALL Cost
• It takes about 30,000 instructions to invoke an external stored procedure
• This number will be less (don’t yet know how much less) for a DB2 V9 NFM native SQL procedure◦ Not surprising, since this type of stored procedure will
execute in the DB2 DBM1 address space
33
Helping DB2 Find the Package
• New special register in DB2 V8, CURRENT PACKAGEPATH, can do for a stored procedure invoked via DDF what PKLIST does for a PLAN◦ Recall that a generic plan is used for all requests coming in
through DDF◦ Calling program can issue SET CURRENT PACKAGEPATH
• In DB2 for z/OS V9, PACKAGEPATH can be specified on CREATE PROCEDURE for external procedures
34
How Expensive is Java?
• A Java stored procedure will use 20-50% more CPU than a COBOL stored procedure that does the same thing◦ If that sound like a lot, consider that the ratio was probably
more like 2:1 a few years ago◦ Java performance has been significantly enhanced on the
mainframe◦ Parity with C?
35