September 16, 2010 Building and Managing Efficient data access to DB2 Vijay Bommireddipalli, [email protected] Solutions Architect, Optim
September 16, 2010
Building and Managing Efficient data access to DB2
Vijay Bommireddipalli, [email protected] Architect, Optim
Information Management
© 2010 IBM Corporation2
Disclaimer
THE INFORMATION CONTAINED IN THIS PRESENTATION IS PROVIDED FOR INFORMATIONAL PURPOSES ONLY. WHILE EFFORTS WERE MADE TO VERIFY THE COMPLETENESS AND ACCURACY OF THE INFORMATION CONTAINED IN THIS PRESENTATION, IT IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. IN ADDITION, THIS INFORMATION IS BASED ON IBM’S CURRENT PRODUCT PLANS AND STRATEGY, WHICH ARE SUBJECT TO CHANGE BY IBM WITHOUT NOTICE. IBM SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT OF THE USE OF, OR OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION. NOTHING CONTAINED IN THIS PRESENTATION IS INTENDED TO, OR SHALL HAVE THE EFFECT OF:
CREATING ANY WARRANTY OR REPRESENTATION FROM IBM (OR ITS AFFILIATES OR ITS OR THEIR SUPPLIERS AND/OR LICENSORS); OR ALTERING THE TERMS AND CONDITIONS OF THE APPLICABLE LICENSE AGREEMENT GOVERNING THE USE OF IBM SOFTWARE.
Information Management
© 2010 IBM Corporation3
Open Source Persistence
Engine
JPA Persistence Engine
JDBC API
Hibernate, iBATIS, EclipseLink,...
JPA for WebSphere, Apache OpenJPA
DAOAPI
DataAccess Object
pureQueryPlain JDBCsMash
PHP, Ruby
Application on-ramps to DB2
JCC driver
JDBC
.Net Applications
.Net applications
JPA APIOpen Source
Persistence API Web API
ADO .Net
DB2
IBM Information Management
© 2007 IBM Corporation4
Layered Application Stack Challenges
� Simplify development, but …� Challenge problem resolution � Impact performance� Obscure impact analysis� Impede capacity planning
� Layers obscure linkages� Where is the problem?� Which application is running?� Which application does this SQL
belong to?
Application Server Database Server
Data A
ccess Logic
Persistence Layer
DB
Java Driver
EJB
Query Language
Connection
Pool
BusinessLogic
1
2
3
Network
A
A
A
IBM Information Management
© 2007 IBM Corporation5
Layered Application Stack Challenges
� Simplify development, but …� Challenge problem resolution � Impact performance� Obscure impact analysis� Impede capacity planning
� Layers obscure linkages� Where is the problem?� Which application is running?� Which application does this SQL
belong to?
Application Server Database Server
Data A
ccess Logic
Persistence Layer
DB
Java Driver
EJB
Query Language
Connection
Pool
BusinessLogic
1
2
3
Network
A
A
A
Application
Developer
System
s A
dministrator
Netw
ork A
dministrator
Database
Adm
inistrator
IBM Information Management
© 2007 IBM Corporation6
� Resource utilization across application and databas e infrastructure
� DBAs have little to no visibility of application SQL before deployment, no opportunity for review and optimization
� Problem isolation takes days due to inability to trace SQL to Java application and source code
� Constantly increasing application workload taxes existing systems – need to fit more work into existing systems
� SQL injection represents an increasing risk to data security
Data Access –Two Views of the World
DeveloperDBA
� No insight into data access performance
� Databases are a different skills domain
� Not a productive development environment
� No single environment that spans Java application and database development
� Problem isolation and resolution
mashuphttp
JSON
JSP
XML
JDBC
SQL
Spring
StoredProcedures
QoS goals Runstats
Response Time! REORG
Partition strategy
IBM Information Management
© 2007 IBM Corporation7
Can use individual
Statements
Using
Dynamic or
static SQL
getCustomers()getCustomersbyZip()getAccountDetails()getAccountReport()
ClientApplications
(business logic)
Static SQL
Packages
Database access development options -
getCustomers()getCustomersbyZip()getAccountDetails()getAccountReport()
Application Version 1
Uses individual SQL
Application Version 2
Uses SP Calls Db access
Using
Stored Procs
Call can be
Dynamic or
Static SQL
Call Individual SQL statements
Call Stored Procedures (SQL SPs, Java SPs),Stored Procedures – Can be SQL bodied or Java
Stored Procedures
IBM Information Management
© 2007 IBM Corporation8
Comparing programming models
A. Application SQL onlyDynamic SQL
B. SQL Stored ProcedureDynamic CALL, Static Procedure
C. Application SQL onlyStatic SQL
D. SQL Stored ProcedureStatic CALL, Static Procedure
SELECT C1 FROM T1
ApplicationDB
CALL P1
ApplicationDB
SELECT C1 FROM T1
SELECT C1 FROM T1
Application
SELECT C1 FROM T1
DB
CALL P1
Application
DB
SELECT C1 FROM T1
CALL P1
Information Management – IBM Data Studio
© 2008 IBM Corporation99
SQL Execution - Dynamic vs. Static
Dynamic SQLCheck auth for package / plan
Parse SQL StatementParse SQL Statement
Check Table / View AuthCheck Table / View Auth
Calculate access pathCalculate access path
Store access path in a temporary package
Store access path in a temporary package
Execute SQL statementExecute SQL statement
Extract access path from catalog or package cacheExtract access path from catalog or package cache
Execute SQL statementsExecute SQL statements
Static SQLCheck auth for package / plan
Information Management – IBM Data Studio
© 2008 IBM Corporation10
SQL Execution AdvantagesDynamic vs. Static
Feature Dynamic SQL(pureQuery, JDBC)
Static SQL(pureQuery, SQLJ)
Performance Can approach static SQL performance with help from dynamic SQL cache. Cache misses are costly
All SQL parsing, catalog access, done at BIND time. Fully optimized during execution.
Access path reliability
Unpredictable – Any prepare can get a new access path as statistics or host variables change
Guaranteed – locked in at BIND time All SQL available ahead of time for analysis by EXPLAIN.
Authorization Privileges handled at object level. All users or groups must have direct table privileges – Security exposure, and administrative burden
Privileges are package based. Only administrator needs table access. Users/Groups have execute authority. Prevent non-authorized SQL execution.
Monitoring, Problem determination
Database View is of the JDBC or CLI package – No easy distinction of where any SQL statement came from.
Package View of applications makes it simple to track back to the SQL statement location in the application
Capacity planning, Forecasting
Difficult to summarize performance data at program level.
Package Level Accounting gives program view of workload to aid accurate forecasting.
Tracking dependent objects
No record of which objects are referenced by a compiled SQL statement
Object dependencies registered in database catalog
IBM Information Management
© 2007 IBM Corporation12
pureQuery ClientOptimization Static
pureQuery API Static(Method Style)
97%115%
0%
20%
40%
60%
80%
100%
120%
ITR
: % d
iffer
ence
from
JD
BC
90% Package Cache hit ratio: DB2 LUW Database ITR comparison to JDBC
Increased throughput as compared to JDBC with 90% Package Cache hit ratio
In comparison with JDBC, Static APIs doubled the throughput (ITR*) of the DB2 LUW database server.
IBM Information Management
© 2007 IBM Corporation13
pureQuery ClientOptimization Static
pureQuery API Static(Method Style)
60% 66%
0%
20%
40%
60%
80%
100%
ITR
: % d
iffer
ence
from
JD
BC
95% Package Cache hit ratio: DB2 LUW Database ITR comparison with JDBC
Increased throughput as compared to JDBC with 95% Package Cache hit ratio
Compared to JDBC, Static APIs increased the throughput (ITR*) ofthe DB2 LUW database server by 60 and 66% respectively
Information Management – IBM Data Studio
© 2008 IBM Corporation14
Performance
� Using IRWW SQLJ and JDBC workloads, CL.1 CPU drops by 19.22%, while CL.2 CPU drops by 20.51%. Throughput increases by 46%
SQLJ and JDBC - dynamic vs. static SQL
0
0.0002
0.0004
0.0006
0.0008
0.001
0.0012
0.0014
0.0016
0.0018
0.002
Dynamic CL1 Static CL1 Dynamic CL2 Static CL2
Mic
rose
cond
s Dynamic CL1
Static CL1
Dynamic CL2
Static CL2
Information Management – IBM Data Studio
© 2008 IBM Corporation15
Static CALL to native SQL SP vs. dynamic CALL
� Results show an 11% reduction in CL.2 CPU time, and 9% reduction in CL.1 CPU time. Elapsed times dropped by roughly 5%.
IBM Information Management
© 2007 IBM Corporation16
Stored Procedures and DB2
� SQL � DB2 for LUW Version 8.2, 9.1, 9.5, 9.7� DB2 for iSeries V5R3, V5R4
� SQL External� DB2 for z/OS Version 8, 9
� SQL Native � DB2 for z/OS Version 9
� Java (JDBC or SQLJ or pureQuery)� DB2 for LUW Version 8.2, 9.1, 9.5, 9.7� DB2 for iSeries V5R3, V5R4� DB2 for z/OS Version 8, 9
� PL/SQL� DB2 for LUW Version 9.7�Oracle V10, V11
IBM Information Management
© 2007 IBM Corporation17
Stored procedures in DB2 9 for z/OS - SQL
� External SQL procedures:• Pre- DB2 9 SQL stored procedures• SQL processor converts SQL statements into a C program• Build utility DSNTPSMP, customizable • Runs in a WLM environment• Debug with Unified Debugger
� Native SQL procedures:• SQL procedure logic runs in the DBM1 address space• Extensive support for versioning• Native SQL procedures support zIIP Processors when invoked
over DDF• Debug with Unified Debugger
IBM Information Management
© 2007 IBM Corporation18
SQL Execution mode in Java Stored Procedures
� Dynamic :• Uses pureQuery or JDBC APIs
� Static :• Uses pureQuery or SQLJ
� Debug both with Unified Debugger
IBM Information Management
© 2007 IBM Corporation19
Comparing programming models
A. Application SQL onlyDynamic SQL
B. SQL Stored ProcedureDynamic CALL, Static Procedure
C. Application SQL onlyStatic SQL
D. SQL Stored ProcedureStatic CALL, Static Procedure
SELECT C1 FROM T1
ApplicationDB
CALL P1
ApplicationDB
SELECT C1 FROM T1
SELECT C1 FROM T1
Application
SELECT C1 FROM T1
DB
CALL P1
Application
DB
SELECT C1 FROM T1
CALL P1
High QoSHigh QoS
C & D Higher performance, access path reliability, improved authorization, monitoring, problem determination, capacity planning, and dependent object tracking.
B & D Use SPs when several SQL statements work together with
minimal flow logic
IBM Information Management
© 2007 IBM Corporation20
� Review SQL coming from applications
High cost of literals
� Consolidate SQL by replacing literals with parameter markers� Optimize database resources (e.g. dynamic statement cache) usage
IBM Information Management
© 2007 IBM Corporation21
Table 1, operation 5
Table 1, operation 1
Table 1, operation 2
Table 1, operation 3
Table 1 , operation 4
Table 1, operation 6
Using JDBC batching
Users
Application Servers
Network
DBMS
Batching – Multiple operations all execute as one batch – minimize chattiness on the network
pStmt.setString(2,"Accounting");
pStmt.setString(3,"A00");
pStmt.addBatch();
int [] numUpdates=prepStmt.executeBatch();
IBM Information Management
© 2007 IBM Corporation22
Table 2, operation 2
Table 1, operation 1
Table 1, operation 2
Table 1, operation 3
Table 2, operation 1
Table 3, operation 1
SQLJ – using Heterogenous batching
Users
Application Servers
Network
DBMSTable 1, operation 4
Heterogenous Batching – Multiple operations across different tables all execute as one batch
IBM Information Management
© 2007 IBM Corporation23
Java Data Access Spectrum – balancing productivity and control
Managed objects Object-relational mapping
Spring templates
Full SQL control
Code all your SQL
Use SQL templates, inline only
Complex OR mapping and persistence management, but loss of controls
Adds container management option
JDBC / SQLJ
iBATIS
Hibernate
OpenJPA (EJB3)
Add basic OR mapping and annotated-method style
pQ DAO
IBM Information Management
© 2007 IBM Corporation24 IBM Data Studio pureQuery For DBAs and Application Developers (v2.1) - Part 1 24242424
SQLJSQLJ
####sqlsqlsqlsql [con] {SELECT [con] {SELECT [con] {SELECT [con] {SELECT firstnmefirstnmefirstnmefirstnme INTO :INTO :INTO :INTO :empNameempNameempNameempName FROM employee WHERE FROM employee WHERE FROM employee WHERE FROM employee WHERE empnoempnoempnoempno = := := := :empnoempnoempnoempno};};};};
JDBC
SringSringSringSring sqlStatementsqlStatementsqlStatementsqlStatement = = = = ““““SELECT SELECT SELECT SELECT firstnmefirstnmefirstnmefirstnme FROM employee WHERE FROM employee WHERE FROM employee WHERE FROM employee WHERE empnoempnoempnoempno = ?= ?= ?= ?””””;;;;
java.sql.PreparedStatementjava.sql.PreparedStatementjava.sql.PreparedStatementjava.sql.PreparedStatement pspspsps = = = = con.PrepareStatement(sqlStatementcon.PrepareStatement(sqlStatementcon.PrepareStatement(sqlStatementcon.PrepareStatement(sqlStatement););););
ps.setString(1, ps.setString(1, ps.setString(1, ps.setString(1, empnoempnoempnoempno););););
java.sql.ResultSetjava.sql.ResultSetjava.sql.ResultSetjava.sql.ResultSet reader = reader = reader = reader = ps.executeQueryps.executeQueryps.executeQueryps.executeQuery();();();();
reader.nextreader.nextreader.nextreader.next();();();();
empNameempNameempNameempName = reader.getString(1);= reader.getString(1);= reader.getString(1);= reader.getString(1);
Reader.closeReader.closeReader.closeReader.close(); (); (); ();
Non-Java syntax, but optimal accessTedious and long code
SQLJ Pros SQLJ Cons
Concise compared to JDBC Not a popular common programming model
Full control of SQL Many IDEs do not have SQLJ support
Most optimal access to the database (static) Deployment has additional steps
Heterogenous batching (reduce network traffic)
Cannot be used with other programming models
Better diagnostics
IBM Information Management
© 2007 IBM Corporation25
Data Access Objects – core J2EE pattern
Using J2EE Data Access Objects
� Methods for each database access
� Each method has only your parameters
� SQL can be in separate file – separate business logic from data access
IBM Information Management
© 2007 IBM Corporation26
What is Hibernate
� Open source Object-Relational Mapping (ORM) library for Java
� Simple object oriented database access
� No complex JDBC coding
� Data access though object query language and SQL
HibernateHibernateHibernate
Database
JNDI JDBC JTA
ApplicationApplicationApplication
IBM Information Management
© 2007 IBM Corporation27
HibernateHibernateHibernate
Database
JNDI JDBC JTA
ApplicationApplicationApplication
Hibernate
Developers code as above
Hibernate generates the following SQL
IBM Information Management
© 2007 IBM Corporation28
Goals for performance management
� Prevent problems before they impact the business�Get early warning of emergent problems� Isolate problems quickly
� Improve quality of service� Use expert advice for faster problem resolution� Stabilize application performance
� Reduce costs� Increase capacity of existing systems� Free up DBA time for value creation activities� Fix problems in development while costs and impact are low� Propagate performance expertise to avoid problems
IBM Information Management
© 2007 IBM Corporation29
Users
Application Servers
Network
Identify, diagnose, solve, and prevent performance problems
• Drill-down into problem detail and related context• Analyze captured SQL
2. Diagnose
• Alert of potential problems• Visual quick scan of complex environment
1. Identify
4. Prevent
• Use proven performance improvement techniques
3. Solve
DBMS & OS
• Use best practices to avoid problems in development
IBM Information Management
© 2007 IBM Corporation30
More Visibility, Productivity, and Control of Application SQL
� Capture � Capture the SQL during development
� Optimize� Review and share SQL� Optimize and replace bad SQL � Create approved SQL list
� Deploy � Configure execution properties� Optionally bind for static execution
Capture
Review Restrict
Configure Bind
Execute
Monitor
Optimize
Java Persistence Architecture
IBM Information Management
© 2007 IBM Corporation31
Simplifying Problem Determination Scenario
Application Developer
�Available for each db access�SQL text generated�Access path�Cost estimates�Estimated response time
Database Administrator
� Available for each SQL�Application name�Java class name�Java method name�Java object name
IBM Information Management
© 2007 IBM Corporation32
Which application has issued the problematic SQL?
Facilitate Problem Determination
� Use database package information to correlate SQL w ith Java or .NET applications
Diagnose
IBM Information Management
© 2007 IBM Corporation33
Unique Package Names Improves PD
DynamicJava SQL
StaticJava SQL
� Hard to identify unique Java applications
� Most dynamic Java applications use packages SYSLNx00
� Unique package names link SQL to Java code modules
Diagnose
IBM Information Management
© 2007 IBM Corporation34
Statically invoking Stored Procedures from Java
Native SQL stored procedure invoked Dynamically from Java application
Native SQL stored procedure invoked Statically from Java
application
Diagnose
IBM Information Management
© 2007 IBM Corporation35
Client Info. fields
� Voluntary – but very useful !� java.sql.Connection.setClientInfo -� setClientInfo("ApplicationName","OrderHistoryApp");� conn.setClientInfo("ClientUser", “Vijay Bommireddipalli"); � conn.setClientInfo("ClientHostname, “xyz");
Diagnose
IBM Information Management
© 2007 IBM Corporation36
Tip: Adding Java clientProgramName will set Correlation Id. Diagnose
IBM Information Management
© 2007 IBM Corporation37
Solve using any or all of the following -
� Stabilize performance and lock access plans using Static SQL
� Improve performance using JDBC batching or even better –Heterogeneous batching capability
� Improve performance by substituting literals with parameter markers
Solve
IBM Information Management
© 2007 IBM Corporation38
Preventing the problem: Proactively closing the loop
Improve usage of database
Execute application
Monitor
Improve databaseperformance
Application Developers
DBA
Prevent
IBM Information Management
© 2007 IBM Corporation39
Building high performance applications proactively
� Take advantage of optimized database access� Static SQL� Batching (JDBC or heterogeneous)� Avoid literals where possible� Avoid unnecessary re-prepares� Stored Procedures
� Add diagnostics where possible� Eg. Client info fields� ‘Side effects’ of static SQL
� Templates� Share templates of frequently used SQL statements
� Review process for SQL� Explain analysis� Tune early, Tune often
Prevent
Information Management
© 2010 IBM Corporation40
Resources
� Optim solutions for accelerating Java database accesshttps://www.ibm.com/developerworks/offers/lp/demos/summary/im-optimsolutionsforjava.html
� IBM Optim Development Studiohttp://www-01.ibm.com/software/data/optim/development-studio/
� IBM Optim pureQuery Runtime web page http://www.ibm.com/software/data/studio/purequery/
� Articles and tutorials covering the Optim portfoliohttp://www.ibm.com/developerworks/spaces/optim