Session #TAC-2104 - Achieving highest availability and best performance with CICS TG on z/OS HA Architecture - User Experience Achieving highest availability and best performance with CICS TG on z/OS HA Architecture – User Experience Elena Nanos Lead Systems Architect Health Care Service Corporation IBM Certified Infrastructure Systems Architect TOGAF Certified Architect - The Open Group Architecture Framework V8.1.1 IBM Certified Solution Expert - Dynamic Infrastructure Technical Leader IBM Certified System Administrator - WebSphere Application Server ND V7.0 IBM Certified Solution Expert - CICS Web Enablement IBM Certified System Specialist - WebSphere MQSeries Email – [email protected]Session TAC-2104 1
48
Embed
Ibm session tac 2104 - ctg presentation for impact 2013 final
This presentation titled "Achieving highest availability and best performance with CICS TG on z/OS HA Architecture – User Experience" was presented at IMPACT 2013.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Session #TAC-2104 - Achieving highest availability and best performance with CICS TG on z/OS HA Architecture - User Experience
Achieving highest availability and best performance with CICS TG on z/OS HA Architecture – User Experience
Elena Nanos Lead Systems Architect
Health Care Service Corporation
IBM Certified Infrastructure Systems Architect
TOGAF Certified Architect - The Open Group Architecture Framework V8.1.1
IBM Certified Solution Expert - Dynamic Infrastructure Technical Leader
IBM Certified System Administrator - WebSphere Application Server ND V7.0
IBM Certified Solution Expert - CICS Web Enablement
IBM Certified System Specialist - WebSphere MQSeries
Session #TAC-2104 - Achieving highest availability and best performance with CICS TG on z/OS HA Architecture - User Experience
Health Care Service Corporation (HCSC) is the fourth largest health insurance company in the nation.
Largest customer-owned health insurer in the U.S., founded in 1936, now with more than 13 million members, HCSC operates Blue Cross and Blue Shield Plans in Illinois, New Mexico, Oklahoma and Texas.
We're greater than 18,000 employees strong with 60 local offices and state-of-the-art technology, including two Tier IV data centers — the industry's highest reliability level – that provide the speed and data security to meet our customers' current and future business needs.
Who Are We? About Health Care Service Corporation
3
Session #TAC-2104 - Achieving highest availability and best performance with CICS TG on z/OS HA Architecture - User Experience
Common Architectures and Standard Used with CICS Connectivity
option
Required
middleware
Main capabilities Usage notes
Web services CICS-only solution • Inbound and outbound
• Low coupling
• Synchronous (HTTP)
• Asynchronous (WMQ)
• QoS based on transport type
• Support for some WS-* standards
Should be first consideration for service
enabling CICS applications, particularly
when you need to support multiple service
requester types or need bi-directional
support
JCA with
CICS TG
CICS TG
(normally WebSphere
Application Server)
• Inbound to CICS
• Medium coupling
• Synchronous
• High qualities of service (QoS)
Most appropriate solution when service
requester is JEE component and when high
QoS required
JCA with
WOLA
WebSphere
Application Server
for z/OS
• Inbound and outbound
• Tight coupling
• Synchronous
• High QoS
Particularly useful for JCA access to and
from CICS, and for very high throughput
and performance requirements.
CICS web
support
CICS-only solution • Inbound and outbound
• Medium coupling
• Synchronous
• Medium QoS
Use with web services, RESTful services,
and Atom feeds, or when remote
client/server only supports HTTP
WebSphere
MQ for z/OS
CICS-only solution • Inbound and outbound
• Medium coupling
• Asynchronous, with almost-synchronous capabilities
• Assured delivery
Exploit WMQ for basic messaging and
flowing web services
4
Session #TAC-2104 - Achieving highest availability and best performance with CICS TG on z/OS HA Architecture - User Experience
HCSC CICS TG Topology
• Topology 1 was our main WASd/CICS TG configuration.
• Topology 2 was proposed configuration moving CICS TG off distributed platform into
z/OS, using CICS TG on z/OS stand-alone (Remote mode). This is our current
topology.
• Topology 3 is also used with WAS on z/OS and CICS TG configuration, using CICS TG
in Local mode. Note: Local mode on z/OS can use either EXCI (which is MRO cross memory), or
IPIC (which is still sockets based, but will be optimized in TCPIP to Fast Local Sockets - effectively cross
memory - when co-located within the same LPAR as CICS).
.
5
Session #TAC-2104 - Achieving highest availability and best performance with CICS TG on z/OS HA Architecture - User Experience
Session #TAC-2104 - Achieving highest availability and best performance with CICS TG on z/OS HA Architecture - User Experience
7
HCSC CICS TG HA Architecture
Session #TAC-2104 - Achieving highest availability and best performance with CICS TG on z/OS HA Architecture - User Experience
CICS TG V7.0
WLM Health reporting for storm drain avoidance that required use of the
SHAREPORTWLM keyword (a variant of SHAREPORT)
WLM Feedback support – Storm drain avoidance Feedback of CICS region status to IP load balancing mechanism via WLM
Allows more intelligent autonomic decisions to be performed by IP load balancers
Pushes CICS health out to the IP load balancer
CICS TG V7.2
Sysplex-wide support for XA with HA which used Sysplex Distributor.
Port-sharing across Sysplex but this could only be combined with DSS for 1-phase commit
transactions and synconreturn.
Support for XA transactions in High Availability topologies using TCPIP load balancing in a z/OS
Sysplex
All CICS TG versions have ability to use SHAREPORT on the PORT statement in the TCP/IP profile
Enables a group of cloned servers to listen on the same port
New connections are distributed across available servers using a weighted round-robin
8
High Availability and XA Transactions
History of Enhancements
Session #TAC-2104 - Achieving highest availability and best performance with CICS TG on z/OS HA Architecture - User Experience
CICS TG V8.0
Provides additional High Availability capabilities, allowing dynamic selection of CICS server
within the Gateway daemon
Provides a working, configurable CICS request exit (including source) for performing
failover or round robin
Support for cross-sysplex port sharing XA/2PC requests in combination with DSS
CICS TG V8.1
Introduced the "out-of-the-box" configurable DSSPOLICY as an alternative to CREXIT, for
those wanting a basic roundrobin or failover implementation. CREXIT still provides maximum
flexibility when it comes to custom algorithms.
CICS TG V9.0
Routing or filtering by client IP address
Full request details for XA transactions
New Request Monitoring exit point “RequestDetails”
Invoked just before calling CICS, after Dynamic
Server Selection
9
High Availability and XA Transactions
History of Enhancements
Session #TAC-2104 - Achieving highest availability and best performance with CICS TG on z/OS HA Architecture - User Experience
Health reporting is effective only in TCP/IP load balancing topologies, with the CICS Transaction Gateway running in remote mode.
At intervals specified by the Health interval, the Gateway daemon monitors error codes like ECI_ERR_NO_CICS, ECI_ERR_RESOURCE_SHORTAGE, ECI_ERR_SYSTEM_ERROR to determine the health of communications with the CICS subsystem. The load balancer can then use this health to set priorities when creating new incoming IP socket connections to cloned Gateway daemons.
MVS console command can be used to display Gateway health status – F <JOB>,APPL=STATS,GS=GD_CHEALTH
10
WLM Feedback
Session #TAC-2104 - Achieving highest availability and best performance with CICS TG on z/OS HA Architecture - User Experience
TCP/IP load balancing • TCP/IP load balancing provides high availability of the Gateway daemon by distributing
connections from clients across multiple Gateway daemons. The efficiency of load balancing can also be increased by taking into account CICS server status provided by dynamic feedback to the IBM z/OS Workload Manager (WLM).
Port sharing • TCP/IP port sharing enables requests for work to be shared between several
Gateway daemons through a single TCP/IP port. Port sharing provides TCP/IP load balancing in a single LPAR. When connections are established between a Client application and a Gateway daemon, the TCP/IP port sharing component of the z/OS TCP/IP. Communications subsystem distributes requests across multiple Gateway daemons. When the connection is established, subsequent requests from the Client application continue to use the same connection.
Health monitoring • A TCP/IP load balancer that is allocating a connection to the Gateway daemon detects whether or
not a CICS server is available. The Gateway daemon reports the health of its CICS server connections to the TCP/IP load balancer.
Support Pack CH51 – CICS TG Simple Request Monitor Utility
Support Pack CA1T – CICS TG High Availability User Exits
MICS report using SMF 111 records
CICS Explorer 11
Technology Utilized in HCSC CICS TG HA Architecture
Session #TAC-2104 - Achieving highest availability and best performance with CICS TG on z/OS HA Architecture - User Experience
Two different workload balancing polices (round robin or fail over) are supplied – both of
which use server rules from the HA configuration file to remap and retry failed requests.
For example, the following server rule defines the server alias SERVER1 as mapping
to 4 actual CICS servers CICSA, CICSB, CICSC & CICSD:
SERVER1=CICSA,CICSB,CICSC,CICSD
The round robin policy will distribute requests in a round robin fashion, so that
requests are evenly distributed across servers CICSA, CICSB, CICSC & CICSD.
The fail over policy will distribute requests in a hierarchical order so that all requests
are sent to CICSA, if this is unavailable requests will be sent to CICSB, if CICSB is
unavailable requests will be sent to CICSC, and if CICSC is unavailable requests will
SupportPack CA1T – CICS TG High Availability User Exits
Session #TAC-2104 - Achieving highest availability and best performance with CICS TG on z/OS HA Architecture - User Experience
Three different CICS request exit implementations are provided as follows:
1. FailOver – uses a list of CICS servers to define how server names are remapped. Requests will be retried using a static list of servers, each server being used in turn from the list until the list is exhausted or the request succeeds.
2. RoundRobin – distributes ECI requests around a list of CICS servers in round-robin fashion. Requests will be distributed evenly across the servers, and retried until the request succeeds or all serves have been used.
3. RequestValidation – validates the transaction ID, program and user ID parameters in the ECI request against a defined list. No server name remapping is performed.
Latest enhancements
CA1T V3.0 support pac has been recently updated and gives ability to set new environment variable CTG_HAVALIDATION=EXCLUSIVE, then any request which matches one of the USERID, TRANSID and PROGRAM validation rules will be rejected with ECI_ERR_INVALID_CALL_TYPE (-14).
Note: Configuration is based on a flexible set of rules defined in a configuration file. On z/OS, the configuration file can reside either on HFS or
in an MVS dataset. Rules for workload balancing can be generic per Gateway, based on server aliases, or on the payload type (for example,
channel or commarea). Rules for request validation can be specified based on user IDs, transaction IDs, or program names, and operate as
either a white list (INCLUSIVE) or black list (EXCLUSIVE). All rules can be dynamically updated while the Gateway daemon is running. A
configurable retry interval is provided to improve the efficiency of request distribution when using workload balancing.
Logging levels available – set at -j-Dcom.ibm.CTG.samples.haexit.ca1t
15
Support Pack CH51: CICS TG
Simple Request Monitor Utility Setup
Can also issue console command. For example to enable level 3 logging issue the following
command - /F <JOB>,APPL=CREXIT,CMD=CA1T_3
Session #TAC-2104 - Achieving highest availability and best performance with CICS TG on z/OS HA Architecture - User Experience
This SupportPac allows integration test tools to intercept ECI calls to CICS TG from Java™ applications before they are processed by CICS TG.
Allows integration test tools ( for example Rational Test Workbench / Rational Integration Tester V8.0.1.1) to intercept External Call Interface (ECI) calls to IBM CICS TG, originating from Java Standard Edition (Java SE) and Java Enterprise Edition (JEE) applications. This, therefore, allows such applications to be tested without requiring a running CICS TG or CICS Transaction Server.
This SupportPac provides updated versions of the CICS TG V8.0 Java base classes and non-extended architecture (non-XA) ECI resource adapter. These updated files support the use of a Gateway intercept plug-in that can intercept application calls made through the Java Application Programming Interfaces (APIs). Documentation for the APIs is also provided by this SupportPac.
When enabled, a Gateway intercept plug-in is notified before a connection to a Gateway daemon is opened or closed, and before a request is sent to the Gateway daemon. Before a connection is opened, the plug-in can change properties of the connection object, and can prevent the connection from opening. Before a request is sent over a connection, the plug-in can change properties of the request object, and can cause the request to be returned to the application without being sent. Before a connection is closed, the plug-in can prevent the connection from closing. This allows a variety of scenarios to be simulated for application testing.
Can be downloaded from – http://www.ibm.com/support/docview.wss?uid=swg24033968
Session #TAC-2104 - Achieving highest availability and best performance with CICS TG on z/OS HA Architecture - User Experience
CICS TG V8.0 or prior has two CICS Resource Adapters – XA and non-XA.
CICS TG V8.1 has support for Java Enterprise Edition 6
The CICS connectors included with CICS TG V8.1 exploit JEE 6 / JCA 1.6
JEE 6 certified application servers (e.g. WAS 8.0) benefit from:
A single ECI resource adapter (rather than two) including support for both XA (two-phase
commit) and non-XA transactions
A more flexible approach to setting the transactional capabilities of the CICS ECI resource
adapter
Lower administrative and maintenance overheads
Resource adapters compatible with JEE 5 (or earlier) application servers are available
use with CICS TG V8.1 in remote mode only from Support Pac CC03
19
CICS JCA Resource Adapters
Session #TAC-2104 - Achieving highest availability and best performance with CICS TG on z/OS HA Architecture - User Experience
The CICS JCA resource adapters are provided, in order to support connectivity from JEE application servers to CICS using the facilities of the CICS Transaction Gateway.
The latest service levels of the CICS Transaction Gateway Java Enterprise Edition Connector Architecture (JCA) resource adapters for CICS TG are available for download for deployment into supported Java Enterprise Edition (JEE) application servers. These resource adapters can be
used to provide JCA connectivity into CICS using the facilities of a licensed CICS TG installation, and are designed around the JCA specification as supported by JEE application servers such as WebSphere Application Server and Oracle Weblogic Server.
This Support Pack includes the CICS JCA resource adapters from the following CICS Transaction Gateway for Multiplatforms releases: V7.0.1.1, V7.1.0.4, V7.2.0.4, V8.0.0.4, V8.1.0.1, V9.0.0.0
Session #TAC-2104 - Achieving highest availability and best performance with CICS TG on z/OS HA Architecture - User Experience
The graph shows the difference in processor usage cost between EXCI and IPIC.
The CICS address space costs were less when using EXCI, because of the efficiency of using cross-memory MRO for communication between the gateway address space and the CICS address space.
Athough the overall cost to use IPIC was greater, the amount of processing offloaded to a zAAP was also greater, which made the use of general-purpose CPs in the gateway address space less critical for IPIC.
22
EXCI vs. IPIC CPU Usage
Session #TAC-2104 - Achieving highest availability and best performance with CICS TG on z/OS HA Architecture - User Experience
Data from redpaper IBM CICS Performance Series: A Processor Usage Study of Ways into CICS
CICS Performance Analyzer V5.1 now includes two CICS TG batch reports, "Activity Summary", and
"Usage and Capacity" which can be run against SMF 111 records from CICS TG for z/OS.
29
CICS Performance Analyzer for z/OS - Sample Reports
Session #TAC-2104 - Achieving highest availability and best performance with CICS TG on z/OS HA Architecture - User Experience
CICS Transaction Gateway plug-in for the CICS Explorer V2.0.0.0 is now available for CICS Explorer V5.1, easily installed through the common V5.1 update site which is pre-configured in the CICS Explorer "Install new software" facility.
30
CICS Transaction Gateway Plug-in For The CICS Explorer
Session #TAC-2104 - Achieving highest availability and best performance with CICS TG on z/OS HA Architecture - User Experience
CSSYS1CIC1_SIPADDR=yourSYS1.network.net (CICS server TCP/IP address)
CSSYS1CIC1_SIPPORT=3168 (CICS server TCP/IP port)
CSSYS1CIC1_LALLREQ=4549989 (Number of requests processed)
CSSYS1CIC1_CWAITING=-2 (Number of requests waiting on a response)
CSSYS1CIC1_SSESSMAX=200 (Number of requested IPIC sessions)
CSSYS1CIC1_CSESSMAX=200 (Number of negotiated IPIC sessions)
CSSYS1CIC1_CSESSCURR=0 (Number of IPIC sessions in use)
CSSYS1CIC1_LCOMMSFAIL=49 (Number of CICS communication failures)
CSSYS1CIC1_LCONNFAIL=4484 (Number of connection failures)
CSSYS1CIC1_LLOSTCONN=32 (Number of lost connections)
CSSYS1CIC1_LSESSFAIL=0 (Number of IPIC session failures)
CSSYS1CIC1_LIDLETIMEOUT=0 (Number of timed out connections)
CSSYS1CIC1_LREQDATA=28649890161 (Amount of CICS request data)
CSSYS1CIC1_LRESPDATA=29043330152 (Amount of CICS response data)
Session #TAC-2104 - Achieving highest availability and best performance with CICS TG on z/OS HA Architecture - User Experience
34
Other Monitoring Tools Available
IBM Tivoli OMEGAMON XE for CICS on z/OS V5.1.0
Session #TAC-2104 - Achieving highest availability and best performance with CICS TG on z/OS HA Architecture - User Experience
35
Performance Tuning Helps to prevent CICS becoming overloaded or underutilized
Allows for optimization of workflow through CICS TG and onto CICS
Alerts users to changes in behavior when abnormal workloads hit CICS TG
Provides users with an account of transactions per minute and CPU
consumption
Availability Alerts users to communication problems between CICS TG and CICS
Managablity Helps to determine where to control resources in CICS TG or CICS
Monitoring Transaction Monitoring in Gateway daemon and WebShere Application
Server on z/OS
Transaction Response Time Analysis, Payload and Wiresize per
transaction
End To End Response Time Analysis
Problem Determination Identifies the bottlenecks within CICS TG daemon
Identifies Abended transactions
Correlates all application Flows within a Transaction
Provides Transaction Flow Trace
Capacity Planning Helps to plan for workload growth, perform trend analysis, detect system
anomalies and to identify performance bottlenecks using the historical data
via the Tivoli Data Warehouse repository
Total Cost of Ownership Opens up the “black-box” of CICS TG thread management and helps to
save precious resources in both CICS TG and CICS
Other Monitoring Tools Available
IBM Tivoli OMEGAMON XE for CICS on z/OS V5.1.0
Session #TAC-2104 - Achieving highest availability and best performance with CICS TG on z/OS HA Architecture - User Experience
Issue
We implemented new Application using CICS TG on z/OS, which had invalid Userid set on Connection Factory. This caused ID to get suspended and Application was redriving error routine to restablish CICS TG connection, after each failure. This caused connections not getting released, causing all connections to be in use across all 4 CICS TGs, resulting in CICS TG -16 return (ECI_ERR_RESOURCE_SHORTAGE).
Solution
Setup automation to issue CICS TG console command –
/F CICS TGname,APPL=STATS,GS every 5 minutes and send alert if field CS_CSESSCURR (total # of IPIC sessions in use with all CICS servers) is at 80% or higher from CS_CSESSMAX (# of IPIC sessions negotiated with CICS servers). Can also have automation (OPS/MVS for example) check the values of CM_CALLOC and WT_ALLOC and alert if the threshold set has been passed.
CICS TG on z/OS V8.0
PM38908: ECI_ERR_RESOURCE_SHORTAGE reported when sending extended mode ECI requests over an IPIC connection to a CICS server
Error description
When an extended mode ECI request does not complete successfully (possibly due to security reasons) before the CICS mirror transaction is attached, the IPIC session is not freed.
• The information contained in this publication is provided for informational purposes only. While efforts were made to verify the completeness and accuracy of the information contained
in this publication, it is provided AS IS without warranty of any kind, express or implied. In addition, this information is based on IBM’s current product plans and strategy, which are
subject to change by IBM without notice. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, this publication or any other materials. Nothing
contained in this publication is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and
conditions of the applicable license agreement governing the use of IBM software.
• References in this presentation to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or
capabilities referenced in this presentation may change at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to
future product or feature availability in any way. Nothing contained in these materials is intended to, nor shall have the effect of, stating or implying that any activities undertaken by
you will result in any specific sales, revenue growth or other results.
• If the text contains performance statistics or references to benchmarks, insert the following language; otherwise delete:
Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will
experience will vary depending upon many factors, including considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage
configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here.
• If the text includes any customer examples, please confirm we have prior written approval from such customer and insert the following language; otherwise delete:
All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual environmental costs
and performance characteristics may vary by customer.
• Please review text for proper trademark attribution of IBM products. At first use, each product name must be the full name and include appropriate trademark symbols (e.g., IBM
Lotus® Sametime® Unyte™). Subsequent references can drop “IBM” but should include the proper branding (e.g., Lotus Sametime Gateway, or WebSphere Application Server).
Please refer to http://www.ibm.com/legal/copytrade.shtml for guidance on which trademarks require the ® or ™ symbol. Do not use abbreviations for IBM product names in your
presentation. All product names must be used as adjectives rather than nouns. Please list all of the trademarks that you use in your presentation as follows; delete any not included in
your presentation. IBM, the IBM logo, Lotus, Lotus Notes, Notes, Domino, Quickr, Sametime, WebSphere, UC2, PartnerWorld and Lotusphere are trademarks of International
Business Machines Corporation in the United States, other countries, or both. Unyte is a trademark of WebDialogs, Inc., in the United States, other countries, or both.
• If you reference Adobe® in the text, please mark the first use and include the following; otherwise delete:
Adobe, the Adobe logo, PostScript, and the PostScript logo are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States, and/or other countries.
• If you reference Java™ in the text, please mark the first use and include the following; otherwise delete:
Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both.
• If you reference Microsoft® and/or Windows® in the text, please mark the first use and include the following, as applicable; otherwise delete:
Microsoft and Windows are trademarks of Microsoft Corporation in the United States, other countries, or both.
• If you reference Intel® and/or any of the following Intel products in the text, please mark the first use and include those that you use as follows; otherwise delete:
Intel, Intel Centrino, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and
other countries.
• If you reference UNIX® in the text, please mark the first use and include the following; otherwise delete:
UNIX is a registered trademark of The Open Group in the United States and other countries.
• If you reference Linux® in your presentation, please mark the first use and include the following; otherwise delete:
Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both. Other company, product, or service names may be trademarks or service marks of
others.
• If the text/graphics include screenshots, no actual IBM employee names may be used (even your own), if your screenshots include fictitious company names (e.g., Renovations, Zeta
Bank, Acme) please update and insert the following; otherwise delete: All references to [insert fictitious company name] refer to a fictitious company and are used for illustration