Ibm session tac 2104 - ctg presentation for impact 2013 final

Session #TAC-2104 - Achieving highest availability and best performance with CICS TG on z/OS HA Architecture - User Experience

Achieving highest availability and best performance with CICS TG on z/OS HA Architecture – User Experience

Elena Nanos Lead Systems Architect

Health Care Service Corporation

IBM Certified Infrastructure Systems Architect

TOGAF Certified Architect - The Open Group Architecture Framework V8.1.1

IBM Certified Solution Expert - Dynamic Infrastructure Technical Leader

IBM Certified System Administrator - WebSphere Application Server ND V7.0

IBM Certified Solution Expert - CICS Web Enablement

IBM Certified System Specialist - WebSphere MQSeries

Email – [email protected]

Session TAC-2104 1

mailto:[email protected]


2

CICS TG Product Positioning

Presented by Catherine Moxey, IBM STSM, Email: [email protected]


Health Care Service Corporation (HCSC) is the fourth largest health insurance company in the nation.

Largest customer-owned health insurer in the U.S., founded in 1936, now with more than 13 million members, HCSC operates Blue Cross and Blue Shield Plans in Illinois, New Mexico, Oklahoma and Texas.

We're greater than 18,000 employees strong with 60 local offices and state-of-the-art technology, including two Tier IV data centers — the industry's highest reliability level – that provide the speed and data security to meet our customers' current and future business needs.

Who Are We? About Health Care Service Corporation

3


Common Architectures and Standard Used with CICS Connectivity

option

Required

middleware

Main capabilities Usage notes

Web services CICS-only solution • Inbound and outbound

• Low coupling

• Synchronous (HTTP)

• Asynchronous (WMQ)

• QoS based on transport type

• Support for some WS-* standards

Should be first consideration for service

enabling CICS applications, particularly

when you need to support multiple service

requester types or need bi-directional

support

JCA with

CICS TG

CICS TG

(normally WebSphere

Application Server)

• Inbound to CICS

• Medium coupling

• Synchronous

• High qualities of service (QoS)

Most appropriate solution when service

requester is JEE component and when high

QoS required

JCA with

WOLA

WebSphere

Application Server

for z/OS

• Inbound and outbound

• Tight coupling

• Synchronous

• High QoS

Particularly useful for JCA access to and

from CICS, and for very high throughput

and performance requirements.

CICS web

support

CICS-only solution • Inbound and outbound

• Medium coupling

• Synchronous

• Medium QoS

Use with web services, RESTful services,

and Atom feeds, or when remote

client/server only supports HTTP

WebSphere

MQ for z/OS

CICS-only solution • Inbound and outbound

• Medium coupling

• Asynchronous, with almost-synchronous capabilities

• Assured delivery

Exploit WMQ for basic messaging and

flowing web services

4


HCSC CICS TG Topology

• Topology 1 was our main WASd/CICS TG configuration.

• Topology 2 was proposed configuration moving CICS TG off distributed platform into

z/OS, using CICS TG on z/OS stand-alone (Remote mode). This is our current

topology.

• Topology 3 is also used with WAS on z/OS and CICS TG configuration, using CICS TG

in Local mode. Note: Local mode on z/OS can use either EXCI (which is MRO cross memory), or

IPIC (which is still sockets based, but will be optimized in TCPIP to Fast Local Sockets - effectively cross

memory - when co-located within the same LPAR as CICS).

.

5


Current Configuration

6

Previous HCSC Distributed Configuration

http://www.google.com/imgres?imgurl=http://parts.mit.edu/wiki/images/thumb/5/5d/ArrowCurvedDown.PNG/120px-ArrowCurvedDown.PNG&imgrefurl=http://parts.mit.edu/wiki/index.php?title=Imperial_College_2006&redirect=no&usg=__cz3yLvoFqSzvQIUWOurE_FcemGo=&h=78&w=120&sz=2&hl=en&start=4&tbnid=p4d1KPBqojOB9M:&tbnh=57&tbnw=88&prev=/images?q=arrow+down+curved&hl=en&sa=N&gbv=2&tbs=isch:1&itbs=1


7

HCSC CICS TG HA Architecture


CICS TG V7.0

WLM Health reporting for storm drain avoidance that required use of the

SHAREPORTWLM keyword (a variant of SHAREPORT)

WLM Feedback support – Storm drain avoidance Feedback of CICS region status to IP load balancing mechanism via WLM

Allows more intelligent autonomic decisions to be performed by IP load balancers

Pushes CICS health out to the IP load balancer

CICS TG V7.2

Sysplex-wide support for XA with HA which used Sysplex Distributor.

Port-sharing across Sysplex but this could only be combined with DSS for 1-phase commit

transactions and synconreturn.

Support for XA transactions in High Availability topologies using TCPIP load balancing in a z/OS

Sysplex

All CICS TG versions have ability to use SHAREPORT on the PORT statement in the TCP/IP profile

Enables a group of cloned servers to listen on the same port

New connections are distributed across available servers using a weighted round-robin

8

High Availability and XA Transactions

History of Enhancements


CICS TG V8.0

Provides additional High Availability capabilities, allowing dynamic selection of CICS server

within the Gateway daemon

Provides a working, configurable CICS request exit (including source) for performing

failover or round robin

Support for cross-sysplex port sharing XA/2PC requests in combination with DSS

CICS TG V8.1

Introduced the "out-of-the-box" configurable DSSPOLICY as an alternative to CREXIT, for

those wanting a basic roundrobin or failover implementation. CREXIT still provides maximum

flexibility when it comes to custom algorithms.

CICS TG V9.0

Routing or filtering by client IP address

Full request details for XA transactions

New Request Monitoring exit point “RequestDetails”

Invoked just before calling CICS, after Dynamic

Server Selection

9

High Availability and XA Transactions

History of Enhancements


Health reporting is effective only in TCP/IP load balancing topologies, with the CICS Transaction Gateway running in remote mode.

At intervals specified by the Health interval, the Gateway daemon monitors error codes like ECI_ERR_NO_CICS, ECI_ERR_RESOURCE_SHORTAGE, ECI_ERR_SYSTEM_ERROR to determine the health of communications with the CICS subsystem. The load balancer can then use this health to set priorities when creating new incoming IP socket connections to cloned Gateway daemons.

MVS console command can be used to display Gateway health status – F <JOB>,APPL=STATS,GS=GD_CHEALTH

10

WLM Feedback


TCP/IP load balancing • TCP/IP load balancing provides high availability of the Gateway daemon by distributing

connections from clients across multiple Gateway daemons. The efficiency of load balancing can also be increased by taking into account CICS server status provided by dynamic feedback to the IBM z/OS Workload Manager (WLM).

Sysplex Distributor • Sysplex Distributor provides TCP/IP load balancing across multiple LPARs.

Port sharing • TCP/IP port sharing enables requests for work to be shared between several

Gateway daemons through a single TCP/IP port. Port sharing provides TCP/IP load balancing in a single LPAR. When connections are established between a Client application and a Gateway daemon, the TCP/IP port sharing component of the z/OS TCP/IP. Communications subsystem distributes requests across multiple Gateway daemons. When the connection is established, subsequent requests from the Client application continue to use the same connection.

Health monitoring • A TCP/IP load balancer that is allocating a connection to the Gateway daemon detects whether or

not a CICS server is available. The Gateway daemon reports the health of its CICS server connections to the TCP/IP load balancer.

Support Pack CH51 – CICS TG Simple Request Monitor Utility

Support Pack CA1T – CICS TG High Availability User Exits

MICS report using SMF 111 records

CICS Explorer 11

Technology Utilized in HCSC CICS TG HA Architecture


Two different workload balancing polices (round robin or fail over) are supplied – both of

which use server rules from the HA configuration file to remap and retry failed requests.

For example, the following server rule defines the server alias SERVER1 as mapping

to 4 actual CICS servers CICSA, CICSB, CICSC & CICSD:

SERVER1=CICSA,CICSB,CICSC,CICSD

The round robin policy will distribute requests in a round robin fashion, so that

requests are evenly distributed across servers CICSA, CICSB, CICSC & CICSD.

The fail over policy will distribute requests in a hierarchical order so that all requests

are sent to CICSA, if this is unavailable requests will be sent to CICSB, if CICSB is

unavailable requests will be sent to CICSC, and if CICSC is unavailable requests will

be sent to CICSD.

http://www-01.ibm.com/support/docview.wss?uid=swg24021484

12

SupportPack CA1T – CICS TG High Availability User Exits


Three different CICS request exit implementations are provided as follows:

1. FailOver – uses a list of CICS servers to define how server names are remapped. Requests will be retried using a static list of servers, each server being used in turn from the list until the list is exhausted or the request succeeds.

2. RoundRobin – distributes ECI requests around a list of CICS servers in round-robin fashion. Requests will be distributed evenly across the servers, and retried until the request succeeds or all serves have been used.

3. RequestValidation – validates the transaction ID, program and user ID parameters in the ECI request against a defined list. No server name remapping is performed.

Latest enhancements

CA1T V3.0 support pac has been recently updated and gives ability to set new environment variable CTG_HAVALIDATION=EXCLUSIVE, then any request which matches one of the USERID, TRANSID and PROGRAM validation rules will be rejected with ECI_ERR_INVALID_CALL_TYPE (-14).

Note: Configuration is based on a flexible set of rules defined in a configuration file. On z/OS, the configuration file can reside either on HFS or

in an MVS dataset. Rules for workload balancing can be generic per Gateway, based on server aliases, or on the payload type (for example,

channel or commarea). Rules for request validation can be specified based on user IDs, transaction IDs, or program names, and operate as

either a white list (INCLUSIVE) or black list (EXCLUSIVE). All rules can be dynamically updated while the Gateway daemon is running. A

configurable retry interval is provided to improve the efficiency of request distribution when using workload balancing.


ftp://public.dhe.ibm.com/software/htp/cics/support/supportpacs/individual/ca1t_v110.pdf 13

(cont.)

SupportPack CA1T V3.0 – CICS TG High Availability User Exits





ftp://public.dhe.ibm.com/software/htp/cics/support/supportpacs/individual/ca1t_v110.pdf


Sample output of CICS Request Exit –

03/11/13 08:26:38:943 CH51 Flow(XaPrepare) Srv(NEWCICSR) Trnid(null) Userid(WRDBC01) ClientIP(10.99.129.223)

RespTime(8)ms CICScall(8)ms

03/11/13 08:26:38:956 CH51 Flow(XaCommit) Srv(NEWCICSR) Trnid(null) Userid(WRDBC01) ClientIP(10.99.129.223)

RespTime(6)ms CICScall(4)ms

03/11/13 08:26:39:196 CH51 Flow(XaStart) Srv(NEWCICSR) Trnid(null) Userid(WRDBC01) ClientIP(10.99.129.223)

RespTime(0)ms

03/11/13 08:26:39:209 CH51 Flow(XaEci) Srv(NEWCICSR) Pgm(PRDFSLCT) MirrorTrnid(1234) Userid(WRDBC01)

ClientIP(10.99.129.223) RespTime(11)ms CICScall(10)ms

….

3/11/13 11:11:04:827 CH51 Flow(EciSynconreturn) Srv(OLDCICSR/NEWCICSR) Pgm(PRDPREPR) MirrorTrnid(ABCD)

Userid(PROD016) ClientIP(10.99.10.5) RespTime(43)ms CICScall(37)ms wait(2)ms

03/11/13 11:11:05:063 CH51 Flow(EciSynconreturn) Srv(NEWCICSRC/NEWCICSR) Pgm(PRDD026) MirrorTrnid(ABC3)

Userid(PRODSY4) ClientIP(10.99.11.32) commarea(28462)bytes RespTime(5)ms CICScall(5)ms

03/11/13 11:11:05:287 CH51 Flow(EciSynconreturn) Srv(OLDCICSR/NEWCICSR) Pgm(PRDPREPR) MirrorTrnid(ABCD)

Userid(PROD016) ClientIP(10.99.10.5) RespTime(17)ms CICScall(10)ms CICS TG_rc(-7) Abend(AEI0)

14

Support Pack CH51: CICS TG

Simple Request Monitor Utility


Set at CTGSTART_OPTS=

-j-Dcom.ibm.CTG.samples.requestexit.out=/usr/cicstg/CH51LOGS/CTGname01.log

-j-Dcom.ibm.CTG.samples.requestexit.log.number=5 -j-Dcom.

ibm.CTG.samples.haexit.out=//DD:HALOG -j-Dcom.ibm.CTG.samples.haexit.ca1t=1

-statsport 3101 -applid=CTGname -j-Xms256M -j-Xmx512M - j-

Xdump:system:dsn=yourprefix.CTGname.JAVA.DUMP

Logging levels available – set at -j-Dcom.ibm.CTG.samples.haexit.ca1t

15

Support Pack CH51: CICS TG

Simple Request Monitor Utility Setup

Can also issue console command. For example to enable level 3 logging issue the following

command - /F <JOB>,APPL=CREXIT,CMD=CA1T_3


This SupportPac allows integration test tools to intercept ECI calls to CICS TG from Java™ applications before they are processed by CICS TG.

Allows integration test tools ( for example Rational Test Workbench / Rational Integration Tester V8.0.1.1) to intercept External Call Interface (ECI) calls to IBM CICS TG, originating from Java Standard Edition (Java SE) and Java Enterprise Edition (JEE) applications. This, therefore, allows such applications to be tested without requiring a running CICS TG or CICS Transaction Server.

This SupportPac provides updated versions of the CICS TG V8.0 Java base classes and non-extended architecture (non-XA) ECI resource adapter. These updated files support the use of a Gateway intercept plug-in that can intercept application calls made through the Java Application Programming Interfaces (APIs). Documentation for the APIs is also provided by this SupportPac.

When enabled, a Gateway intercept plug-in is notified before a connection to a Gateway daemon is opened or closed, and before a request is sent to the Gateway daemon. Before a connection is opened, the plug-in can change properties of the connection object, and can prevent the connection from opening. Before a request is sent over a connection, the plug-in can change properties of the request object, and can cause the request to be returned to the application without being sent. Before a connection is closed, the plug-in can prevent the connection from closing. This allows a variety of scenarios to be simulated for application testing.

Can be downloaded from – http://www.ibm.com/support/docview.wss?uid=swg24033968

16

Support Pack CA5F: IBM CICS TG

'gateway' Intercept Plug-in Support

http://www.ibm.com/support/docview.wss?uid=swg24033968


( set in Runtime env CICSCLI)

SECTION IPICSERVER = SYS1CIC1

description = IPIC connection to CICS region #1

hostname = your.host.net

port = 3101

tcpkeepalive = Y

srvidletimeout = 60

srvretryinterval = 0

connecttimeout = 10

sendsessions = 200

ENDSECTION




port = 3102

tcpkeepalive = Y

srvidletimeout = 60


connecttimeout = 10

sendsessions = 200

ENDSECTION




port = 3103

tcpkeepalive = Y

srvidletimeout = 60


connecttimeout = 10

sendsessions = 200

ENDSECTION




port = 3104

tcpkeepalive = Y

srvidletimeout = 60


connecttimeout = 10

sendsessions = 200 ENDSECTION

On the Gateway you can set - CONNECTTIMEOUT, which is a numeric value between 0 and 3600, specifying the maximum

time in seconds that establishing a connection is allowed to take. The default value of 0 means that no limit is set. For

more details see - http://publib.boulder.ibm.com/infocenter/txformp/v7r1/index.jsp?topic=/com.ibm.cics.tx.doc/tasks/t_config_ipic.html

17

Sample CICS TG CONFIG File

http://publib.boulder.ibm.com/infocenter/txformp/v7r1/index.jsp?topic=/com.ibm.cics.tx.doc/tasks/t_config_ipic.html


CICS TG V8.1 and WebSphere V6, V7 & V8 - http://www-01.ibm.com/support/docview.wss?uid=swg21577454

Runtime Environments for Java and JEE Applications

18

CICS TG Compatibility with WebSphere





CICS TG V8.0 or prior has two CICS Resource Adapters – XA and non-XA.

CICS TG V8.1 has support for Java Enterprise Edition 6

The CICS connectors included with CICS TG V8.1 exploit JEE 6 / JCA 1.6

JEE 6 certified application servers (e.g. WAS 8.0) benefit from:

A single ECI resource adapter (rather than two) including support for both XA (two-phase

commit) and non-XA transactions

A more flexible approach to setting the transactional capabilities of the CICS ECI resource

adapter

Lower administrative and maintenance overheads

Resource adapters compatible with JEE 5 (or earlier) application servers are available

use with CICS TG V8.1 in remote mode only from Support Pac CC03

19

CICS JCA Resource Adapters


The CICS JCA resource adapters are provided, in order to support connectivity from JEE application servers to CICS using the facilities of the CICS Transaction Gateway.

The latest service levels of the CICS Transaction Gateway Java Enterprise Edition Connector Architecture (JCA) resource adapters for CICS TG are available for download for deployment into supported Java Enterprise Edition (JEE) application servers. These resource adapters can be

used to provide JCA connectivity into CICS using the facilities of a licensed CICS TG installation, and are designed around the JCA specification as supported by JEE application servers such as WebSphere Application Server and Oracle Weblogic Server.

This Support Pack includes the CICS JCA resource adapters from the following CICS Transaction Gateway for Multiplatforms releases: V7.0.1.1, V7.1.0.4, V7.2.0.4, V8.0.0.4, V8.1.0.1, V9.0.0.0

http://www-1.ibm.com/support/docview.wss?uid=swg24008817&myns=swgother&mynp=OCSSGMJ2&mync=E

20

Support Pack CC03: CICS JCA Resource

Adapters for Use With JEE Application Servers





J2C connection factories > CICS Connection Factory > Connection pools

Maximum connections Specifies the maximum number of physical connections that you can create in this pool. These are

the physical connections to the backend resource. Once this number is reached, no new physical

connections are created and the requester waits until a physical connection that is currently in use

returns to the pool, or a onnectionWaitTimeoutException is thrown. Ensure that there are no waits

for connections and that CICS can handle concurrent volume set here.

Minimum connections Specifies the minimum number of physical connections to maintain. If the size of the connection pool

is at or below the minimum connection pool size, the Unused Timeout thread does not discard

physical connections. Best to set to zero, to allow stale connections to be discarded.

Reap time Specifies the interval, in seconds, between runs of the pool maintenance thread. Do not set too

low, as it will increase the overhead.

Unused timeout Specifies the interval in seconds after which an unused or idle connection is discarded. Set the

Unused Timeout value higher than the Reap Timeout value for optimal performance.

Aged timeout Specifies the interval in seconds before a physical connection is discarded. Setting Aged Timeout

to 0 supports active physical connections remaining in the pool indefinitely. Do not set it at

zero to avoid connection retries on stale connections.

21

Connection Pool Tuning in WAS

https://pwauslwasmgt10.app.hcscint.net:9043/ibm/console/navigatorCmd.do?forwardName=J2CConnectionFactory.content.main&WSC=true

https://pwauslwasmgt10.app.hcscint.net:9043/ibm/console/j2CConnectionFactoryCollection.do?EditAction=true&refId=J2CConnectionFactory_1334082585850&contextId=cells:p-shrdm-was70-2Cell01:nodes:pwausxwasapp58Node01&resourceUri=resources.xml&perspective=tab.configuration


The graph shows the difference in processor usage cost between EXCI and IPIC.

The CICS address space costs were less when using EXCI, because of the efficiency of using cross-memory MRO for communication between the gateway address space and the CICS address space.

Athough the overall cost to use IPIC was greater, the amount of processing offloaded to a zAAP was also greater, which made the use of general-purpose CPs in the gateway address space less critical for IPIC.

22

EXCI vs. IPIC CPU Usage


Data from redpaper IBM CICS Performance Series: A Processor Usage Study of Ways into CICS

at: http://www.redbooks.ibm.com/redpapers/pdfs/redp4906.pdf

The graph shows the total microseconds of processor usage per transaction to get

32 KB of data into CICS and back to the client using the access methods tested

Environment Hardware

The following hardware was used: • IBM zEnterprise® 196 (Z196) 2817-779 model M80

• LPAR with three dedicated central processors (CPs)

and one dedicated IBM System z®

• (zAAP) on zIIP)

• DASD DS8800

• Internal coupling facility peer (ICP)

• 1 GB OSA-Express3

• Crypto Express3 coprocessor

• CP Assist for Cryptographic Function feature (enabled)

Software

The following software was used: • IBM z/OS® Version 1, Release 13

• CICS Transaction Server Version 4, Release 2

• IBM WebSphere MQ Version 7, Release 1

• Communication Server IP Sockets for CICS

• CICS Transaction Gateway Version 8, Release 1

23

CPU Cost Getting CICS Data

http://www.redbooks.ibm.com/redpapers/pdfs/redp4906.pdf

http://www.redbooks.ibm.com/redpapers/pdfs/redp4906.pdf


24

POC Testing – High zAAP Offload Rate


25

CA Wily Introscope CICS TG Metrics


26

Wily Introscope ECI Pool Size Metrics


Compare to week prior

27

Wily Introscope Response Time Metrics


28

Wily Introscope CICS TG/JCA Metrics


CICS Transaction Gateway - Activity Summary

_____________________________________________

0STTG0001 Printed at 11:29:15 3/01/2013 Data from 08:00:00 2/21/2013 to 08:00:00 2/21/2013 Page 1

CICS TG GATEWAY 02/21/13 REPORT

Gateway ID Start time Up time Number of Number of Peak Peak TPS time Connect Worker

Intervals requests TPS Timeouts Timeouts

ZOSCGRP0.CICS TG2P003 2013-02-14 06.10 7days 17.34.09 95 1643676 13 2013-02-21-17.15.00 0 0




Gateway ID Start time Collection time Region Java heap Clients Workers EXCI pipes IPIC sessions

Size/Max Size/Max Peak/Max Peak/Max Num/Max Num/Avail

ZOSCGRP0.CICS TG2P003 2013-02-14 06.10 2013-02-21-16.00.00 875M/1000M 71M/512M 19/500 3/800 0/0 0/400




CICS Performance Analyzer V5.1 now includes two CICS TG batch reports, "Activity Summary", and

"Usage and Capacity" which can be run against SMF 111 records from CICS TG for z/OS.

29

CICS Performance Analyzer for z/OS - Sample Reports


CICS Transaction Gateway plug-in for the CICS Explorer V2.0.0.0 is now available for CICS Explorer V5.1, easily installed through the common V5.1 update site which is pre-configured in the CICS Explorer "Install new software" facility.

30

CICS Transaction Gateway Plug-in For The CICS Explorer


CSSYS1CIC1_SIPADDR=yourSYS1.network.net (CICS server TCP/IP address)

CSSYS1CIC1_SIPPORT=3168 (CICS server TCP/IP port)

CSSYS1CIC1_LALLREQ=4549989 (Number of requests processed)

CSSYS1CIC1_CWAITING=-2 (Number of requests waiting on a response)

CSSYS1CIC1_SSESSMAX=200 (Number of requested IPIC sessions)

CSSYS1CIC1_CSESSMAX=200 (Number of negotiated IPIC sessions)

CSSYS1CIC1_CSESSCURR=0 (Number of IPIC sessions in use)

CSSYS1CIC1_LCOMMSFAIL=49 (Number of CICS communication failures)

CSSYS1CIC1_LCONNFAIL=4484 (Number of connection failures)

CSSYS1CIC1_LLOSTCONN=32 (Number of lost connections)

CSSYS1CIC1_LSESSFAIL=0 (Number of IPIC session failures)

CSSYS1CIC1_LIDLETIMEOUT=0 (Number of timed out connections)

CSSYS1CIC1_LREQDATA=28649890161 (Amount of CICS request data)

CSSYS1CIC1_LRESPDATA=29043330152 (Amount of CICS response data)

CSSYS1CIC1_LAVRESP=64 (Average CICS response time)

CSSYS1CIC1_CAPPLIDQ=ILA (APPLID qualifier of connected CICS)

CSSYS1CIC1_CAPPLID=YourCICSApplid (APPLID of connected CICS)

CSSYS1CIC1_ICOMMSFAIL=0 (Number of CICS communication failures)

CSSYS1CIC1_ICONNFAIL=0 (Number of connection failures)

GD_IAVRESP=42 (Average Gateway daemon response time)

GD_LAVRESPIO=109 (Average Gateway daemon response time with

I/O)

GD_IAVRESPIO=98 (Average Gateway daemon response time with

I/O)

GD_SNAME=CICS TGPRD1 (Gateway daemon name)

GD_CHEALTH=100 (Gateway daemon health)

GD_LHAEXIT=4687171 (Number of CICS request exit calls)

GD_IHAEXIT=642 (Number of CICS request exit calls)

GD_SDFLTSRV= (Default server name)

GD_LRUNTIME=1758839 (Gateway daemon running time)

GD_SSTATINT=001500 (Length of the statistics interval HHMMSS)

GD_SSTATEOD=000000 (Logical End Of Day time HHMMSS)

GD_CNEXTRESET=150000 (End of interval time HHMMSS)

GD_IRUNTIME=199 (Interval running time)

WT - Worker thread

WT_SMAX=800 (Maximum number of worker threads)

WT_SINIT=200 (Initial number of worker threads)

WT_CCURR=200 (Current number of worker threads)

WT_CALLOC=0 (Currently allocated worker threads)

WT_LTIMEOUTS=0 (Number of times worker time out reached)

WT_ITIMEOUTS=0 (Number of times worker time out reached)

With CICS TG on z/OS stand alone we can use MVS console commands to display statistical

information about the CICS TG, or can obtain statistics using the Statistics API interface.

/F CICS TGname,APPL=STATS,GS

31

MVS Console Commands


If you would like to analyze CICS TG SMF 111 records, 1st you would need to extract these

records in a separate dataset by the IFASMFDP utility. Here is a sample JCL -

//* COPY SMF RECORDS FROM DAILY TAPES

//* THIS PARTICULAR JOB COPIES SMF 111 RECORDS FOR CTG

//* BEFORE RUNNING, SET DATE AND LPAR

// SET SMFHLQ=TSMF <<== HLQ OF SMF

// SET SMFTYPE=CTG

// SET LPAR=A <<== LPAR <- set LPAR name

// SET DATE=D032013 <- set date

//STEP2 EXEC PGM=IFASMFDP

//SYSPRINT DD SYSOUT=*

//SMFIN DD DISP=SHR,DSN=&SMFHLQ..SMF.&LPAR.NOSSDAY.&DATE

//DUMPOUT DD DSN=&SMFHLQ..SMF.&LPAR..&SMFTYPE..&DATE,

// DISP=(NEW,CATLG),UNIT=3490E,RETPD=9

INDD(SMFIN,OPTIONS(DUMP))

OUTDD(DUMPOUT,TYPE(111))

You can collect and report on the following statistics -

CICS Server Instance (CSx) SMF statistics for EXCI

CICS Server Instance (CSx) SMF statistics for IPIC

Gateway daemon (GD) statistics SMF data

Protocol handler (PH) statistics SMF data

System environment (SE) statistics SMF data

Worker thread (WT) statistics SMF data

For more details please see 'CICS Transaction Gateway Version 8 or 9 z/OS Administration' – section 'Displaying statistics‘. 32

CICS TG SMF 111 Records Reporting CICS TG SMF 111 Records Reporting


Inquiry : CICLDG CA MICS - CICS Transaction Gateway Daily Thr

Run Date: 07MAR13 Data Presented by HOUR for each CT

REPORT PAGE: Data for Date=06MAR SYSID : 1

Gateway ID: A001 APPLID: CICS TG1P001 Release Level:8.0.0

+------------- Gateway Daemon Activity ------------+

Day Client Client Client Client Client CICS

of Requests Request Resp Request Avg Resp Requests

Week HOUR Processed MB Data MB Data per/sec Time Processed

---- --------- ------- ------- ------- -------- -------- -------- Wed 1 16272 158.01 188.92 4.520 0.1250 1627

….

6 28525 9845.18 9877.02 7.924 0.0890 2852

7 57660 17330.8 17389.7 16.017 0.0820 5766

8 92410 17048.2 17124.3 25.669 0.0920 9241

9 204752 16063.8 16036.2 56.876 0.0690 20475

10 245463 8153.38 8069.41 68.184 0.0730 24547

11 270587 7347.09 7206.97 75.163 0.0680 27057

12 224123 12615.7 12494.0 62.256 0.0630 22412

13 197175 9026.42 8954.67 54.771 0.0670 19717

14 224994 7117.15 6964.56 62.498 0.0710 22499

15 201345 2994.01 2853.23 55.929 0.0560 20134

16 183724 3153.67 3025.13 51.034 0.0500 18372

17 144724 2194.84 2086.04 40.201 0.0540 14472

33

Sample MICS CICS TG Report


34

Other Monitoring Tools Available

IBM Tivoli OMEGAMON XE for CICS on z/OS V5.1.0


35

Performance Tuning Helps to prevent CICS becoming overloaded or underutilized

Allows for optimization of workflow through CICS TG and onto CICS

Alerts users to changes in behavior when abnormal workloads hit CICS TG

Provides users with an account of transactions per minute and CPU

consumption

Availability Alerts users to communication problems between CICS TG and CICS

Managablity Helps to determine where to control resources in CICS TG or CICS

Monitoring Transaction Monitoring in Gateway daemon and WebShere Application

Server on z/OS

Transaction Response Time Analysis, Payload and Wiresize per

transaction

End To End Response Time Analysis

Problem Determination Identifies the bottlenecks within CICS TG daemon

Identifies Abended transactions

Correlates all application Flows within a Transaction

Provides Transaction Flow Trace

Capacity Planning Helps to plan for workload growth, perform trend analysis, detect system

anomalies and to identify performance bottlenecks using the historical data

via the Tivoli Data Warehouse repository

Total Cost of Ownership Opens up the “black-box” of CICS TG thread management and helps to

save precious resources in both CICS TG and CICS

Other Monitoring Tools Available

IBM Tivoli OMEGAMON XE for CICS on z/OS V5.1.0


Issue

We implemented new Application using CICS TG on z/OS, which had invalid Userid set on Connection Factory. This caused ID to get suspended and Application was redriving error routine to restablish CICS TG connection, after each failure. This caused connections not getting released, causing all connections to be in use across all 4 CICS TGs, resulting in CICS TG -16 return (ECI_ERR_RESOURCE_SHORTAGE).

Solution

Setup automation to issue CICS TG console command –

/F CICS TGname,APPL=STATS,GS every 5 minutes and send alert if field CS_CSESSCURR (total # of IPIC sessions in use with all CICS servers) is at 80% or higher from CS_CSESSMAX (# of IPIC sessions negotiated with CICS servers). Can also have automation (OPS/MVS for example) check the values of CM_CALLOC and WT_ALLOC and alert if the threshold set has been passed.

CICS TG on z/OS V8.0

PM38908: ECI_ERR_RESOURCE_SHORTAGE reported when sending extended mode ECI requests over an IPIC connection to a CICS server

Error description

When an extended mode ECI request does not complete successfully (possibly due to security reasons) before the CICS mirror transaction is attached, the IPIC session is not freed.

http://www-01.ibm.com/support/docview.wss?uid=swg1PM38908

36

Issues To Watch Out For






GD_IAVRESP - The average time taken in milliseconds for the Gateway daemon to respond to API (ECI, ESI, EPI)

and XA requests from remote clients. Successful and failed requests are included. This value is inclusive of the CICS

response time, as provided by the corresponding CS_IAVRESP statistic.

GD_IAVRESPIO - The average time in milliseconds for the Gateway daemon to respond to API (ECI) and XA

requests from remote clients including network I/O time. Successful and failed requests are included. This value is

inclusive of the Gateway response time, as provided by the corresponding GD_IAVRESP statistic.

The latest versions of the CICS TG plug-in for CICS Explorer V1.1 for CICS Explorer V1.1 and V2.0 for CICS Explorer

V5.1 now shows Network Latency data. You can get lifetime average value for network latency, under the column

"GwyRespTimeIO", and is included in the pre-defined Column layout "Requests", of the Gateway daemons view.

37

CICS TG V8.1 New SMF Network Latency Data


Limitations of 31-bit Gateway daemon

31-bit Gateway daemon, 375MB heap, REGION 0M (1632MB)

1350 Connection Mangers

999 Worker threads

1 IPIC connection with 999 sessions

32KB payload

64-bit Gateway daemon scalability

Note: Data above is a rough guide, your ‘milage’ may vary …

38

CICS TG Scalability


For best performance with 64-bit HWM, use compressed references

(-j-Xcompressedrefs)

Migration to 64-bit can reduce number of address spaces

Fully exploit IPIC and Dynamic Server Selection after migrating from the

constraint of 250 EXCI pipes

Rule of thumb for Gateway MEMLIMT

MEMLIMIT should be at least

128MB (core) + Heap size + (#CM+WT threads) * 3MB

39

Tuning Suggestions


GA December 14, 2012

Please do not miss IMPACT session TAC-3022 ‘Connecting to CICS with CICS

Transaction Gateway V9’ scheduled for 3:45pm on May 1st, presented by Andrew Smithson.

40

CICS Transaction Gateway V9.0


Routing or filtering by client IP address

CICS Request Exit now provides the TCPIP address of the application

originating the request

InetAddress map object “ClientLocation”

Allows routing decisions based upon client location (e.g. subnet)

Allows rejection of requests from unauthorised locations

Full request details for XA transactions

CICS Request Exit allows custom Dynamic Server Selection

CICS TG V8 has limited request meta-data for XA transactions

CICS TG V9.0 provides all available request meta-data for both XA and non-

XA ECI requests

Allows customized routing logic for High Availability solutions with full

distributed two-phase commit

New Request Monitoring exit point “RequestDetails”

Invoked just before calling CICS, after Dynamic Server Selection

Allows clear visibility of target CICS server for HA solutions

Very useful for time outs, or hung request scenarios 41

CICS TG V9 HA Enhancements


Cross component trace (XCT) contexts

Available on requests originating from WebSphere

Application Server V8.5, with XCT support enabled

Enabled/disabled

• Dynamically through the WAS admin console run time options

• Statically through the WAS admin console configuration

Uses High Performance Extensible Logging (HPEL)

XCT contexts

XCT context Begin and End demarcate component boundaries

• Contexts are hierarchical

A thread of execution can have upto 3 XCT contexts at any one time:

• Root – initial context (Request ID) of the component at the point of entry

• Parent – context of the calling component

• Current – context of the current component 42

CICS TG V9 Cross Component Trace


43

CICS TG V9 Enhancements To IPIC Connectivity


RFE ID Headline

17173 Asynchronous call support for ECI Version 2 applications

17258 CICS Gateway Support for OEL 5u4

19319 CICS TG Explorer Plugin - Logical grouping of Gateway daemons

21081 Transaction tracking from WAS into CICS through CICS TG

21082 Support for Gateway daemon running in a 64-bit JVM

21085 Additional exit point for Request Monitoring

21633 64-bit compile and run time support for ECI Version 2

21642 Remote Mode SSL for IPIC Connections

21828 Bi-directional language support

21871 Make program and transaction available to the CICS Request Exit for XA transactions

21872 ECI_TIMEOUT default value for IPIC connections

21875 64-bit C module compilation support for ECIv2 on AIX (IBM POWER) and Red Hat (x86)

22036 CICS Request Exit to provide client IP address

22049 Assembler DSECTs for mapping the SMF 111 records

22081 Sample SMF formatter CICS TGSMFRD support for mixed-release installations

22088 Information about channels in request monitoring exits

22108 Asserted identity support for Multiplatforms C applications making ECI requests

See details at http://www-01.ibm.com/support/docview.wss?uid=swg27037911

44

Customer Requirements Answered in CICS TG V9.0

http://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=17173





































45

Questions


We love your Feedback!

Don’t forget to submit your Impact session and speaker

feedback!

• Your feedback is very important to us – we use it to improve

next year’s conference

• Go to the Impact 2013 SmartSite (http://impactsmartsite/com):

– Use the session ID number to locate the session

– Click the “Take Survey” link

– Submit your feedback

46

http://impactsmartsite/com


Legal Disclaimer

• © IBM Corporation 2013. All Rights Reserved.

• The information contained in this publication is provided for informational purposes only. While efforts were made to verify the completeness and accuracy of the information contained

in this publication, it is provided AS IS without warranty of any kind, express or implied. In addition, this information is based on IBM’s current product plans and strategy, which are

subject to change by IBM without notice. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, this publication or any other materials. Nothing

contained in this publication is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and

conditions of the applicable license agreement governing the use of IBM software.

• References in this presentation to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or

capabilities referenced in this presentation may change at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to

future product or feature availability in any way. Nothing contained in these materials is intended to, nor shall have the effect of, stating or implying that any activities undertaken by

you will result in any specific sales, revenue growth or other results.

• If the text contains performance statistics or references to benchmarks, insert the following language; otherwise delete:

Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will

experience will vary depending upon many factors, including considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage

configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here.

• If the text includes any customer examples, please confirm we have prior written approval from such customer and insert the following language; otherwise delete:

All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual environmental costs

and performance characteristics may vary by customer.

• Please review text for proper trademark attribution of IBM products. At first use, each product name must be the full name and include appropriate trademark symbols (e.g., IBM

Lotus® Sametime® Unyte™). Subsequent references can drop “IBM” but should include the proper branding (e.g., Lotus Sametime Gateway, or WebSphere Application Server).

Please refer to http://www.ibm.com/legal/copytrade.shtml for guidance on which trademarks require the ® or ™ symbol. Do not use abbreviations for IBM product names in your

presentation. All product names must be used as adjectives rather than nouns. Please list all of the trademarks that you use in your presentation as follows; delete any not included in

your presentation. IBM, the IBM logo, Lotus, Lotus Notes, Notes, Domino, Quickr, Sametime, WebSphere, UC2, PartnerWorld and Lotusphere are trademarks of International

Business Machines Corporation in the United States, other countries, or both. Unyte is a trademark of WebDialogs, Inc., in the United States, other countries, or both.

• If you reference Adobe® in the text, please mark the first use and include the following; otherwise delete:

Adobe, the Adobe logo, PostScript, and the PostScript logo are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States, and/or other countries.

• If you reference Java™ in the text, please mark the first use and include the following; otherwise delete:

Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both.

• If you reference Microsoft® and/or Windows® in the text, please mark the first use and include the following, as applicable; otherwise delete:

Microsoft and Windows are trademarks of Microsoft Corporation in the United States, other countries, or both.

• If you reference Intel® and/or any of the following Intel products in the text, please mark the first use and include those that you use as follows; otherwise delete:

Intel, Intel Centrino, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and

other countries.

• If you reference UNIX® in the text, please mark the first use and include the following; otherwise delete:

UNIX is a registered trademark of The Open Group in the United States and other countries.

• If you reference Linux® in your presentation, please mark the first use and include the following; otherwise delete:

Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both. Other company, product, or service names may be trademarks or service marks of

others.

• If the text/graphics include screenshots, no actual IBM employee names may be used (even your own), if your screenshots include fictitious company names (e.g., Renovations, Zeta

Bank, Acme) please update and insert the following; otherwise delete: All references to [insert fictitious company name] refer to a fictitious company and are used for illustration

purposes only.

47

http://www.ibm.com/legal/copytrade.shtml


Please Note

IBM’s statements regarding its plans, directions, and intent are subject to change

or withdrawal without notice at IBM’s sole discretion.

Information regarding potential future products is intended to outline our general

product direction and it should not be relied on in making a purchasing decision.

The information mentioned regarding potential future products is not a

commitment, promise, or legal obligation to deliver any material, code or

functionality. Information about potential future products may not be incorporated

into any contract. The development, release, and timing of any future features or

functionality described for our products remains at our sole discretion.

Performance is based on measurements and projections using standard IBM

benchmarks in a controlled environment. The actual throughput or performance

that any user will experience will vary depending upon many factors, including

considerations such as the amount of multiprogramming in the user’s job stream,

the I/O configuration, the storage configuration, and the workload processed.

Therefore, no assurance can be given that an individual user will achieve results

similar to those stated here.

48

Ibm session tac 2104 - ctg presentation for impact 2013 final

Technology

cics tg cics tg

zos cics

cics health

cics web support cics

cics tg versions

cics medium

cics applications

hcsc cics tg ha architecture