Z/TPF Application experiences and tools By Michael Bahlen Applications Development KLM TPFug 21 April 2009 Lincolnshire IL.

z/TPF Application experiences and tools

ByMichael Bahlen

ApplicationsDevelopmentKLM

TPFug21 April 2009Lincolnshire IL

21-Mar-2009z/TPF Experiences and Tools2

Contents

• Where did we start?• Where are we now?• Single Source, re-compile and re-assembly• 1st z/TPF system up• Testing and the tools used• Keeping the sources in sync between TPF4.1 and z/TPF• Most common errors• Demo• Questions


Where did we start?

• At the start of the z/TPF project we were (and still are) running TPF4.1 put 19+

• Started Single Source Updates in October 2007• Started re-compile and re-assembly in 1st quarter 2008• Started z/TPF testing 2nd quarter 2008• Integration testing started December 2008• Acceptance testing started March 2009


Where are we now?

• We are in the middle of the acceptance tests, and have at this moment no NO-GO items open

• All but 1 test-system are converted to z/TPF• Application development is already developing future

enhancements for z/TPF• Planned cut-over on May 16th


Single Source, re-compile and re-assembly

• Single Source was applied to the software in about 3 months by 4 programmers, testing and loading to TPF4.1 took over a year.

• Re-compile/re-assembly took 2 weeks, solving the problems of this step took another month.

• During the process of testing and applying new APARS we had to re-compile/re-assemble several times

Note: For information on Single Source and re-compile/re-assembly see my presentations on previous TPFug’s.


1st z/TPF system up

• Around May-June 2008 the 1st test-system running with z/TPF based on our own builds came up.

• At first this was a vanilla system, but after a short time it included most of the KLM modifications and all applications

• Since this project was also a nice way of doing a clean-up of things no longer used, we started of with many snap-63’s.

• It took a few weeks to solve most of these problems


Testing and the tools used

• What did we use for testing before z/TPF?– SST– Debugger– ODF (Online Dump Facility)– ERREPA (Sort of snap-dump)– Stress tests

• What was available for testing on z/TPF?– Debugger

• Was this enough?– NO! But we had to start with only the debugger.

• How to go on from here?– Convert the tools from TPF4.1 to z/TPF! This excluded

SST, its not a KLM product and it’s too much work.


Testing and the tools used

• So before the tools were converted we had to do with the IBM dump list display (very short).

• Post processing dumps (time consuming).• Use the debugger, which was highly unstable; we had

to apply many APARS before it started to be useful.

Later ODF and Stress-test came along and testing became easier


Stress-test

What is the KLM stress-test and how does it work?• It is based on another tool KLM built into TPF called

PMC (Permanent Message Collector)It writes all incoming and outgoing messages to RTAThis is the base for stress testing

• What else is needed for stress-testing?Full capture of the production database and restore onto a test-system.

• This database, with a special extraction of PMC messages from RTA are the base to perform a stress-test

• The database is a certain point in time of the production system, sending in the input PMC messages on a high speed basis into this system is a stress-test


Stress-test

• Results:– (Snap) Dumps– Other Error messages– Console logging– RTA tape with PMC messages (can be used to compare

output on same entry, but is a lot of work)

• At first we ran a few 100,000 messages and had a lot of things to look into (also stress-test related)

• Now we are processing between 3mln and 5mln messages with almost no problems.


Online Dump Facility

• During the 1st tests, ODF came along with more and more details of dumps.

• At 1st it gave a full report on all (snap) dumps that happened on the test system.

• Secondly more and more details of the data dumped came into the displays, helping the programmers with direct information of what happened.For example: Displacement of the dump in the causing program, the program itself, data blocks attached to the dump

• Last but not least the Macro trace, which provides much more information in z/TPF.


ERREPA

• ERREPA is KLM’s version of a snap-dump and was created (1989) before the snap-dump existed.

• This was the 1st KLM tool completely converted. Some parts had to be re-written to 64-bit code due to the fact that lots of C-programs passed constant strings.

• Many applications contain ERREPA code, so a lot of these type of reports were investigated.

• Later with ODF available, we could use the full-dump option on the ERREPA, to get more information out of these error situations.


Debug view in toolkit

• Before ODF was available on z/TPF we used the TPF-Toolkit dump viewer.

• Shows a lot of details about the dump, but since the Toolkit – z/TPF relation was still highly unstable it was causing dumps as well

• Also the view shows a list of dumps that are only sequence numbers, finding your dump was trial and error.


Scripts

• Some tests where performed through scripting (called run-files in KLM)

• These scripts log every in and output to file, running the script on a TPF4.1 system and a z/TPF system gives 2 output files.

• Comparing these files resulted in finding several problems.


Keeping the sources in sync between TPF4.1 and z/TPF

• To understand how we did this a little look into our development environment is needed.

TPF4.1 environment

• USS based

• Repository in DB2 under z/OS

• Idefix Package

• IdefixGui on PC

• NFS maestro for PC – USS connection

z/TPF environment

• z/Linux based

• Repository in DB2 under z/OS

• z/Idefix Package (ported and extended Idefix)• TPFfix on PC• NFS maestro for PC – z/Linux connection

TPFfix was started by me as a replacement for the IdefixGui as IdefixGui had too many limitations

TPFfix stands for: The Professional Fantastic File Interface eXtension (Yes it was started as a joke)



• TPFfix was extended to be able to work with both Idefix versions, so through this interface a coupling was possible.

• TPFfix is and was extended by many special functions to co-op with transporting TPF source from Idefix to z/Idefix

• Also functions are available to keep both program bases in sync.

• How is this done?



• In Idefix every component has a name, version, status, language and platform.

• In z/Idefix only the version and status are different.• Idefix and z/Idefix allow a comment in the metadata.• The copy and update functions follow these metadata

comments and the version and status in Idefix.• As soon as a component is copied, the metadata

comment will show ‘Copy41 version:9AOPR’• At each update round, this comment is compared to

the ‘new’ version in Idefix, if there is a difference, a copy is made and the metadata comment is also updated.



• To keep track of updates made for the z/TPF system, this comment can also have the following syntax:– Updated!9AOPR– Differs!9AOPR

• The “Updated!” means that the component has updates specially for z/TPF which are possible on TPF4.1 but not (yet) applied on TPF4.1

• The “Differs!” means that the component has updates specially for z/TPF, that are not possible on TPF4.1, or specific conditional code is needed.

• In these cases, an upgrade from TPF4.1 involves manual labor!


Most common errors

• C-language– String arrays too short– Passing wrong type of pointers to assembler (stack programs,

KLM specific)

• Assembler language– Use of R14, R15 after IBM macro’s– Pointers passed to C (pointer to pointer type)– CPROC created for assembler program– No CPROC created for C-program

• Most curious one: LTR R0,R0 FLIPC D2,D5 BZ SOMEWHEREOf course this was dinasour code


Demo

Demo includes:• Some ODF displays• Some ERREPA displays• Some TPFfix views

Note: The following slides will show some examples of the live demo


ODF examples (report)

UODF0001I 18.10.16 Online Dump Error Report * CpuID-DDate Time IS Seq. Type Addr Prog Trc DupsRTA/RTL Volume: C0003616MAR 17.16.27 02 SN-000065 SNP-S0BDBD110 C0014F YLID 2RTA/RTL Volume: C0003516MAR 12.10.30 01 SE-002361 OPR-I000004 E99A00 CP CPSE 116MAR 12.10.30 01 SE-002360 OPR-I000004 E99A00 KQT90A KQT9 116MAR 11.57.45 02 SE-002359 OPR-SDBCB20 D703FA Y77E1A Y77E 816MAR 11.43.04 *** System Up ***16MAR 11.43.00 01 SE-002358 CTL-I007856 8 RTA/RTL Volume: C0000716MAR 11.43.14 02 SE-002258 CTL-I0000E2 010000 CP NSTA 16MAR 11.43.14 03 SN-000064 SNP-S0BDBD11A C05812 YLID 616MAR 11.43.14 02 SN-****** SNP-S0BDBD110 C0BEA3 YLID 1016MAR 11.42.48 *** System Down ***End Of Display


ODF examples (Normal dump display)

UODF0007I 18.14.01 Online Mini Dump Display SE-002360Date - 16MAR Time - 12.10.30 OPR - I000004 Terminal

- E99A00Program - KQT90A Trace - KQT9 Offset - 0000095C Loadset

- BASE Object - kqt9 SS - BSS SSU - BSS CpuId - D IS - 01 ADSpace

- EVM

*** Program Interrupt Code - 0038 - ***ASCE-type Exception

*** General Registers At Dumptime ***R0 275CEE0A R1 275CEE3CR2 275CEE0D R3 275CEE17R4 275CEE1B R5 275CEE25R6 275CEE2E R7 C9R8 5514078 R9 25300000R10 1 R11 1000R12 2000 R13 2750F9C0R14 2750F780 R15 F20000002750F9C0

Continued


ODF examples continuation

*** Current PSW At Time Of Dump *** 64PU1 0715000180000000 000000000551495C

*** Data Areas ***ECB At 25459000 For 12288 BytesD0 At 2530C000 For 4096 BytesD1 At 25306BE0 For 1056 BytesCSTATIC At 381317000 For 4096 BytesRCT At 500C7D008 For 36 BytesRLT At 500C76E50 For 104 BytesTRACES At 2B800000 For 11648 BytesREGAREAS At 24675200 For 1408 BytesHEAP At 275BA100 For 168 BytesHEAP At 275CEBE0 For 8008 BytesHEAP At 275D0B28 For 32767 Bytes

*** DECB Areas ***DECB FRAME 1 At 25308000 For 4096 BytesDECB 001 ATTACHED BLOCK At 25312000 For 4096 BytesDECB 002 ATTACHED BLOCK At 2530A000 For 4096 BytesEnd Of Display


ODF examples (DECB)

UODF0007I 18.19.08 Online Mini Dump Display SE-002360

DISPLAY IN USE DECB DATA LEVELS

INDEX NAME FARW CBRW DET

001 IDFSLEVELDD@ 00000000 00000000 00000000 25312000 4 000

002 IDFFRW1FKL . C16F8F01 00000000 FFFFFFFF 2530A000 4 000

End Of Display


ODF examples (DECB/DF block)UODF0007I 18.20.58 Online Mini Dump Display SE-002360

DECB ATTACHED CORE BLOCKFRAME 25308000 DECB 002 BLOCK 2530A000 FFF Bytes

TPFDF Formatted Filename - RW1FKL

Record Content And Current Pointer *---*0001 0000001A 2530A01A 000F80 G E N1F1D 1C H W P 000000 0002 00000029 2530A029 003A80 U N B I A T B 1 K L 1 E T 00000000 1 A P P C000000 00 0 9 9 1 E 1 3 0 0 0 0 A 0 E00 00000000 00000000 00000000 0000 0003 00000063 2530A063 004280 U N H 200 00000000 00000000 00000000 H W P R E S 9 6 2 I A 0 9 9 1 E 1 3 0 0 0 0 A 0 E61 E 9 9 A 0 0 00000000 00000000 00000000 0000 0004 000000A5 2530A0A5 005180 P O S0000 1 A40 M U C4040 1 B40 S G4040 S I N4040 0000 A 3 7 9 2 0 0 1 340 404040 B 2 1 0 0 E80 S I N 1 B 2 1 0 000 2680 R C I 1 A00 3 5 5 6 2 W0000 00000000 00 0005 000000F6 2530A0F6 000D80 M S G0000 0000 300 7 0006 00000103 2530A103 002680 R C I 1 A 00 3 5 5 6 2 W00 00000000 00000000 00000000 00 K L00 T K X G H D 0007 00000129 2530A129 001480 U N T 200 00000000 00000000 00000000 0008 0000013D 2530A13D 001580 U N Z 1 0 9 9 1 E 1 3 0 0 0 0 A 0 E End Of Display


ODF examples (Trace)UODF0007I 18.28.09 Online Mini Dump Display SE-002360*** Macro Trace ***Psw Macro Prog Obj Disp Lsn Time of macro ISKLMAPPL :64SM0 KJVE0A kjve.0a F9C 01:57:13.014048 01 KJVE

IZTPFDF :64SM0 CTDF7A ufge.3a 6E0 01:57:13.014047 01 __dbcls

31PU1 BACKC UTDF3J uad0.2a 1AE BASE 01:57:13.014047 01 R0 F0 R1 2750E720 R2 381314522 R3 0 R4 0 R5 0 R6 275CEE25 R7 3DDA14060 R8 2750E3A8 R14 A185A1B2 R15 2750E3A8

31PU1 DECBC UTDF3J ubc0.3a 11FE BASE 01:57:13.014047 01 Type-Release AD-253084A0 R0 0 R1 2750E720 R2 325314000 R3 275D1500 R4 25308020 R5 253084A0 R6 275C5192 R7 3A1887CF0 R8 21883380 R14 253084A0 R15 1

31PU1 EHEAPC UTDF3J ubc0.3a 1192 BASE 01:57:13.014046 01 R0 0 R1 2750E720 R2 325314000 R3 275D1500 R4 25308020 R5 253084A0 R6 275C5192 R7 3A1887CF0 R8 21883380 R14 275C5130 R15 1

31PU1 EHEAPC UTDF3J ubc0.3a 1180 BASE 01:57:13.014045 01 R0 1 R1 2750E720 R2 325314000 R3 275D1500 R4 25308020 R5 253084A0 R6 275C5184 R7 3A1887CF0 R8 21883380 R14 275B8000 R15 1


ERREPA (report)

ERREP OVERVIEW FOR PROJECT : RES

DATE: 15MAR

6149 03.06 KOTZ 0604BE 2 6155 11.40 KOTZ 0604BE 1

6156 16.46 T910 06013A 19 6157 17.02 KGRN 060233 1

6158 17.08 KGRM 060567 1 6159 21.21 TIV2 060511 2

6160 22.15 T12D 06012C 5

DATE: 16MAR

6164 16.10 T910 06013A 3 6165 17.03 KGRN 060233 1

6166 17.10 KGRM 060567 1 6167 18.10 TAC4 06054A 1

6168 18.45 KSBA 06FFFD 2


ERREPA (display)

16MAR09 17.03.43 060233 RES PGM:KGRN SYS:600 SEQNR:6165 000000

*** PROGRAM NESTING AREA ***LVL BASE NSI PAT PBI PGM0 00000000069C4078 00000000869C41C4 0000000023A50E60 0000 KGRN0A1 00000000069BC078 00000000869BC898 0000000023A51140 0000 KGRO0A2 00000000069B4078 00000000869B4412 0000000023A502E0 0000 KGRB0A3 00000000069A6078 00000000869A62FE 0000000023A505C0 0000 KGRC0A4 0000000000000000 0000000000006133 0000000000000000 0000

GIF CROSSCHECK

NUMBER IN PARTY GROUP DOESN'T MATCH NUMBER IN PARTY OF PNR'S

GROUPID: 325077


TPFfix (normal view)


TPFfix (actions on components)


TPFfix (Xref-view)


TPFfix (label search results)

z/TPF Application experiences and tools

Questions?

[email protected]

Z/TPF Application experiences and tools By Michael Bahlen Applications Development KLM TPFug 21 April 2009 Lincolnshire IL.

Documents

started ztpf testing

ztpf application experiences

ztpf project

ztpf system uptesting

acceptance testing

tools1st ztpf system

reassemblysingle source

easierztpf experiences