Development of Local COBOL Programs

®

IBM Software Group

© 2008 IBM Corporation

RDz Workbench – Integration with Fault AnalyzerRDz Workbench – Integration with Fault Analyzer

Jon Sayles/IBM Rational Eco System TeamJon Sayles/IBM Rational Eco System Team

2

IBM Trademarks and Copyrights © Copyright IBM Corporation 2007,2008, 2009. All rights reserved.

The information contained in these materials is provided for informational purposes only, and is provided AS IS without warranty of any kind, express or implied. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, these materials. Nothing contained in these materials is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software. References in these materials to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates.

This information is based on current IBM product plans and strategy, which are subject to change by IBM without notice. Product release dates and/or capabilities referenced in these materials may change at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any way.

IBM, the IBM logo, the on-demand business logo, Rational, the Rational logo, and other IBM Rational products and services are trademarks or registered trademarks of the International Business Machines Corporation, in the United States, other countries or both. Other company, product, or service names may be trademarks or service marks of others.

3

Course Contributing Authors Thanks to the following individuals, for assisting

with this course: Russ Courtney/IBM Reginaldo Barosa/IBM-Rational David Bean/IBM-Rational Ed Steele/IBM-Rational Dave Willsey/IBM-Rational

4

Course Overview Audience

This course is designed for application developers who have learned or programmed in COBOL, and who need to do z/OS Traditional Development and Maintenance as well as build leading-edge applications using COBOL and Rational Developer for System z.

Prerequisites This course assumes that the student has a basic understanding and knowledge

of software computing technologies, and general data processing terms, concepts and vocabulary, as well as a working knowledge of COBOL and z/OS.

Knowledge of SQL (Structured Query Language) is assumed for database access is assumed as well.

Basic PC and mouse-driven development skills, terms and concepts are also assumed.

5

Course Topics Course Name: Rational Developer for System z Foundation Training Course Description: Learn how to use Rational Developer for System z to do z/OS traditional

development, maintenance, support and for Enterprise Modernization of z/OS applications Pre-requisites: Some experience developing COBOL applications using z/OS is expected. A working

knowledge of SQL is also recommended.

Course Length: ~5days – or if done in self-paced mode, at your own pace Topics (Agenda)

Getting Started - installing and configuring RDz - and the course materials, and using Eclipse The RDz Workbench

– Code analysis tools– Editing – Compiling programs– Debugging local COBOL programs

The Data Perspective:– Working with relational data sources– Modifying test data– Editing and testing SQL statements

Working with remote system resources:– Connecting to a mainframe– Data management – Accessing and editing files

z/OS Application Development– Creating MVS Subprojects– Creating and customizing project properties

Debugging z/OS Applications– Debugging Batch Applications– Setting Debug Tool for Online Applications

Working with File Manager– Creating test data– Editing complex file-types

Working with mainframe ABENDs using Fault Analyzer– Creating Fault History views– Analyzing and solving mainframe ABENDs

Creating and modifying BMS Maps using the BMS Map Editor

6

UNIT

Topics:

The RDz WorkbenchThe RDz Workbench

Analyzing Mainframe AbendsAnalyzing Mainframe Abends ABEND Codes and Reasons Using IBM's Fault Analyzer Appendix

7

Unit objectivesAfter completing this unit on Production Support/Application Testing/Software Defect

and IBM Mainframe COBOL ABEND Research, you should be able to:Define the steps in a generalized methodology of ABEND resolutionList the various sources of ABEND inputs, including:

PD Tools documents Other sysout Dynamic trace facilities

List the common types of COBOL program ABENDS

8

ABEND OverviewABEND Overview When an application ABEND (ABnormal END-of-job) occurs, z/OS stops executing

your program, closes files and buffers and generates a single high-level message in the form of a System Completion Code (Sxxx).

The System Completion Code is usually written to an output listing file through your //SYSOUT DD *//SYSOUT DD * JCL entry.

This completion code indicates why the system has decided to stop executing your application.

It is related to, but often only loosely related to what is really wrong with your application

Because of this the System Completion Code represents the starting point for your analysis of the problem.

She won't be laughing She won't be laughing when she gets back to when she gets back to her desk and finds out her desk and finds out

that last night's that last night's production jobs blew production jobs blew

sky high!sky high!

9

Debugging AssistanceDebugging Assistance Along with the System Completion Code, using

IBM’s Problem Determination tools (PD Tools) Fault AnalyzerFault Analyzer you can obtain various reports which describe What/WhereWhat/Where and HowHow the ABEND occurred.

Valuable information contained in the Fault Analyzer report-files includes: The System Completion Code (and often a short text description of what it designates) A short explanation of the cause of the ABEND The COBOL instruction (statement) or line number, which contained the invalid operation

causing z/OS to halt execution Variables of interest – and code surrounding the instruction that halted execution A "core-dump" (a hexadecimal printout) of the internal machine storage and registers relevant to

the areas of your program surrounding the COBOL instruction which caused z/OS to halt execution.

This information is critical to begin understanding and researching the problem, but it is sometimes insufficient to solve the underlying application problem, which could be any combination of: Incomplete, incorrect or invalid COBOL procedural logic A typo such as a misplaced period, or incorrectly specified field Incorrect or invalid input data Batch jobs run out of sequence Input files missing or corrupted (hardware errors) Errors which relate to JCL problems etc.

10

Getting StartedGetting StartedThere are as many different ways to analyze and research COBOL ABENDs as there are

individual approaches to writing procedural logic. However, if you've never done software production support work, consider starting with the following structured problem-solving approach:

1. Preparation2. Research3. Hypothesis4. Solution5. Resolution

As a final note before we begin discussing the above, understand that there are often two distinct phases of application Production Support:

1. Data Center “on-call” ABEND resolution - wherein a technician receives notification that a job or transaction has ABEND’d and must be "fixed" within an extremely short timeframe (usually minutes to hours). In this case, the technician's main concern is to "patch" the problem - get the system back online, or get the batch job-stream back into production ("Patch-It").

2. NextDay problem resolution - wherein technician(s) actually track down and solve the problem that caused the ABEND ("Fix-It").

The steps that follow represent a structured approach to "FixIt" – in that they go well beyond the scope of the emergency measures used to "patch" the problem during an OnCall emergency.

11

1. Preparation and Information Gathering – 1 of 21. Preparation and Information Gathering – 1 of 2 Collect all necessary background information on the ABEND. WHAT

happened, WHERE the ABEND occurred and HOW the ABEND occurred:\

Start with Fault Analyzer's reports – as they contain a broad and deep set of formatted, structured, analysis information

Collect additional supporting ABEND output (SYSOUT) from the job, DISPLAY statements, etc.)

Obtain copies of the run-time:

JCL Program source All copybooks (or

expanded source listings)

12

1. Preparation and Information Gathering – 2 of 21. Preparation and Information Gathering – 2 of 2 If batch, from the JCL learn the dataset names of input and output files accessed by the program (which you may need to browse as part of your research)

Learn the nature of the batch job from system documentation, or from an application business expert (at least at the level of module-flow and file-access)

RAA Batch Job Diagram showing datasets

being passed from step to step

13

Research – Analyze the Gathered ABEND DataResearch – Analyze the Gathered ABEND Data From the preparation step, construct a mental map (understanding) of the program's

execution (HOW the ABEND occurred - which will start you down the path of research to understand "WHY" the ABEND occurred

WHY determination usually requires a combination of "Static" and "Dynamic" analysis - complementary research and investigative approaches.

These steps need notnot be followed in this order. Rather, in time you will develop an "intuition" as to which kind(s) of analysis will be most likely to provide the information you need to solve your problem.

To assist, you can use application research and analysis tools such as IBM’s Rational Asset AnalyzerRational Asset Analyzer, Rational Developer for System zRational Developer for System z – Static

Analysis IBM's Debug Tool Debug Tool or CICS I.A.CICS I.A. – Dynamic Analysis

So – if the reason "why" the ABEND occurred is not apparent at this point, perform Static and/or Dynamic Analysis on the specific areas of the application relating to the ABEND.

Note: Recompiling old COB2 programs with Enterprise COBOL can create many of these

problems, even though "Nothing has changed in the program"

14

Static Analysis – Overview of TechniquesStatic Analysis – Overview of Techniques1. Structural Visualization: is the generation of an accurate mental map, understanding or mental image of the

program's control structure, or logic-architecture. Using the starting point represented by the ABEND condition (the statement which caused z/OS to halt execution) and using electronic-assisted tools (such as IBM’s Rational Asset Analyzer or Rational Developer for System z), build an accurate understanding of the code invocation at:

The module/file level (System View) - Paragraph/Section level (Hierarchy chart) - Statement level (Flow chart)The module/file level (System View) - Paragraph/Section level (Hierarchy chart) - Statement level (Flow chart)Structural Visualization can done be "top-down", by asking open-ended questions; such as learning how a particular routine "hangs-together logically", or it can be used "bottom-up", by asking specific close-ended questions about a program, such as "How does this particular paragraph get executed?" "How did this module get invoked?"

2. Data Flow Analysis: A combination of control structure analysis and data item analysis, which seeks to determine the usage of particular fields throughout a program. Data flow analysis is used to determine (from a given instance of a data item) where the next occurrences of that item exist in your program, and how the data item is used; (as a receiving field in a MOVE or mathematical operation, as the sending field in a MOVE statement, as part of a logic-branch (IF, PERFORM UNTIL/VARYING, etc.).

3. Data Impact Analysis: An expansion of Data Flow Analysis which traces the movement of data from field-to-field throughout a program, or throughout an entire application; including I/O (screens and files). Using Data Impact Analysis, you can identify all fields that might have had an impact on the contents of a field (before the ABEND occurred). And just as importantly - you can learn the affect changing this field will have on the behavior of the application.

4. Textual or Data Item Usage: Utilized more for application maintenance and enhancement requests, this type of Static Analysis involves searching for "categories" of program-items, such as "List all fields that contain *JUL*, *GREG*, *YR*, *YEAR* (suspect date candidates for Year2000 conversion), or list all such fields with two digits (numeric) or two-byte (alphanumeric) definitions.

5. Code Partitioning: Again, utilized more for application maintenance, enhancements and application reengineering, Code Partitioning involves mentally organizing and analyzing code by function or process, such that you understand and can distinguish the usage of code by business process. For example: Find all code that relates to the calculation of premium renewal payments … or… Isolate the code that edits a particular file, with an eye towards creating a shared subroutine from the code.

15

Dynamic Analysis – Overview of TechniquesDynamic Analysis – Overview of Techniques1. Tracing: Source-level interactive debugging. Watch the program execute statement-by-

statement, and line-by-line. This is very useful for detailed-debugging, particularly of dense or complex instructions. Some software (for example, the Rational Developer for System z) allows you to trace the program logic, attempting to re-create the sequence of events (COBOL statements) that transpired up to and including the ABEND condition. Tracing is an invaluable method for detailed debugging. However, given the size and scope of production applications, it is generally more practical to trace specific problem areas of a program.

2. Interactive Execution: Execute (run) a program, stopping at selective Breakpoints (Pause execution each time a certain field-value changes, or when a value exceeds some threshold), and examining the contents (value) of specific fields. Interactive Execution must be done by (or with) an application analyst who understands how the system is supposed to operate. Interactive Execution is useful for observing control flow, and is often combined with line-by-line tracing by setting selective breakpoints, monitoring values, "running" the application to the breakpoints, and then tracing the code line-by-line.

3. Selective Data State Collection: Execute code and establish a functional summary of specific data states that it creates. Use these states in subsequent test runs to compare results of current values to expected values.

4. Coverage: Analyze the number of times each COBOL statement is executed for a given run. Note that PD Tools/Debug Tool can run a report that shows code coverage. This technique is extremely useful for analyzing test data coverage of a given application. And it can be used effectively for debugging if it makes apparent problems such as infinite loops (S222, S322 and B37 ABENDs), over-loading tables - (loading tables beyond the maximum OCCURS clause and overlaying storage, which cause things like: S0C1, S0C4, and S0C7 ABENDs).

16

Hypothesis - Determine WHY the ABEND occurred – 1 of 3 Your preparation and research is probably all you need to be able to

describe WHAT, WHERE and HOW the ABEND occurred At what point in the program the logic failed, and what sequence of

COBOL statements caused the failure).

However, before modifying any logic, you must determine WHYWHY these statements (or sequence of events) caused this particular failure "Why did this production input file contain spaces in a numeric field?" "Why did the program's logic perform the Initialization routine twice?" "Why did the Read routine execute past end-of-file?"etc.

Only through a determination of WHY will you be able to make a change to production business logic safely, and with confidence that; Your change will resolve the ABEND Your change will not introduce new (additional) ABENDs

17

Hypothesis - Determine WHY the ABEND occurred – 2 of 3 Sometimes it is relatively easy to come to an understanding of WHY certain ABEND conditions

occurred. For example, perhaps a period was left off the appropriate termination point for an IF statement - which caused execution to perform an operation out of sequence. Or perhaps an IF .. NUMERIC test (which should have been coded for all numeric fields in a file) was forgotten. Or a paragraph was performed through the wrong paragraph-exit, or a production job was released before certain files were available (causing I/O errors). These types of ABEND situations can be understood (and usually resolved) fairly quickly. But, not always. What if - in the case of the IF statement with the incorrect termination point - the logic that has been

coded, correctly processed the first 100,000 records in the file? Making a change to a critical IF condition could very well affect other down-stream processing within the

program, wrecking havoc with subsequent routines. Or what if - in the case of the file containing blanks in the numeric fields - the input file was supposed to

be "clean" (validated) by this point in the job-stream - having gone through allegedly "exhaustive" edits in prior modules.

By simply adding an IF test you may solve your program's specific ABEND, but you will not have resolved the actual problem - which exists somewhere else in the system.

In other words, localized/piecemeal approaches to resolving production ABENDs are not recommended - as they usually change the problem, instead of solving it. And sometimes they just spawn new problems.

It should be noted that, a clear understanding of the business functionalitybusiness functionality automated by this process is usually required to resolve WHY something has gone wrong. Calling on business experts or "application/business" experts

who understand "the big picture" - and the context in which the job executes is the rule rather than the exception to this

process.

18

Hypothesis - Determine WHY the ABEND occurred – 3 of 3 Developing a clear and accurate determination of WHY a problem that lead

to an ABEND condition exists may take a considerable amount of time, depending on the: Size, complexity and structure of the code Your familiarity with the program's business purpose - coupled with your ability to

grasp the point of each statement (assuming you didn't write the code) Type of ABEND and reason for the problem (some are more diabolical than

others) Size of the input/output files, and capabilities of your file editor

Note that, in addition to an understanding of the reason for the ABEND, the results of your investigation should produce an understanding of the solution to the problem (the fix itself).

19

Solution - Fix the Problem and Test Your SolutionTake the appropriate action to resolve any business - or system-wide

issues. Depending on how extensive the damage caused by the problem,

or for how long any problems have persisted undetected: Files may have to be restored from backups from a previous point-in-time Jobs may have to be re-run from a previous point-in-time (synchronized with file generations) Files may have to be modified with "one-shot" programs, written to resolve issues that require

"surgery" on the data

Take the appropriate action to fix the technical (coding) problem: Edit program source - modifying the existing production

logic …and/or… Modify the JCL (if the error included JCL issues) You may have to edit files using File Manager

Test your solution: Compile and Link the new version of the application Create an "image copy" of the production file system, in

order to test your fix Re-Run the batch job and analyze results Run "Regression Tests" against the new code – analyze for

unexpected results

20

Resolution Build in the test environment Migrate changes to back in to production Promote your changes to production Schedule and re-run the cycle

http://en.wikipedia.org/wiki/IBM_Rational_ClearCase

BatchBatchJobJob

21

Unit objectivesAfter having completed this unit on Production Support/Application Testing/Software

Defect and IBM Mainframe COBOL ABEND Research, you should now be able to:Define the steps in a generalized methodology of ABEND resolutionList the various sources of ABEND inputs, including:

PD Tools documents Fault Analyzer reports Other SYSOUT Static analysis Dynamic trace facilities

List the common types of COBOL program ABENDS

22

UNIT

Topics:

The RDz WorkbenchThe RDz Workbench

Analyzing Mainframe Abends ABEND Codes and Reasons ABEND Codes and Reasons Using IBM's Fault Analyzer Appendix

23

ABEND Completion Codes and some typical causes While there is a wide variety of reasons for ABEND conditions ("WHYWHYs") in

production systems, it is possible (and useful) to categorize and organize HOWHOW certain conditions often lead to certain types of ABEND completion codes - in order to expedite or streamline your analysis and research (an 80/20 approach to analysis).

The following information on a few common z/OS ABEND completion

codes, and the conditions which generated them is included for you to make effective use of PD Tools/Fault Analyzer listings and the above debugging, research and analysis process.

Notes:Notes:This information is available to some degree within the RDz product in

the Lookup ViewThere is another source of ABEND/Debug information you might want to

take a peek at: http://www.jsayles.com/ibm/cobol/ABEND.htm http://it.geocities.com/aminardi/computer/CODES.html

24

S001: Record Length/Block Size DiscrepancyReason(s)

S001-0: Conflict between record length specifications (program vs. JCL vs. dataset label)S001-2: Damaged storage media or hardware errorS001-3: Fatal QSAM errorS001-4: Conflict between Block specifications (program vs. JCL)S001-5: Attempt to read past end-of-file

Instructions:OPEN, CLOSE, READ, WRITEOPEN, CLOSE, READ, WRITE

Frequent Coding Causes:S001-0: Typos in FD or JCLS001-2: Corrupt disk or tape datasetS001-3: Internal z/OS problemS001-4: Forgot to code BLOCK CONTAINS 0 RECORDS in FDBLOCK CONTAINS 0 RECORDS in FD (default Block is 1)S001-5: Logic error (either forgot to close file, or end-of-file-switch not set, overwritten or ignored)

Tools to debug/RDz equivalent return codes:S001-0: Cannot occur on RDz with Local ASCII/Windows (Line Sequential) filesS001-2: Norton Utilities – if on Workstation/COBOL applicationS001-4: Cannot occur on Workstation/COBOL (no blocking for Line Sequential files)S001-5: Logic error: Use RDz's Perform Hierarchy or RAA's Program Flow Diagram to detect

Dynamic:S001-0: During Debug – set a Watch Monitor on the 01 recordS001-2: Need to have PC/IT technician investigate (may need to reformat disk)S001-4: Always code BLOCK CONTAINS 0BLOCK CONTAINS 0

25

S013: Conflicting DCB ParametersReason(s)

S013-10: Dummy data set needs buffer space; specify BLKSIZE in JCLS013-14: DD statement must specify a PDSS013-18: PDS member not foundS013-1C: I/O error search PDS directoryS013-20: Block size is not a multiple of the LRECLS013-34: LRECL is incorrectS013-50: Tried to open a printer for Input of I/OS013-60: Block size not equal to LRECL for unblocked fileS013-64: Attempted to Dummy out indexed or relative fileS013-68: Block size > 32KS013-A4: SYSIN or SYSOUT not QSAM fileS013-A8: Invalid RECFM for SYSIN/SYSOUTS013-D0: Attempted to define PDS with RECFM FBS or FSS013-E4: Attempted to concatenate > 16 PDSs

Instructions:OPEN, CLOSE, READ, WRITEOPEN, CLOSE, READ, WRITE

Frequent Coding Causes:Most of these ABENDs occur running und z/OS (some may not even occur under z/OS, although older modules running

OSVS or VS COBOL II code that have not been recompiled can produce them). Most are due JCL/COBOL FD inconsistencies.

Tools to debug:StaticS013-18: Open multiple windows on RAA Batch Job Diagram and program Environment Division - SELECT SELECT

ASSIGN. ASSIGN.

26

SOC1: Invalid InstructionReason(s)

- SYSOUT DD statement missing- The value in an AFTER ADVANCING clause is < 0 or > 99- And Index or Subscript is out of range- An I/O verb was issued against an unopened dataset

Instructions:OPEN, CLOSE, READ, WRITEOPEN, CLOSE, READ, WRITE, Table handling routines Note also that during Debug SYSOUT-DISPLAYs are written to the "console", recall how to view

Frequent Coding Causes:- Incorrect logic in setting AFTER ADVANCING variable (or failure to understand 0-99 limits)- Incorrect logic in table handling code, or number of table entries has outgrown PIC of variable (e.g. PIC 99, but 100 entries)

Tools to debug:Static

SYSOUT problem: Open multiple windows on RAA Batch Job Diagram and program Environment Division - SELECT ASSIGN. In RAA: Double-click on GO TO verb, or PERFORM chain, or paragraph name.In RDz: Select Paragraph name/Perform chain and select: Open Declaration

Dynamic:Set Watch Breakpoint and Monitor on table index or AFTER ADVANCING variable.Set conditional advanced break point on subscript (i.e. SUB<100).

27

S0C4: Protection ExceptionReason(s)

The program is attempting to access a memory address that is not within the applications z/OS Address Space

Frequent Coding Causes:- JCL DD statement is missing or incorrectly coded- Incorrect logic in table handling code (referencing a table subscript < 1 or > max-table-size), - Number of table entries has outgrown PIC of variable (i.e. PIC 99, but 100 entries).- In IMS/TM systems, an MFS LL (length) field value is smaller than the actual input MSG length.

Tools to debug:Static

- DD statement problem: Open multiple windows on RAA Batch Job Diagram and program Environment Division - SELECT ASSIGN - IMS LL problem: Analyze through multiple Edit Windows (same solution as DD).- Incorrect linkage problem:- Open multiple windows on CALLing and CALLed programs - verify linkage declarations.

DynamicIncorrect linkage problem:- Set Breakpoint and Monitor on linkage declarations.- Set conditional advanced break point on subscript (i.e. IDX < 100).RAA - Utilize Run-Unit View to view all modules in Debug run-stream (both active and inactive).Incorrect logicIn RDz/Debug - set a conditional break point on subscript (i.e. IDX < 100).

28

S0C7: Data ExceptionReason:

Machine instruction expecting numeric data found invalid dataInstructions:

Arithmetic, IF-THEN-ELSE, MOVE (if receiving field is numeric - )Note: RDz will also S0C7 if sending field is numeric and contains non-numeric (MOVE pic9field TO picXfield)

Frequent Coding Causes:- Incorrectly initialized, or uninitialized variable- Missing or incorrect data edit- 01 to 01 level MOVE if sending field is shorter than receiving field- Move of Zeros to Group-level numeric fields- MOVE CORRESPONDINGMOVE CORRESPONDING incorrect- or -- MOVE field1 to field2 incorrect assignment

Tools to debug:Static

RAA report with options data selector on MOD or ALLDynamic Set Watch points and Monitor on field. Run through to S0C7. Locate the field definition, or use CSI report.Solutions:

Add edit checks for all numeric fields and MOVE statements.

29

S0CB: Divide by zeroReason:

CPU attempted to divide a number by 0.Instructions:DIVIDE, COMPUTEDIVIDE, COMPUTE

Frequent Coding Causes:- Incorrectly initialized, or un-initialized variable- Missing or incorrect data edits (i.e. failed to check divisor for zero value)

Tools to debug:Static

RAA report on all DIVIDEDIVIDE and COMPUTECOMPUTE instructions – or using RDz double-click on these verbs and select Filter from the Context Menu

DynamicRun through to the S0CBLocate to field definitions of the offending fields

Solution:Add edit to check for zero divide:IF divisor > ZEROIF divisor > ZEROTHENTHENCOMPUTE ...COMPUTE ...ELSEELSEPERFORM error-processing routinePERFORM error-processing routine

Add ON SIZE ERRORON SIZE ERROR to all arithmetic verbs.

30

S222/S322: Timeout/Endless loopReason:

Timeout due to program logic caught in "loop" through instruction set with no exit.Note: RDz Workstation binaries (Run-Time System) does not honor the concept of timeout.

Frequent Coding Causes:- Invalid logic or fall-through logic- Invalid end-of-file logic- End-of-file switch overlaid- Subscript not large enough- Perform Thru wrong Exit- PERFORM UNTIL "End-Of-File", but not performing "READ" routine to reach EOF condition

Tools to debug: Static

RAA screens on logic in PERFORM chainRAA Display Perform Thru

DynamicPD Tools (mainframe) Debug to S222Analyze counts (color)Query and Monitor on subscriptSet an Advanced Break Point - Conditional on count

Solution:From within Debug, use RAAi to identify logic which could cause looping.Select and click on PERFORM THRU, PERFORM UNTIL, GO TO.

Place break points on potential error lines.

31

S806: Module Not FoundReason:

CALL made to program which could not be located along normal search path (STEPLIB top-to-bottom, JOBLIB top-to-bottom, LINKPACK)

Instructions:CALL, or JCL EXEC PGM=XXXX

Frequent Coding Causes:- Module deleted from library, or never compiled to library- Module name spelled incorrectly- STEPLIB does not contain load library with module- I/O error occurred while z/OS searched the directory of the library

Tools to debug:Static

Do RSE search on module name.Dynamic

Set Program Advanced Break Point (Entry) to set program break before entry to system.Solution:

Spell name correctly

32

B37/D37/E37 – dataset space exceededABENDS - B37/D37/E37 (RTS-028)

B37: Disk volume out of space.D37: Primary space exceeded, no secondary extents defined.E37: Primary and secondary extents full. In TSO, PDS directory needs compress.E37-04: Disk volume table of contents (VTOC) is full.

Reason:MVS could not find space for output WRITE.Note: RDz Local COBOL Run-Time System does not honor the concept of Primary/Secondary space "Out of space" condition will not occur until disk actually fills up

Instructions:WRITE

Frequent Coding Causes:- Not enough space initially allocated to output file(s).- (more likely) Logic error - program in (infinite) loop writing output file(s) - see S222/S322 reasons.

Tools to debug:Static - if (unlikely as this may be) you're debugging locally (on your Workstation) you can find the out-of-space file and statistics on it as follows:.

Do directory list on the file you are writing to:Go to DOS WindowType "DIR fname.ext"

On the host the JCL will show the DDNAME and z/OS filespec of the dataset in questionDynamic

Set an advanced conditional break point to break on a certain number on iterationsSee S222/S322 reasons and solutionsAlso, set break point on file WRITE statementsNote: If not logic error and actually have full disk, you may have to clean up your disk and erase files:

33

Database AbendsDB2 application access rarely abends, they "die gracefully" due to return code processing.

DB2:SQLCODE A unique integer which describes DB2's reaction to your request.

SQLCA A large 01 block, which contains several other fields pertinent to debugging,

particularly the SQLWARNs.

Suggestion:Set Line Breakpoint and/or Variable Monitor on SQLCODE and other key feedback areas

- or -Set Line Breakpoint and Watch Monitor for /"On-Change Break"Double-click on field, Ctrl/F3

Notes on UDB Offloading and Database Abends UDB handles SQLCODE processing in a manner consistent with DB2. However, it should be noted

that UDB does not handle SQLWARNO-n and the SQLCA (SQLERRD, etc.) fields compatibly. DB2 ABENDs

If DB2 tables have not been defined, you may get a -204 return code on your SQL statement. UDB may allow you to fix Database errors dynamically, without completely stopping your animator session:

34

UNIT

Topics:

The RDz WorkbenchThe RDz Workbench

Analyzing Mainframe Abends ABEND Codes and Reasons Using IBM's Fault AnalyzerUsing IBM's Fault Analyzer Appendix

35

Topic Considerations

Note:Note: In this topic you will learn how to use In this topic you will learn how to use Fault AnalyzerFault Analyzer to to debug an ABEND. The screen captures all describe connecting to a public z/OS debug an ABEND. The screen captures all describe connecting to a public z/OS machine that IBM makes available – during classes. machine that IBM makes available – during classes.

If you are taking this course through standard IBM services delivery you should If you are taking this course through standard IBM services delivery you should be able to use the properties (I/P address, port#s, etc.), logon IDs and passwords be able to use the properties (I/P address, port#s, etc.), logon IDs and passwords that your instructor provides you with.that your instructor provides you with.

But you may also be taking this course standalone – and in that case, you will But you may also be taking this course standalone – and in that case, you will need to speak to your company's Systems Programming staff to learn how to need to speak to your company's Systems Programming staff to learn how to connect and logon.connect and logon.

It goes without saying that the actual file names in the screen captures of It goes without saying that the actual file names in the screen captures of mainframe libraries and datasets will vary. So you should focus on the process mainframe libraries and datasets will vary. So you should focus on the process and steps and "how to" – and don't be perplexed at differences in screen and steps and "how to" – and don't be perplexed at differences in screen captures.captures.

You also may be using your company's own Source Control Management system You also may be using your company's own Source Control Management system – to do things like builds, compiles, etc. In that case much of the remote – to do things like builds, compiles, etc. In that case much of the remote functionality in RDz will be customized and tailored to your company's unique and functionality in RDz will be customized and tailored to your company's unique and idiosyncratic procedures and protocols.idiosyncratic procedures and protocols.

36

Unit objectivesAfter completing this unit, you should be able to:Work with ABEND analysis reports created by IBM Fault Analyzer Browse Report and Mini-Dump pagesRetrieve various Fault Analyzer view informationBrowse and search ABEND codesUse the various productivity features in the Fault Analyzer perspective

Note:Note:This presentation and Lab is not a comprehensive IBM Fault Analyzer unit. It is only intended to introduce you to the RDz/Fault Analyzer interface. It is assumed that you are already working with IBM Fault Analyzer on the z/OS host.

37

Overview and Incentive!Overview and Incentive!

Face facts:

ABEND research ("shooting a dumps") is not ABEND research ("shooting a dumps") is not how you want to spend your week-nightshow you want to spend your week-nights

The good news? You don't have to.

Fault Analyzer: Identifies the line where execution halted Shows the points-of-interest surrounding the ABEND:

Variables and variable values Statements Data and buffers

Gives you a serious head start on the What/Where and How of ABEND debugging

38

What is Fault Analyzer?What is Fault Analyzer? IBM Fault Analyzer is a tool that helps you determine the cause of an application

ABEND. It is used to determine: What happened, how it happened, what program, what line/statement,

which variableswhich variables, what files, were involved, etc. Fault Analyzer provides the necessary information to perform root cause analysis

on an application ABEND. You do not have to interpret low-level, system dumps and wade through HEX data.

Information is presented in report format

IBM Fault Analyzer for z/OS® gathers information about an application and the surrounding environment at the time of an abnormal end (ABEND), providing you with the valuable information you need to work through

After analyzing information about your application and its environment, Fault Analyzer generates an analysis report (IDIREPORTIDIREPORT) that describes the problem in terms of application/program statements and variables

39

Fault Analyzer for z/OS OverviewFault Analyzer for z/OS Overview Fault Analyzer is part of IBM’s Problem Determination Tool family of products:

http://www-01.ibm.com/software/awdtools/deployment/

It runs in both test and production with very little overhead, and: Provides support for analyzing***:

IMS and CICS® online application and system failures - with debugging facilities for all of the IBM-mainstream online files and databases

– IMS-DL/I, DB2, VSAM, IDMSIMS-DL/I, DB2, VSAM, IDMS, etc. WebSphere® Application Server for z/OS system failures WebSphere MQ application failures Batch (QSAM/VSAM/DB2®) application failures

Helps you analyze failures when they occur or reanalyze them after the fact Expands error messages and codes that apply to your failure with interactive reanalysis and includes a

feature for using application-specific messages and codes to supplement those supplied by IBM Creates a fault history file with an interactive display that helps you track and manage application failures Starts automatically when an application fails, eliminating the need to recompile programs or change the job

control language (JCL) Integrates with Rational Developer for System z enables developers to diagnose application problems

without changing user interface For Fault Analyzer product information, see: http://www-01.ibm.com/software/awdtools/faultanalyzer/

***See Notes***See Notes

40

Fault Analyzer – Operational ProcessFault Analyzer – Operational Process The purpose of Fault Analyzer (FA) is to determine the cause of any ABENDs in an

application program. You do not have to read through application or system dumps, because the product

has the ability to isolate the exact instruction that caused a particular error.The Analysis engine provides automatic analysis when the application fails. When an ABEND occurs, Fault Analyzer activates automatically, and then records details in

a fault history file (see screen capture below)Fault History files contain information about the faults analyzed by Fault Analyzer for z/OS.Using Fault History files, re-analysis is available when real-time ABEND analysis isn’t enough

(you can extract additional information in batch or interactive mode)

ABEND happens

Fault Analyzer exits are invoked

Salient details (points of interest) written and stored

41

Fault Analyzer – Batch Job OutputsFault Analyzer – Batch Job OutputsFault AnalyzerIs automatically invoked if your program ABENDs

An IDIREPORTIDIREPORT is produced that provides ABEND analysis information

42

Reviewing ABENDs in the Fault Analyzer PerspectiveReviewing ABENDs in the Fault Analyzer Perspective Besides the IDIREPORT, you may also wish

to use RDz's Fault Analyzer perspective, to analyze and debug an ABEND situation.

To do that you'll need to:1. Switch to the Fault Analyzer perspective in RDz2. Specify the history file to connect with, that populates a

Default ABEND view with failed online and batch job IDIREPORTs and other outputs

3. Learn how to navigate the Fault Analyzer perspective, to make use of the information contained therein

The next slides contain the step details

43

Fault Analyzer Perspective – 1 of 2Fault Analyzer Perspective – 1 of 2 Steps:

1.1. Open the Fault Analyzer PerspectiveOpen the Fault Analyzer Perspective

44

Fault Analyzer Perspective – 2 of 2Fault Analyzer Perspective – 2 of 22. Enter: FAULTANL.<version>.HIST

Ex. FAULTANL.V10R1.HISTFAULTANL.V10R1.HIST

45

Fault Analyzer Perspective – Overview Fault Analyzer Perspective – Overview

Fault AnalyzerExplorer

Content Area shows Fault Analyzer IDIREPORTIDIREPORT – detailed information on ABEND

IDIREPORTAvailable in Outline view

Can navigate using the Outline view

Additional Fault Analyzer views exist for other ABEND documentation including history data

46

Fault Analyzer – Default List of History FilesFault Analyzer – Default List of History FilesFrom the DefaultDefault tab

Scroll up and down – to find a particular ABEND Double-click an ABEND history file, to bring up its IDIREPORT and other stats Sort the list by any of the column headings

Can also work with options of the Context MenuContext Menu – with each ABEND entry

47

Fault Analyzer – Browse Report – S0CBFault Analyzer – Browse Report – S0CBThe IDIREPORT presents a formatted, high-level summary of the points of interest necessary to debug ABEND conditions in your application.

Specifically, to answer the questions:•What happened?

•What z/OS ABEND condition

•Where did it happen?•what line or statement was executing when it happened

•How did it happen?•What additional information is available for debugging purposes

Program line where the S0CB occurred

Click S0CB for an explanation of this ABEND

48

Fault Analyzer – Browse Report – S0C4Fault Analyzer – Browse Report – S0C4Here's an example of an IDIREPORT which shows that RPT-REC is "not addressable"

…which is a euphemism for: "There's something really wrong with the file/FD/JCL DD connection"

49

Fault Analyzer – Browse Report – S0C7Fault Analyzer – Browse Report – S0C7The IDIREPORT and supporting text varies from ABEND to ABEND depending on:

• Type of ABEND

• Information available at the time of the ABEND

• Run-time platform

Note: Note: CUST-ACCT-BALANCE's value is shown in hex (as it represents invalid data)

50

Fault Analyzer – Browse Report – S0C9Fault Analyzer – Browse Report – S0C9A S0C9 is like a S0CB (divide by zero) except that a S0C9 occurs because of an excessively large fixed-point number obtained as the result of a decimal division operation

51

Fault Analyzer – Browse Report – S0C1Fault Analyzer – Browse Report – S0C1Here is information on an IMS (DL/I) S0C1 ABEND

52

Fault Analyzer – Browse Report – S806Fault Analyzer – Browse Report – S806IDIDREPORT information on a module-not-found (S806) ABEND

Most likely SAM2 is either a typo, or the program did not successfully compile/link into the Load Module

53

Fault Analyzer – Lookup View for MVS and File Return CodesFault Analyzer – Lookup View for MVS and File Return Codes

The Lookup view shows a great deal of background information on:• ABEND codes• DB2 SQLCODE• IMS PCB Feedback• VSAM File Status etc.

You can use the view, or double-click on the ABEND code shown in the IDIREPORT

54

Fault Analyzer Integration – Mini-Dump Reading 1 of 2Fault Analyzer Integration – Mini-Dump Reading 1 of 2Fault Analyzer also provides for the reading/browsing of System Dump data – in Hex/Character format.

Select an ABEND

Scroll through the dump – Issue navigation commands: Show nnn, +nn, etc.Show nnn, +nn, etc.

55

Fault Analyzer Integration – Mini-Dump Reading 2 of 2Fault Analyzer Integration – Mini-Dump Reading 2 of 2

You can assign analysis notes to the dump.

1. Right-click over the storage address

2. Add your note (click OK)

3. Your note becomes highlighted text inside the dump

56

Fault Analyzer – Organize ABEND History Views

1

23

Define views which contain a set of history files.

Display fault entries from all history files as if they were in one dataset.

57

Fault Analyzer – Fault History ViewFault Analyzer – Fault History View

The Default view shows either:• A History File• A view of a History file

58

Fault Analyzer – The TSO/ISPF Interface

Available Line

Commands

A list of History ABENDs

Batch and/or Online may be in the same list

59

Checkpoint

1. What RDz Perspective is used to view Fault Analyzer reports?

2. How does RDz obtain Fault Analyzer Information? Where does the information originate?

3. RDz Fault Analyzer interface has a Lookup View. What is it used for?

4. How can you jump to the program statement where the ABEND occurred with the RDz Fault Analyzer interface?

60

SummaryHaving completed this unit, you should now be able to:Work with ABEND analysis reports created by IBM Fault Analyzer Browse Report and Mini-Dump pagesRetrieve various Fault Analyzer view informationBrowse and search ABEND codesUse the various productivity features in the Fault Analyzer perspective

61

Workshop (for Sandbox Users) – Big PictureWorkshop (for Sandbox Users) – Big PictureIn this workshop you are going to:In this workshop you are going to:1. Copy a several datasets from your instructor's zServerOS TSO ID to your ID

Details on the next slide

2. Modify JCL dataset names (and high-level qualifiers) to match your Sandbox ID

3. Compile a program named: HOSPCALC – which contains different types of COBOL ABENDs generated from invalid COBOL logic in different parts of the program

4. Run HOSPCALC (it will ABEND) – and from the Fault Analyzer IDIREPORT: Find the error in the COBOL source, and use the IDIREPORT ABEND analysis data to fix the error After you've solved the problem, you will save your edits, and re-compile HOSPCALC. Then run the

program until you either get the next ABEND … or get a zero return code

HOSPCALCHOSPCALCLoad ModuleLoad Module

Analyze IDIREPORTAnalyze IDIREPORT

Fix HOSPCALC COBOL error

CompileCompileLink EditLink Edit

62

Workshop – Setup Steps – 1 of 2If you are working on the Sandbox, do the following:1. In Remote Systems, create a new MVS Filter named: RDZFARDZFA

for: DDS0001.POT.*DDS0001.POT.*2. Expand the RDZA filter, and:

Copy DDS0001.POT.COBOL member: HOSPCALC to <<EM4Zxx>>.POT.COBOL PDS Copy DDS0001.POT.JCL member: HOSPCALC to <<EM4Zxx>>.POT.JCL PDS

This is the compile JCL for HOSPCALC

Copy DDS0001.POT.JCL member: HOSPCRUN to <<EM4Zxx>>.POT.JCL PDS This is the run JCL, for HOSPCALC

3. Right-click over DDS0001.POT.DATA and use Allocate LikeAllocate Like to create a new PDS named: <<yourEM4ZxxyourEM4Zxx>>.POT.DATA.POT.DATA - with the same dataset characteristics

Example with ID: EM4Z07EM4Z07

63

Workshop – Steps – 2 of 24. Expand the EM4zxx.POT.JCL EM4zxx.POT.JCL library::

Edit HOSPCALC. Replace all occurrences of: DDS0001 with <<yourEM4ZxxyourEM4Zxx>> Save your editsSave your edits

Edit HOSPCRUN. Replace all occurrences of: DDS0001.POT with <<yourEM4ZxxyourEM4Zxx>>.POT.POT Save your editsSave your edits

5. Submit your modified5. Submit your modified HOSPCALC HOSPCALC JCL. When the job finishes, check the results in your When the job finishes, check the results in your JES My Jobs JES My Jobs filter (it should compile successfully – but check that the LKED step ran)

6. Submit your modified 6. Submit your modified HOSPCRUN HOSPCRUN JCL. The job shouldThe job should ABEND. ABEND. In JES - My Jobs JES - My Jobs, view the IDIREPRTIDIREPRT step, to see the specific system completion code and surrounding Fault Analyzer ABEND analysis information

Note that HOSPCALC contains four separate ABEND errors – that you will hit, one at a time: an 0C7, an 0CB, an 0C4 and a combined 0C7/0C4 ABEND (Sounds strange ... Essentially Fault Analyzer will find two different problems on one statement)

The comments and descriptions in the IDIREPRT combined with your COBOL knowledge should be sufficient to debug and fix these problems, but feel free to elicit help from your instructor if you're stuck.

7. Fix the COBOL error that caused the ABEND. Then repeat steps 5 and 6 until you find and fix all of the ABENDs in HOSPCALC. You're done when HOSPCRUN finishes with a zero return code.