When Hell Breaks Loose

© Isaac Yassin – IBM Information Champion, IBM Certified Solution Expert , IBM Certified Data Base Administrator DB2 V7/8/9

When Hell Breaks LoosePart 2

Data-Sharing failure Anatomy

Session B16DB2 for z/OSThursday, Oct 08, 200909:45 - 10:45

Isaac Yassin – IYC, DBMS & IT Consulting1

1

2


Isaac Yassin

When Hell Breaks Loose

Data-Sharing failure Anatomy

2

© Isaac Yassin –IBM certified Solution Expert -3-


The presentation material, code examples and timing measurements are given as-is, without any warranty of any kind, either expressed or implied. The author shall not be liable for any loss or damage caused directly or indirectly by using the examples given.

Isaac Yassin is an IBM Information Champion whose experience in computer systems dates back to 1976. Isaac works mainly as a consultant to many organizations, both in Israel and Europe, covering wide aspects of performance tuning, troubleshooting, system design and education. Isaac has been working closely with DB2 since it's debut and followed up all versions and is an IBM certified solution expert – DB2 V7 and an IBM Certified Database Administrator for V8 & 9. Isaac is a well known lecturer and conducts many courses dealing with computer science (mainly in the DBMS area). Isaac holds a bachelor degree in Economics and computer Science from BAR-ILAN University and holds several other titles from MAMRAM computer center at the IDF.Isaac serves as Israel RUG co-leader and also served on IDUG EUCPC 2005-2008

IBM and DB2 are trademarks of IBM corporation.Copyright © 2009 – Isaac Yassin – DBMS & IT Consulting Services

3

© Isaac Yassin – IBM Information Champion, IBM Certified Solution Expert , IBM Certified Data Base Administrator DB2 V7/8/9 4


Global RESET (1)

• RESET all PGLOGRBA to zero • DSN1COPY

• That’s few TB of data• Data and Indexes • Need time and DASD

• Time is an issue, and• We can have the needed amount of DASD

only if we use the mirror DASD

5


Global RESET (2a)• Use PK22872:

• ADJUST DATA SHARING ENABLING RBA VALUES WHEN TRUNCATING THE LOGS TO AN EARLIER RBA VALUE

• The MAX RBA FOR TORBA value in the BCR2 records must be adjusted when doing a conditional restart of the data sharing enabling member that truncates the logs to an RBA value that is less than the MAX RBA FOR TORBA in the BCR2 records.

• Maintenance is way behind• Will take too long to implement

• We cannot use the other member• Need to build a new member

6


Global RESET (2b)LOG MAP OF BSDS DATA SET COPY 2, DSN=XXXX.YYYY.BSDS02

LTIME INDICATES LOCAL TIME, ALL OTHER TIMES ARE GMT. DATA SHARING MODE IS ON SYSTEM TIMESTAMP - DATE=2009.216 LTIME= 3:22:42.23 UTILITY TIMESTAMP - DATE=2009.056 LTIME=17:38:12.19 VSAM CATALOG NAME=DB2D HIGHEST RBA WRITTEN 007E9F98188F 2009.216 00:22:43.2 HIGHEST RBA OFFLOADED 007E9EBCBFFF RBA WHEN CONVERTED TO V4 006CCE82D585 MAX RBA FOR TORBA 006CCE82D585 MIN RBA FOR TORBA 000000000000 STCK TO LRSN DELTA 000000000000

THIS BSDS HAS MEMBER RECORDS FOR THE FOLLOWING MEMBERS: HOST MEMBER NAME: YYYY

MEMBER ID: 1 GROUP NAME: DSNDB2D BSDS COPY 1 DATA SET NAME: XXXX.YYYY.BSDS01 BSDS COPY 2 DATA SET NAME: XXXX.YYYY.BSDS02 ENFM START RBA/LRSN: 000000000000

MEMBER NAME: ZZZZ MEMBER ID: 2 GROUP NAME: DSNDB2D BSDS COPY 1 DATA SET NAME: XXXX.ZZZZ.BSDS01 BSDS COPY 2 DATA SET NAME: XXXX.ZZZZ.BSDS02

7


Global RESET (3a)• Use PK28576 :

• When SPRMRRBA is set, the COPY Utility will reset the RBAs to zeroes in the DB2 data and in the output image copy for non data sharing DB2 subsystems.

• The COPY requested must be a full copy (FULL YES is the default). Either SHRLEVEL CHANGE or SHRLEVEL REFERENCE may be specified.

• To enable the special processing by the COPY Utility to reset the RBAs in the DB2 data, proceed as follows:

• Edit member DSN6SPRC of the prefix.SDSNMACS library and locate the entry for SPRMRRBA.

• Change the setting for SPRMRRBA to '1' and save your changes.

• Run the first two steps of your customized copy of job DSNTIJUZ to rebuild your DB2 system parameter (DSNZPxxx) module.

• SPRMRRBA is not online changeable so after rebuilding the DB2 system parameter module, you need to stop and start DB2 in order for the change to become effective.

• When the special processing is completed, repeat the above steps but change the SPRMRRBA setting to '0' instead of '1'.

8


Global RESET (3b)

• Use PK28576 : • Serviceability enhancement to the COPY Utility to

reset the RBAs to zeroes in the DB2 data for non data sharing DB2 subsystems.

• New DSNZPARM: SPRMRRBA

• We are in Data Sharing mode • 1 member only

• Maintenance is way behind• Will take too long to implement

9


DBA 3rd rule

10


Agenda

• To err is humane• But to really mess things up –

you need a computer ☺• Divide and Conquer?• Expect the un-expected• Lessons learned• Technical notes

• For those who are really interested

11


Establish a base line• We have wrong current BSDS

• We cannot update the “STCK to LRSN delta”information in the BSDS

• We have older BSDS (from archive log) and older Catalog and Directory (from 2 / 3 days ago)

• We have everything mirrored on remote DASD

• Global RESET is not feasible

12


Preferred Solution• Disable Data Sharing

• Reset the STARTRBA in BSDS to a value higher then any value generated by -

LRSN + STCK to LRSN delta

• COLDSTART DB2 with that value

• Return to Data Sharing later

13



Disable data-sharing• Disabling DB2 data sharing is a complex procedure and is very disruptive. • Do not attempt to disable DB2 data sharing without a thorough understanding

of the process.• Do not make disabling DB2 data sharing part of your contingency plans for

handling recovery situations.

• For temporary bypass to data sharing problems, try moving to one-way data sharing, which involves stopping all but one member and having that member perform the work for the group.

• If one-way data sharing does not work, your data sharing problem might be resolved by performing a group restart.

• Situations in which it might be necessary to disable DB2 data sharing are:

• You made a strategic decision to move away from a data sharing environment. • One-way data sharing is not working.

15


Disable data-sharing (1)• Decide which member will be the surviving member of the group.

• Stop all members by entering the following command for each member of the data sharing group:

• STOP DB2 MODE(QUIESCE)

• Start the surviving member in maintenance mode by using the following command:

• START DB2 ACCESS(MAINT)

16


Disable data-sharing (2)• Make sure that data is consistent Enter the following commands from the surviving

member of the group. Do not go on to the next step until all problems are resolved.

• DISPLAY GROUP

• Make sure that the status of the surviving member is ACTIVE and that the status of all other members is QUIESCED.

• If a non-surviving member’s status is not QUIESCED, take the following actions, depending on the member’s status:

• If the member’s status is FAILED, restart and then stop the member and stop it successful.• If the member has castout problems, indoubt units of recovery, or outstanding resynchronization problems, start

that member in maintenance mode and fix the problem.

• DISPLAY UTILITY(*)• If there is any remaining utility work for any member of the group, restart that member with ACCESS(MAINT)

and either stop the utility or let it finish.

• DISPLAY DATABASE(*) SPACENAM(*) RESTRICT• If there are any restricted table spaces or index spaces (such as write error ranges, recovery pending status, or

logical page list entries), recover them from the surviving member.

17


Disable data-sharing (3)• Stop the surviving member by using the following command:

• STOP DB2 MODE(QUIESCE)

• Stop any IRLMs that have not stopped by using the following command:

• STOP irlmproc

• If IRLM is not stopping you can use:

• F irlmproc,abend,nodump

18


Disable data-sharing (4)• Dismantle the data sharing group.

• Enter the following command to display the structures for the data sharing group:

• D XCF,STRUCTURE,STRNAME=grpname*

• For all structures that are still allocated (STATUS:ALLOCATED) and that still have connections (which appear as FAILED PERSISTENT), enter the following command to force the connections off of those structures:

• SETXCF FORCE,CONNECTION,STRNAME=strname,CONNAME=ALL

• If your site is running z/OS with APAR OA02620 applied, you do not need to force failed-persistent connections off of the lock structure.

• Delete all the DB2 coupling facility structures by using the following command for each structure:

• SETXCF FORCE,STRUCTURE,STRNAME=strname

19


Disable data-sharing (5)• Edit the JCL in job DSNTIJGF to point to the correct BSDS data sets.• DSNTIJGF is a change log inventory job that sets up the surviving member• for a cold start.

• Important: Do not change the hex values that appear in the change log inventory CRESTART control statement. They are not real RBA values.

• X'C4C9E2C1C2D3'

• Run job DSNTIJGF.• After you run this job, do not try to restart any of the non-surviving members. • None of those members can start successfully.

20


Disable data-sharing (6)• Change the IRLM procedure to SCOPE=LOCAL.• Start the surviving member with ACCESS(MAINT). Specify the old DSNZPxxx from the

non-data sharing environment. • If the surviving member is not the originating member, you must reassemble the surviving

member’s subsystem parameters, specifying the subsystem parameter DSHARE=NO in the invocation of the DSN6GRP macro. Also, comment out all steps from the DSNTIJUZ job except for those that reassemble and link-edit the subsystem parameters.

• When you start DB2 after running DSNTIJGF, respond with Y to a cold start prompt (message DSNJ246I on the z/OS console).

• This is a cold start because DB2 increases the log RBA to a value higher than any LRSN used while sharing data. From this point on, your RBAs look like LRSNs.

• Problem:• To get Out of Data Sharing you need to use X'C4C9E2C1C2D3‘

for cold-start • We needed X‘D40000000000’ according to the labs

• Another cold-start to establish that one

21


Disable data-sharing (7)• Edit and run job DSNTIJFT, if necessary, to ensure that the surviving member’s work file

database is DSNDB07.

• The surviving member must use DSNDB07 as its work file database. If the work file database for the surviving member is not DSNDB07, drop that work file database and run job DSNTIJFT.

• Verify that the surviving member works by running a subset of the installation verification sample jobs.

• To establish a new recovery point, take a full or incremental image copy or non-DB2 backup of all data. Run job DSNTIJIC to image copy the DB2 catalog and directory.

22



Insurance policy

• Build a totally new DB2 subsystem• Copy all DDL

• Resides in a PDS, updated daily• Install 3rd party tools to extract data from image

copy and load it to the new DB2 • Runstats & Bind (from DBRMLIB PDS)

• Split DASD and have two groups work in parallel

24


Schedule

• Schedule an update meeting every 2 hours• Keep it short

• Keep the two recovery teams in touch• Transfer knowledge

• Keep IBM labs in touch around the clock• They have a task force there

• Arrange shifts• Tired people make more mistakes

25


DB2 Catalog & Directory

• Restored from older Image Copy• Was taken with SHRLEVEL REFERENCE

• Used DSN1COPY + RESET • Indexes were re-built

• This time, correctly ☺

• DB2 started without any problem• Non Data Sharing• Cold start

• RBA > LRSN (X’D40000000000’)

26


Agenda




27


On-Line Reorg (fastswitch yes)• We use DB2 Catalog from 2 days ago• We use current user data• There were many OLR in between• The file names on DASD are different from

those in DB2 Catalog • I / J differences

• Need to update all the differences• REXX to locate the differences and to

generate the needed update commands• Can be found at IDUG site – CODE PLACE: FINDIJ

28


Identity columns

• Remember it’s V7 – behind on maintenance• Identity values in user data are higher then

those in DB2 Catalog• We need to update them manually• Open the DB2 Catalog to update

• Hidden Zparm (with IBM labs approval)

• Update the identity values via SQL• Close the DB2 Catalog for update

29


History Statistics

• After restoring the DB2 to working mode we got the known “00C200C1”on the *_HIST tables.• From utility runs

• Collection of HISTSTAT was closed• Zparm value

30


More 00C200C1• Batch jobs started to fail with 00C200C1 on

user data

• At first, we did a DSN1COPY + RESET on the failing tablespace / indexspace

• After about 50 of those we decided to do a COLDSTART with higher RBA

• X’D80000000000’

• That solved the problem ☺

31


Agenda




32


DBA lessons• Never go “out of support”• If you open a PMR – follow it closely• Don’t lag behind on maintenance• Think before you act• Backup before any test

• And, never, never – overwrite backups

• Have a set of JOBs for eventualities

33


DBA lessons• Stay current with education• Don’t play around hoping the problem will

disappear by itself• DBA job is not a one man show

• It’s a team work

• Have all books on hand• Internet access is not always feasible

• Have a phone list on hand• Swallow your pride and ask for help before

the situation goes out of control

34


DBA lessonsGive me five minutes more,Only five minute more,Let me stay,Let me stay In your armsFIVE MINUTES MORE From the film "Sweethearts Of Sigma Chi" (1946) (Sammy Cahn / Jule

Styne) Frank Sinatra - 1946 (Re-Recorded 1961) Tex Beneke & The Glenn Miller Orch. - 1946 The Three Suns - 1946 Skitch Henderson & His Orch.(vocal: Ray Kellogg) - 1946 Bob Crosby & His Orch. - 1946 Frankie Carle & His Orch. - 1946 Carroll Gibbons & His Orpheans - 1946 Dick Haymes - 1946 Robin Luke - 1958 Chico Hamilton - 1958 Herb Alpert & The Tijuana Brass - 1966 Sammy Cahn - 1972 Also recorded by: Tony Burgos Orch; Bing Crosby; Carlo & The Belmonts; Tony Devlin; The Emeralds; Lionel Ferbos; Gary Nichols; Five By Design

35


DBA 4th rule

36

Confucius


Management lessons

• Killing the messenger is NOT a good idea

37


Management lessons

• Don’t be cheap on education• Penny smart, Pound foolish

• Don’t be cheap on internet connection• Fast line costs, but uploading a 1GB dump is not for slow line

• Follow closely but don’t intervene every five minutes• Especially, don’t stand behind the DBA’s back and continue

to ask questions while (s)he’s working

• Arrange food for night shifts

38


Management lessons

• Ask for all details before reaching a decision• Be ready to change decisions as new facts

come in.• Be there for any request• Arrange shifts• Swallow your pride and ask for help before

the situation goes out of control

39


Red Herring section ☺

• A red herring is an idiom referring to a device which intends to divert the audience from the truth or an item of significance.

• During the crisis we had many of them

• Sends you in tangent to real problem

40


Red Herring section ☺

• Old sysplex timer• Caused numerous IPLs and physical connection checks

• DB2 did start up, it just did not work• Do you know that DB2 will start up without SYSCOPY?

• CF – Coupling Facility• Switching from one CF to the other and back,

does not cause you to loose data

• GRS – Global Resource Serialization• Ring vs. Star – has no relation to the DB2 problem

41


Agenda




42


BSDS N.D.S

DATA SHARING MODE IS OFFUTILITY TIMESTAMP - DATE=2009.001 LTIME=19:35:05.64VSAM CATALOG NAME=DB2Z HIGHEST RBA WRITTEN 000000000000 0000.000 00:00:00.0 HIGHEST RBA OFFLOADED 000000000000 RBA WHEN CONVERTED TO V4 000000000000

THIS BSDS HAS MEMBER RECORDS FOR THE FOLLOWING MEMBERS: HOST MEMBER NAME: DB2Z MEMBER ID: 0 GROUP NAME: DSNDB2Z

43


BSDS D.SDATA SHARING MODE IS ONSYSTEM TIMESTAMP - DATE=2009.0001 LTIME=19:36:25.28 VSAM CATALOG NAME=DB2Z HIGHEST RBA WRITTEN 000000000000 0000.000 00:00:00.0 HIGHEST RBA OFFLOADED 000000000000 RBA WHEN CONVERTED TO V4 000000000000 MAX RBA FOR TORBA 000000000000 MIN RBA FOR TORBA 000000000000 STCK TO LRSN DELTA 000000000000

THIS BSDS HAS MEMBER RECORDS FOR THE FOLLOWING MEMBERS: HOST MEMBER NAME: DBZA MEMBER ID: 2 GROUP NAME: DSNDB2Z BSDS COPY 1 DATA SET NAME: DB2Z.DSNDB2Z.DBZA.BSDS01 BSDS COPY 2 DATA SET NAME: DB2Z.DSNDB2Z.DBZA.BSDS02 MEMBER NAME: DBZB MEMBER ID: 1 GROUP NAME: DSNDB2Z BSDS COPY 1 DATA SET NAME: DB2Z.DSNDB2Z.DBZB.BSDS01 BSDS COPY 2 DATA SET NAME: DB2Z.DSNDB2Z.DBZB.BSDS02

44


CRCR N.D.SCONDITIONAL RESTART CONTROL RECORD

11:38:41 NOVEMBER 18, 2008 **** ACTIVE CRCR RECORD ****

NO CRCR RECORDS ARE ACTIVE **** CRCR IDENTIFIER 0002

USE COUNT 1 RECORD STATUS

CRCR NOT ACTIVE SUCCESSFUL RESTART

PROCESSING STATUS COLD START (STARTRBA = ENDRBA) FORWARD = NO BACKOUT = NO

STARTRBA 00019DFD5000 ENDRBA 00019DFD5000 ENDLRSN NOT SPECIFIED EARLIEST REQUESTED RBA 000000000000 FIRST LOG RECORD RBA 000000000000 ORIGINAL CHECKPOINT RBA 000000000000 NEW CHECKPOINT RBA (CHKPTRBA) NOT SPECIFIED CRCR CREATED 08:08:42 MARCH 08, 2005 BEGIN RESTART 08:09:05 MARCH 08, 2005 END RESTART 08:09:35 MARCH 08, 2005 RESTART PROGRESS STARTED ENDED

======= ===== CURRENT STATUS REBUILD YES YES FORWARD RECOVERY PHASE YES YES BACKOUT RECOVERY PHASE YES YES

45


SCAD XCF,STR,STRNAME=DSNDB2Z_SCA IXC360I 14.09.41 DISPLAY XCF 990 STRNAME: DSNDB2Z_SCA STATUS: ALLOCATED EVENT MANAGEMENT: POLICY-BASED TYPE: LIST POLICY INFORMATION: POLICY SIZE : 32000 K POLICY INITSIZE: 16000 K POLICY MINSIZE : 16000 K FULLTHRESHOLD : 80 ALLOWAUTOALT : YES REBUILD PERCENT: 1 DUPLEX : DISABLED ALLOWREALLOCATE: YES PREFERENCE LIST: CFDEV2 ENFORCEORDER : NO

EXCLUSION LIST IS EMPTY

46

ACTIVE STRUCTURE ----------------ALLOCATION TIME: 25/07/2006 16:30:43 CFNAME : CFDEV2 COUPLING FACILITY: 002096.IBM.83.0000000XXXXXPARTITION: 01 CPCID: 00

ACTUAL SIZE : 16896 K STORAGE INCREMENT SIZE: 512 K USAGE INFO TOTAL CHANGED % ENTRIES: 18234 129 0 ELEMENTS: 36305 464 1 PHYSICAL VERSION: XXXXXXXX XXXXXXXXLOGICAL VERSION: XXXXXXXX XXXXXXXXSYSTEM-MANAGED PROCESS LEVEL: 8 DISPOSITION : KEEP ACCESS TIME : 0 MAX CONNECTIONS: 32 # CONNECTIONS : 2

CONNECTION NAME ID VERSION SYSNAME JOBNAME ASID STATE ---------------- -- -------- -------- -------- ---- ----------------DB2_DBZA 01 00010022 SYSA DBZAMSTR 0080 ACTIVE DB2_DBZB 02 00020046 SYSB DBZBMSTR 0063 ACTIVE DIAGNOSTIC INFORMATION: STRNUM: 00000007 STRSEQ: 00000000

MANAGER SYSTEM ID: 00000000 EVENT MANAGEMENT: POLICY-BASED


DBA 5th rule

47

Pete Seeger


Reading material• DB2 V9.1 for z/OS Data Sharing: Planning and

administration SC18-9845-02

• The same for V8

• SG242079 - IBM z/OS Parallel Sysplex Operational Scenarios

48

49


Isaac Yassin

[email protected]

49

Session: B16Title: When Hell Breaks Loose,

Anatomy of a Data Sharing Failure

When Hell Breaks Loose

Documents