Transcript
Using SDA Extensions to Troubleshoot System Hangs and Performance Issues
Presented by
Jim Mehlhopmehlhop@parsec.com
www.parsec.com | 888-4-PARSEC
To Download
http://www.parsec.com/public/SDA_Extensions.pdf
Activating SDA
Symbols
Read Global Symbols from System Loadable Images
Read Global Symbols from Any Object Module
Topics
Format Instructions and Blocks of Data
Copy the System Dump File
Send Output to a File or Device
SDA Extensions What are they
How they work
They are undocumented, getting help can be a challenge
Topics (contd)
Topics (contd)
Clue Example
Example of using SDA PCS to troubleshoot a hung system
Examples using SDA SPL to troubleshoot high MP SYNCHRONIZATION time
Example using FLT to isolate source of alignment faults, a major source of performance issues on Integrity Servers.
Additional extensions
Activating SDA
There are two ways of getting into SDA. One is to examine the live system and the other is to analyze a crash dump file. In both cases it reads a basic symbol table: REQSYSDEF.STB
To analyze a live system, issue the following: $ analyze/system
Need the CMKRNL privilege Since it is a live system, things change REMEMBER that
To analyze a crash dump, issue the following: $ analyze/crash dump-filespec
Need read access to the dump file Everything is static
Activating SDA (contd)
Either way you start SDA, it always looks for an initialization file.
First it looks for a logical SDA$INIT
Second it looks in the current users default login directory for a file named sda.init
The purpose for this file is to execute SDA commands that you ALWAYS want to execute.
An example of this would be commands to read in symbols. We will be talking about symbols shortly.
Reading Symbols
Symbols are read both automatically and under user control. Automatically
When SDA gets activated, it reads reqsysdef.stb from the appropriate directory for the current architecture.
User control SDA> read/exec
Reads symbols from system loadable image list SDA> read
Default file type is .stb, however it reads .obj files as well VMS supplies .stb files for most loadable_images and subsystems
$DIR SYS$LOADABLE_IMAGES:*.stb (Alpha) Subsystem STBs DCLDEF, F11BXQPDEF, IMGDEF,
IODEF, RMSDEF, SCSDEF, and a system-wide SYSDEF
Reading Symbols (contd)
Users can supplement the VMS symbols
Create a Macro source file with needed definitions
$ edit symbols_example.mar
Compile that source
$ mac symbols_example.mar
Reading Symbols Examples #3Reading User-Created Object File
There is an offset in a drivers Function Decision Table that indicates which functions support 64 bit addressing
named FDT$Q_OK64BIT by default VMS does not make that symbol available to you. So we create a macro
source program that pulls the FDT symbols in from the Macro library.
CLASS3$ type symbols_example.mar
.LIBRARY /SYS$LIBRARY:LIB.MLB/
.LIBRARY /SYS$LIBRARY:STARLET.MLB/
$FDTDEF GLOBAL
$PFNDEF GLOBAL ;PFN DATA RECORD(Hopefully)
$PFREEDEF GLOBAL ;FREE POOL PACKET DEFINITIONS
$PRDEF GLOBAL ;PROCESSOR REGISTER DEFINITIONS
$PSIGDEF GLOBAL ;CALL SIGNATURE DEFINITIONS
$PTEDEF GLOBAL ; Page Table Entry DEFINITIONS
$PTRDEF GLOBAL ; V7.3 PAGEFILE POINTER STRUCURE
.END
CLASS3$ mac symbols_example.mar
SDA> sho sym FDT$Q_OK64BIT
%SDA-E-BADSYM, unknown symbol "FDT$Q_OK64BIT"
SDA> read SYMBOLS_EXAMPLE.obj
%SDA-I-READSYM, 1394 symbols read from STAFF_DISK:[MEHLHOP]SYMBOLS_EXAMPLE.OBJ;1
SDA> sho sym FDT$Q_OK64BIT
FDT$Q_OK64BIT = 00000000.00000108
SDA Commands Format
The format command allows you to display a block of memory that is defined as a data structure.
It relies on the convention that the byte at offset 10(decimal) 0A(hex) defines which data structure the block represents.
If there are symbols that represent that data structure, then SDA will display those names and the data in a formatted output as displayed in the next slide.
If the block does not follow the convention or does not have a value that SDA understands, then SDA will either not format it or format it as the wrong type of structure.
SDA Commands Format (contd)
SDA> sho proc ! To define PCB symbol
.
.
.
SDA> ex pcb+0A
%SDA-W-UNALIGNED, unaligned address FFFFFFFF.81E8200A; converting to aligned address
PCB+00008: 000C0580 "...."
SDA> !0c=PCB
SDA> format pcb
%SDA-E-NOSYMBOLS, no "PCB" symbols found to format this block
SDA> read sysdef
SDA> format pcb
FFFFFFFF.81E82000 PCB$L_SQFL 818099E8 SCH$AQ_COMT+001A4
FFFFFFFF.81E82004 PCB$L_SQBL 818099E8 SCH$AQ_COMT+001A4
FFFFFFFF.81E82008 PCB$W_SIZE 0580
FFFFFFFF.81E8200A PCB$B_TYPE 0C
FFFFFFFF.81E8200B
00
.
.
.
SDA Commands Format (contd)
SDA> format phd !This structure does NOT follow convention note type=0
%SDA-E-INVBLKTYP, invalid block type in specified block
SDA> form phd/type=phd
FFFFFFFF.82866000 PHD$Q_PRIVMSK 00000040.D05DC205
FFFFFFFF.82866008 PHD$W_SIZE 0000
FFFFFFFF.8286600A PHD$B_TYPE 00
FFFFFFFF.8286600B 00
FFFFFFFF.8286600C PHD$L_WSLIST 00000001
FFFFFFFF.82866010 PHD$L_WSLOCK 0000000A
FFFFFFFF.82866014 PHD$L_WSDYN 00000012
FFFFFFFF.82866018 PHD$L_WSNEXT 00000263
FFFFFFFF.8286601C PHD$L_WSLAST 000002E2
FFFFFFFF.82866020 PHD$L_WSEXTENT 00000400
FFFFFFFF.82866024 PHD$L_WSQUOTA 00000120
.
.
.
SDA Commands Misc.
SDA> show executive
SDA> map 810B8050
SDA> show cluster
SDA> show lan
SDA> copy dev::[dir]filename
SDA> set output dev::[dir]filename
SDA> set log dev::[dir]filename
Extensions What are they
Special purpose code that calls SDA callable routines to examine memory under the protection of SDA.
They are searched for whenever you issue an SDA command that is not a native SDA command.
Searches for a logical name in the format of whatever the command was plus $SDA. For example SDA> PCS show trace will look for the logical PCS$SDA. If that logical does not exist it will look in the SYS$LIBRARY directory for the image PCS$SDA.
Extensions What are they (cont)
Anyone can create an SDA extension. The first extension was CLUE. The second one was written by an ISV to support troubleshooting their application.
The extensions supplied by OpenVMS can be identified by issuing the following command $DIRECTORY SYS$LIBRARY:*SDA.EXE
Extensions What are they (cont)
Extensions are version/patch/Architecture specific
Alpha
SYS6$ show system/noprocess
OpenVMS V8.3 on node SYS6 12-SEP-2011 16:10:23.94 Uptime 0 00:31:38
SYS6$ dir sys$share:*$sda.exe
Directory SYS$COMMON:[SYSLIB]
CLUE$SDA.EXE;1 CNX$SDA.EXE;1 DECDTM$SDA.EXE;1 DKLOG$SDA.EXE;1
EXC$SDA.EXE;1 FC$SDA.EXE;1 FLT$SDA.EXE;1 IO$SDA.EXE;1
IPC$SDA.EXE;1 ISCSI$SDA.EXE;1 LAN$SDA.EXE;1 LCK$SDA.EXE;1
LNM$SDA.EXE;1 MTX$SDA.EXE;1 OCLA$SDA.EXE;1 PCS$SDA.EXE;1
PE$SDA.EXE;1 PKM$SDA.EXE;1 PRF$SDA.EXE;1 PSH$SDA.EXE;1
PTHREAD$SDA.EXE;1 PWIP$SDA.EXE;1 RMS$SDA.EXE;1 SHAD$SDA.EXE;1
SPL$SDA.EXE;1 TCPIP$SDA.EXE;1 TQE$SDA.EXE;1 TR$SDA.EXE;1
USB$SDA.EXE;1 XFC$SDA.EXE;1 XX$SDA.EXE;1
Total of 31 files.
SYS6$
Integrity
SYS226$ show system/noprocess
OpenVMS V8.3-1H1 on node SYS226 12-SEP-2011 16:15:05.95 Uptime 49 04:31:46
SYS226$ dir sys$share:*$sda.exe
Directory SYS$COMMON:[SYSLIB]
CLUE$SDA.EXE;1 CNX$SDA.EXE;1 DECDTM$SDA.EXE;1 DKLOG$SDA.EXE;1
EXC$SDA.EXE;1 FC$SDA.EXE;1 FLT$SDA.EXE;1 IO$SDA.EXE;1
IPC$SDA.EXE;1 ISCSI$SDA.EXE;1 LAN$SDA.EXE;1 LCK$SDA.EXE;1
LNM$SDA.EXE;1 MTX$SDA.EXE;1 PCS$SDA.EXE;1 PE$SDA.EXE;1
PKM$SDA.EXE;1 PKR$SDA.EXE;1 PRF$SDA.EXE;1 PSH$SDA.EXE;1
PTHREAD$SDA.EXE;1 PWIP$SDA.EXE;1 RMS$SDA.EXE;1 SHAD$SDA.EXE;1
SPL$SDA.EXE;1 TCPIP$SDA.EXE;1 TQE$SDA.EXE;1 TR$SDA.EXE;1
USB$SDA.EXE;1 XFC$SDA.EXE;1 XX$SDA.EXE;1
Total of 31 files.
SYS226$
Extensions What are they (cont)
IntegrityCLASS3$ show system/noprocess
OpenVMS V8.4 on node CLASS3 12-SEP-2011 15:58:48.80 Uptime 5 23:26:10
CLASS3$ dir sys$share:*$sda.exe
Directory SYS$COMMON:[SYSLIB]
CLUE$SDA.EXE;1 CNX$SDA.EXE;1 DECDTM$SDA.EXE;1 DKLOG$SDA.EXE;1
EXC$SDA.EXE;1 FC$SDA.EXE;1 FLT$SDA.EXE;1 GSP$SDA.EXE;1
ICC$SDA.EXE;1 IO$SDA.EXE;1 IPC$SDA.EXE;1 ISCSI$SDA.EXE;1
LAN$SDA.EXE;1 LCK$SDA.EXE;1 LES$SDA.EXE;1 LNM$SDA.EXE;1
MTX$SDA.EXE;1 NET$SDA.EXE;1 PCS$SDA.EXE;1 PE$SDA.EXE;1
PKD$SDA.EXE;1 PKM$SDA.EXE;1 PKR$SDA.EXE;1 PRF$SDA.EXE;1
PSH$SDA.EXE;1 PTHREAD$SDA.EXE;1 PWIP$SDA.EXE;1 RMS$SDA.EXE;1
SHAD$SDA.EXE;1 SPL$SDA.EXE;1 SWIS$SDA.EXE;1 TCPIP$SDA.EXE;1
TQE$SDA.EXE;1 TR$SDA.EXE;1 USB$SDA.EXE;1 XFC$SDA.EXE;1
XX$SDA.EXE;1
Total of 37 files.
Extensions What are they (cont)
How they work
When you use an extension command SDA loads the extension into YOUR process address space.
They may then collect for display information that is contained in the OS
Many of them also include an image in the SYS$LOADABLE_IMAGES directory that gets loaded into system space (GH) with a load subcommand to the extension. This image will stay in memory until either unloaded or the system reboots
How they work(cont)
For those extensions that collect information that is not normally collected by the OS the extension will also allocate a ring buffer in nonpaged pool.
This allows for you to start collection, log off, and later examine the collection after an event (including a system crash).
How they work(cont)
Process address space System address space
SDA.EXE
Shareable images
xxx$SDA
xxx$DEBUG
(system loadable image)
Ring Buffer
How they work(cont)CLASS8$ an/sy
OpenVMS system analyzer
SDA> pcs load
PCS$DEBUG load status = 00000001
SDA> sho exec pcs$debug
VMS Executive layout
--------------------
Image Base End Length Image Offset SymVec
--------------------------------------- ----------------- ----------------- ----------------- ----------------- --------
PCS$DEBUG
Code FFFFFFFF.818E6000 FFFFFFFF.818E76EF 00000000.000016F0 00000000.00010000
Data (read only) FFFFFFFF.886FD200 FFFFFFFF.886FD367 00000000.00000168 00000000.00020000
Data (read/write) FFFFFFFF.886FD400 FFFFFFFF.886FD40F 00000000.00000010 00000000.00030000
Short data (read/write) FFFFFFFF.886FD600 FFFFFFFF.886FD84F 00000000.00000250 00000000.00060000
Linked 29-JUN-2006 23:20 LDRIMG = 88FA7A40 SeqNum = 000000B6 GP = FFFFFFFF.888FD600
SDA> pcs unload
PCS$DEBUG unload status = 00000001
SDA> sho exec pcs$debug
No loadable image matching "PCS$DEBUG" found
SDA>
Getting help
Some have limited help that you get by specifying just the command and no qualifiers (subcommand)
Some have built in help
Some have a help library
Getting help (cont)
SDA> flt
Alignment Fault Tracing Utility FLT commands:
FLT LOAD
FLT UNLOAD
FLT START TRACE [/BUFFER=pages]
[/BEGIN=pc_range_low]
[/END=pc_range_high]
[/MODE=(K,E,S,U)] (default is ALL modes)
[/INDEX=pid] (default is ALL processes)
FLT STOP TRACE
FLT SHOW TRACE [/SUMMARY]
SDA> pe help
PE$SDA Extension on CLASS2 (AlphaServer 4100 5/400 4MB) crash at 24-APR-2007 15:15:58.06
----------------------------------------------------------------------------------------
PE
This command executes the PEDRIVER specific SDA extension image
which is a collection of PEDRIVER specific commands which mirror
the display functions of SYS$SYSTEM:SCACP.EXE in SDA.
By default this image is loaded from SYS$SHARE. However, it can
be redirected with the following logical:
$ define pe$sda [dir-spec]pe$sda.exe
Format:
SDA> PE command
Additional information available:
Press RETURN for more.
SDA>
PE$SDA Extension on CLASS2 (AlphaServer 4100 5/400 4MB) crash at 24-APR-2007 15:15:58.06
----------------------------------------------------------------------------------------
BUS CHANNEL LAN_DEVICE TRACE HELP VC
PE Subtopic? vc
Getting help(cont)
PE Subtopic? vc
PE
VC
Displays PEDRIVER virtual circuit data. Each VC is NISCACP
communications path between the local system and a remote
system comprised of a set of channels.
Format
PE VC
Additional information available:
Additional information available:
Press RETURN for more.
SDA>
PE$SDA Extension on CLASS2 (AlphaServer 4100 5/400 4MB) crash at 24-APR-2007 15:15:58.06
----------------------------------------------------------------------------------------
Parameters Qualifiers
/ALL /COUNTERS /SUMMARY /VC /n
Example
PE VC Subtopic? exam
PE
VC
Example
SDA> PE VC/ERRORS/VC=FOO
Displays VC error counters for all VCs whose name (remote node name)
starts with FOO.
PE VC Subtopic?
PE Subtopic?
Topic?
Getting help(cont)
SDA> pwip
SDA:PWIP> help
HELP
DEBUG [ON]
DEBUG OFF
DEBUG BREAKPOINT
PWIP HEADER
PWIP DATAGRAMS [ BRIEF | FULL | RAW | NUM ]
PWIP LISTENERS [ BRIEF | FULL | RAW | NUM ]
PWIP SESSIONS [ BRIEF | FULL | RAW | NUM ]
PWIP [ALL] ( BRIEF of Datagrams, Listeners & Sessions)
PWIP TRACE
PWIP DEFAULT [ BRIEF | FULL | RAW ] (When NUM specified)
SDA:PWIP>
Getting help(cont)
SDA> spawn /nowait dir sys$help:*sda.hlb
SDA>
Directory SYS$COMMON:[SYSHLP]
SDA.HLB;1 TCPIP$SDA.HLB;1 XFC$SDA.HLB;1
Total of 3 files.
SDA> tcpip help
TCPIP
Sorry, no documentation on TCPIP
Additional information available:
About_TCPIP$SDA Development_Qualifiers READ SEARCH
SHOW SYSCONFIG TAG FORMAT
Topic? devel
Sorry, no documentation on DEVEL
Additional information available:
TCPIP
Topic? tcpip
Getting help(cont)
Topic? tcpip
TCPIP
The TCPIP verb prefixes all of the TCPIP (TCP/IP Services for
The TCPIP verb prefixes all of the TCPIP (TCP/IP Services for
Press RETURN for more.
SDA>
OpenVMS) SDA extension's commands.
Format:
TCPIP [qualifiers]
TCPIP FORMAT ...
TCPIP READ ...
TCPIP SEARCH ...
TCPIP SHOW ...
TCPIP SYSCONFIG ...
TCPIP TAG ...
Additional information about the TCPIP extension can be found
under the `About_TCPIP$SDA' topic.
Additional information available:
About_TCPIP$SDA Development_Qualifiers READ SEARCH
SHOW SYSCONFIG TAG FORMAT
Getting help(cont)
SDA> tcpip help format
TCPIP
FORMAT
Formats the block of memory specified by the "address" parameter.
To do this the "MALLOC_HDR", located at negative 40 hex from the
address is interpreted to determine the type of the block, and
in some cases the subtype of the block.
Format:
TCPIP FORMAT address
/VERIFY
Additional information available:
Parameters
Press RETURN for more.
SDA>
Getting help(cont)
Clue Crash
SDA> clue crash
Crashdump Summary Information:
------------------------------
Crash Time: 16-MAY-2006 08:36:28.28
Bugcheck Type: SSRVEXCEPT, Unexpected system service exception
Node: SYS6 (Cluster)
CPU Type: Digital Personal WorkStation
VMS Version: V8.2
Current Process: MEHLHOP
Current Image: $6$DKC200:[MEHLHOP.BOOTCAMP.SOURCE]CRASH.EXE;1
Failing PC: 00000000.00030194 LAN$C_MCA_L_LAT+00061
Failing PS: 10000000.00000003
Module: CRASH
Offset: 00030194
Boot Time: 10-MAY-2006 21:25:06.00
System Uptime: 5 11:11:22.28
Crash/Primary CPU: 00/00
System/CPU Type: 1E05
Saved Processes: 0
Pagesize: 8 KByte (8192 bytes)
Physical Memory: 640 MByte (81920 PFNs, contiguous memory)
Press RETURN for more.
Clue Crash (cont)
SDA>
Crashdump Summary Information:
------------------------------
Dumpfile Pagelets: 1310720 blocks
Dump Flags: olddump,writecomp,errlogcomp
Dump Type: raw,full,shared_mem
EXE$GL_FLAGS: poolpging,init,bugdump
Paging Files: 1 Pagefile and 1 Swapfile installed
Stack Pointers:
KSP = 00000000.7FF87BE8 ESP = 00000000.7FF8C000 SSP = 00000000.7FF9CC80
USP = 00000000.7AE35AB0
General Registers:
R0 = 00000000.0000000C R1 = 00000000.7FF87E38 R2 = FFFFFFFF.81C38CC0
R3 = FFFFFFFF.818CA460 R4 = 00000000.7FF87CC0 R5 = 00000000.7FF87E38
R6 = 00000000.7FF87E80 R7 = 00000000.7FF87FC0 R8 = 00000000.7FF9CDE8
R9 = 00000000.7FF9DDF0 R10 = 00000000.7FFA4F28 R11 = 00000000.7FFCDBE8
R12 = 00000000.7FFCDA68 R13 = 00000000.00010080 R14 = 00000000.00000000
R15 = 00000000.7AF1C5E0 R16 = 00000000.000003C4 R17 = 00000000.7FF87CC0
R18 = 00000000.00000020 R19 = 00000000.00000002 R20 = 00000000.00000000
R21 = 00000000.00000000 R22 = 00000000.7FFF03C8 R23 = 00000000.7FFF0040
Press RETURN for more.
Clue Crash (cont)
SDA>
Crashdump Summary Information:
------------------------------
R24 = 00000000.00000000 AI = 00000000.00000003 RA = FFFFFFFF.81810200
PV = FFFFFFFF.818C9260 R28 = 00000000.000005AC FP = 00000000.7FF87BF0
PC = FFFFFFFF.8009B43C PS = 28000000.00000000
Exception Frame:
R2 = 00000000.00000003 R3 = FFFFFFFF.818CA460 R4 = FFFFFFFF.82027880
R5 = 00000000.00000088 R6 = 00000000.00030050 R7 = 00000000.7FF87FC0
PC = 00000000.00030194 PS = 10000000.00000003
Signal Array: 64-bit Signal Array:
Arg Count = 00000005 Arg Count = 00000005
Condition = 0000000C Condition = 00000000.0000000C
Argument #2 = 00000000 Argument #2 = 00000000.00000000
Argument #3 = 00000000 Argument #3 = 00000000.00000000
Argument #4 = 00030194 Argument #4 = 00000000.00030194
Argument #5 = 00000003 Argument #5 = 10000000.00000003
Mechanism Array:
Arguments = 0000002C Establisher FP = 00000000.7AE35AB0
Press RETURN for more.
Clue Crash (cont)
SDA>
Crashdump Summary Information:
------------------------------
Flags = 00000001 Exception FP = 00000000.7FF87E80
Depth = FFFFFFFD Signal Array = 00000000.7FF87E38
Handler Data = 00000000.00000000 Signal64 Array = 00000000.7FF87E50
R0 = 00000000.00000000 R1 = 00000000.00000000 R16 = 00000000.00000000
R17 = 00000000.00010040 R18 = 00000000.00000005 R19 = 00000000.00000000
R20 = 00000000.00000000 R21 = FFFFFFFF.FFFFFFFF R22 = 00000000.00010040
R23 = FFFFFFFF.8016663C R24 = 00000000.00000001 R25 = 00000000.00000001
R26 = 00000000.00030178 R27 = 00000000.00010090 R28 = 00000000.00000006
System Registers:
Page Table Base Register (PTBR) 00000000.0000BA35
Processor Base Register (PRBR) FFFFFFFF.81C38000
Privileged Context Block Base (PCBB) 00000000.22B16080
System Control Block Base (SCBB) 00000000.00000A61
Software Interrupt Summary Register (SISR) 00000000.00000000
Address Space Number (ASN) 00000000.00000034
AST Summary / AST Enable (ASTSR_ASTEN) 00000000.0000000F
Floating-Point Enable (FEN) 00000000.00000001
Interrupt Priority Level (IPL) 00000000.00000000
Machine Check Error Summary (MCES) 00000000.00000000
Virtual Page Table Base Register (VPTB) FFFFFEFC.00000000
Press RETURN for more.
Clue Crash (cont)
SDA>
Crashdump Summary Information:
------------------------------
Failing Instruction:
LAN$C_MCA_L_LAT+00061: LDL R0,(R0)
Instruction Stream (last 20 instructions):
LAN$C_MCA_L_LAT+00011: LDQ R7,#X0028(SP)
LAN$C_MCA_L_LAT+00015: LDQ R13,#X0030(SP)
LAN$C_MCA_L_LAT+00019: LDQ FP,#X0038(SP)
LAN$C_MCA_L_LAT+0001D: ADDQ SP,#X40,SP
LAN$C_MCA_L_LAT+00021: RET R31,(R28)
LAN$C_MCA_L_LAT+00025: BIS R31,R31,R31
LAN$C_MCA_L_LAT+00029: BIS R31,R31,R31
LAN$C_MCA_L_LAT+0002D: SUBQ SP,#X10,SP
LAN$C_MCA_L_LAT+00031: STQ R26,(SP)
LAN$C_MCA_L_LAT+00035: STQ R13,#X0008(SP)
LAN$C_MCA_L_LAT+00039: BIS R31,R27,R13
LAN$C_MCA_L_LAT+0003D: ADDQ R13,#X10,R27
LAN$C_MCA_L_LAT+00041: BSR R26,#X000006
LAN$C_MCA_L_LAT+00045: LDQ R28,(SP)
LAN$C_MCA_L_LAT+00049: LDQ R13,#X0008(SP)
Press RETURN for more.
Clue Crash (cont)
SDA>
Crashdump Summary Information:
------------------------------
LAN$C_MCA_L_LAT+0004D: ADDQ SP,#X10,SP
LAN$C_MCA_L_LAT+00051: RET R31,(R28)
LAN$C_MCA_L_LAT+00055: BIS R31,R31,R31
LAN$C_MCA_L_LAT+00059: BIS R31,R31,R31
LAN$C_MCA_L_LAT+0005D: BIS R31,R31,R0
LAN$C_MCA_L_LAT+00061: LDL R0,(R0)
LAN$C_MCA_L_LAT+00065: RET R31,(R26)
LAN$C_MCA_L_LAT+00069: HALT
LAN$C_MCA_L_LAT+0006D: HALT
LAN$C_MCA_L_LAT+00071: HALT
Clue Stack
Clue Stack attempts to decode and annotate the stack. It sometimes has problems
with null stack frames and in fact in this version got confused with the CHF structure
as well.
SDA> clue stack
Stack Decoder:
--------------
Normal Process Kernel Stack:
Stack Pointer 00000000.7FF87BE8
Stack Limits (low) 00000000.7FF84000
(high) 00000000.7FF88000
Press RETURN for more.
Clue Stack (cont)
SDA>
SSRVEXCEPT Stack:
-----------------
Stack Pointer SP => 00000000.7FF87BE8
Information saved by Bugcheck:
a(Signal Array) 00000000.7FF87BE8 00000000.7FF87E38
EXE$EXCPTN[E] Stack Frame:
PV 00000000.7FF87BF0 FFFFFFFF.818C9260 EXE$EXCPTN
Entry Point FFFFFFFF.8009B380 EXE$EXCPTN_C
00000000.7FF87BF8 00000000.00000000
return PC 00000000.7FF87C00 FFFFFFFF.8009A93C SYS$CALL_HANDL_C+0002C
saved R2 00000000.7FF87C08 00000000.00000000
saved R4 00000000.7FF87C10 FFFFFFFF.82027880 KTB
saved R5 00000000.7FF87C18 00000000.00000088
saved R6 00000000.7FF87C20 00000000.00030050 SYS$K_VERSION_03+00050
saved FP 00000000.7FF87C28 00000000.7FF87C40
SYS$CALL_HANDL Temporary Storage:
00000000.7FF87C30 FFFFFFFF.818C9028 SYS$CALL_HANDL
00000000.7FF87C38 00000000.00000000
Press RETURN for more.
Clue Stack (cont)
SDA>
SSRVEXCEPT Stack:
-----------------
SYS$CALL_HANDL Stack Frame:
PV 00000000.7FF87C40 FFFFFFFF.818C9028 SYS$CALL_HANDL
Entry Point FFFFFFFF.8009A910 SYS$CALL_HANDL_C
00000000.7FF87C48 00000000.00000000
return PC 00000000.7FF87C50 FFFFFFFF.80098370 CHF_BADSTACK_C+00060
saved FP 00000000.7FF87C58 00000000.7FF87EE0
Fixed Exception Context Area:
Linkage Pointer 00000000.7FF87C60 FFFFFFFF.818C8240 CHF$GR_LINKAGE_SECTION
a(Signal Array) 00000000.7FF87C68 00000000.7FF87E38
a(Mechanism Array) 00000000.7FF87C70 00000000.7FF87CC0
a(Exception Frame) 00000000.7FF87C78 00000000.7FF87E80
Exception FP 00000000.7FF87C80 00000000.7FF87EE0
Unwind SP 00000000.7FF87C88 00000000.00000000
Reinvokable FP 00000000.7FF87C90 00000000.00000000
Unwind Target 00000000.7FF87C98 00000000.00000008
#Sig Args/Byte Cnt 00000000.7FF87CA0 82027880.81EEA8C0
a(Msg)/Final Status 00000000.7FF87CA8 8200E640.81DE2400
Flags/Arguments 00000000.7FF87CB0 00000005.00000260 BUG$_NETTRANCNT
Press RETURN for more.
Clue Stack (cont)
SDA>
SSRVEXCEPT Stack:
-----------------
a(Establisher FP) 00000000.7FF87CB8 818BCF64.000008F8 BUG$_SEQ_NUM_OVF
Mechanism Array:
Flags/Arguments 00000000.7FF87CC0 00000001.0000002C
a(Establisher FP) 00000000.7FF87CC8 00000000.7AE35AB0
reserved/Depth 00000000.7FF87CD0 FFFFFFFF.FFFFFFFD
a(Handler Data) 00000000.7FF87CD8 00000000.00000000
a(Exception Frame) 00000000.7FF87CE0 00000000.7FF87E80
a(Signal Array) 00000000.7FF87CE8 00000000.7FF87E38
saved R0 00000000.7FF87CF0 00000000.00000000
saved R1 00000000.7FF87CF8 00000000.00000000
saved R16 00000000.7FF87D00 00000000.00000000
saved R17 00000000.7FF87D08 00000000.00010040 SYS$K_VERSION_16
saved R18 00000000.7FF87D10 00000000.00000005
saved R19 00000000.7FF87D18 00000000.00000000
saved R20 00000000.7FF87D20 00000000.00000000
saved R21 00000000.7FF87D28 FFFFFFFF.FFFFFFFF
saved R22 00000000.7FF87D30 00000000.00010040 SYS$K_VERSION_16
saved R23 00000000.7FF87D38 FFFFFFFF.8016663C AMAC$EMUL_CALL_NATIVE_C+0005C
Press RETURN for more.
Clue Stack (cont)
SDA>
SSRVEXCEPT Stack:
-----------------
saved R24 00000000.7FF87D40 00000000.00000001
saved R25 00000000.7FF87D48 00000000.00000001
saved R26 00000000.7FF87D50 00000000.00030178 LAN$C_MCA_L_LAT+00045
saved R27 00000000.7FF87D58 00000000.00010090 LAN$C_CCA_W_BROADCAST+00009
saved R28 00000000.7FF87D60 00000000.00000006
saved F0 00000000.7FF87D68 08000000.00000000
saved F1 00000000.7FF87D70 00000000.00000000
saved F10 00000000.7FF87D78 00000000.00000000
saved F11 00000000.7FF87D80 00000000.00000000
saved F12 00000000.7FF87D88 00000000.00000000
saved F13 00000000.7FF87D90 00000000.00000000
saved F14 00000000.7FF87D98 00000000.00000000
saved F15 00000000.7FF87DA0 00000000.00000000
saved F16 00000000.7FF87DA8 00000000.00000000
saved F17 00000000.7FF87DB0 00000000.00000000
saved F18 00000000.7FF87DB8 00000000.00000000
saved F19 00000000.7FF87DC0 00000000.00000000
saved F20 00000000.7FF87DC8 00000000.00000000
saved F21 00000000.7FF87DD0 00000000.00000000
Press RETURN for more.
Clue Stack (cont)
SDA>
SSRVEXCEPT Stack:
-----------------
saved F22 00000000.7FF87DD8 00000000.00000000
saved F23 00000000.7FF87DE0 00000000.00000000
saved F24 00000000.7FF87DE8 00000000.00000000
saved F25 00000000.7FF87DF0 00000000.00000000
saved F26 00000000.7FF87DF8 00000000.00000000
saved F27 00000000.7FF87E00 00000000.00000000
saved F28 00000000.7FF87E08 00000000.00000000
saved F29 00000000.7FF87E10 00000000.00000000
saved F30 00000000.7FF87E18 00000000.00000000
a(Signal64 Array) 00000000.7FF87E20 00000000.7FF87E50
SP Align = 10(hex) [...............]
Signal Array:
Arguments 00000000.7FF87E38 00000005
Condition 00000000.7FF87E3C 0000000C
Argument #2 00000000.7FF87E40 00000000
Argument #3 00000000.7FF87E44 00000000
Argument #4 00000000.7FF87E48 00030194 LAN$C_MCA_L_LAT+00061
Argument #5 00000000.7FF87E4C 00000003
Press RETURN for more.
Clue Stack (cont)
SDA>
SSRVEXCEPT Stack:
-----------------
64-bit Signal Array:
Arguments 00000000.7FF87E50 00002604.00000005
Condition 00000000.7FF87E58 00000000.0000000C
Argument #2 00000000.7FF87E60 00000000.00000000
Argument #3 00000000.7FF87E68 00000000.00000000
Argument #4 00000000.7FF87E70 00000000.00030194 LAN$C_MCA_L_LAT+00061
Argument #5 00000000.7FF87E78 10000000.00000003
Interrupt/Exception Frame:
saved R2 00000000.7FF87E80 00000000.00000003
saved R3 00000000.7FF87E88 FFFFFFFF.818CA460 EXE$CMODEXECX
saved R4 00000000.7FF87E90 FFFFFFFF.82027880 KTB
saved R5 00000000.7FF87E98 00000000.00000088
saved R6 00000000.7FF87EA0 00000000.00030050 SYS$K_VERSION_03+00050
saved R7 00000000.7FF87EA8 00000000.7FF87FC0
saved PC 00000000.7FF87EB0 00000000.00030194 LAN$C_MCA_L_LAT+00061
saved PS 00000000.7FF87EB8 10000000.00000003 IPL INT CURR PREV
SP Align = 10(hex) [...............] 00 0 Kern User
Press RETURN for more.
.
.
.
Troubleshooting a Hung System
This process works in the case where you have a system that is periodically hanging to the point where you cannot get to a $ prompt.
In this case it was an artificially created case where the system was hanging at IPL 8
Note: When this occurs on an Integrity Server you do NOT see the ^P menu on the console.
Prior to the hang you MUST use the PCS extension to load and start a trace
The ring buffer will continue to be written until the system is crashed
Troubleshooting a Hung System Integrity
CLASS8$ run new
(system hangs ^P followed by ^P)
Crash (y/n)> y
**** OpenVMS I64 Operating System V8.3 - BUGCHECK ****
** Bugcheck code = 0000064D: OPERCRASH, Operator forced system crash
** Crash CPU: 00000000 Primary CPU: 00000000 Node Name: CLASS8
** Supported CPU count: 00000002
** Active CPUs: 00000000.00000003
** Current Process: MEHLHOP
** Current PSB ID: 00000001
** Image Name:
**** Starting compressed selective memory dump at 14-MAY-2007 17:22...
................................................................................
.............................................
** System space, key processes, and key global pages have been dumped.
** Now dumping remaining processes and global pages...
................................
...Complete ****
**** Primary HALTED with code HWRPB_HALT$K_WARM_REBOOT
***********************************************************
Troubleshooting a Hung System Integrity (cont)
CLASS8$ an/cr SYSDUMP_HANG_IPL8
OpenVMS system dump analyzer
...analyzing an I64 compressed selective memory dump...
Dump taken on 14-MAY-2007 17:27:14.60 using version V8.3
OPERCRASH, Operator forced system crash
SDA> sho exception
Exception Frame Summary
-----------------------
Exception Frame Type Stack IIP / Ret_Addr Trap_Type / Service_Number
----------------- ---- ----- ----------------- --------------------------
00000000.7FF42D80 ORIGINAL_INTSTK Kernel FFFFFFFF.810B5B70 00000041 Bugcheck Breakpoint Trap
00000000.7FF430A0 INTSTK Kernel FFFFFFFF.80145950 00000080 External Interrupt
00000000.7FF430F0 INTSTK Kernel FFFFFFFF.80442A20 00000028 Report alignment fault
00000000.7FF43150 INTSTK Kernel FFFFFFFF.7E34A000 00000028 Report alignment fault
00000000.7FF433B0 INTSTK Kernel 000002D0.02000000 00000060 Interval Clock Interrupt
00000000.7FF433C0 INTSTK Kernel FFFFFFFF.805100E0 00000060 Interval Clock Interrupt
00000000.7FF436E0 STALE_SSENTRY Kernel FFFFFFFF.883AA600 0100019C SYS$PERSONA_EXPORT_ARB
00000000.7FF43700 SSENTRY Kernel FFFFFFFF.80B3D690 01000022 SYS$DALLOC
00000000.7FF438F0 SSENTRY Kernel FFFFFFFF.80B396B0 010000F1 SYS$KRNDWN
00000000.7FF43990 SSENTRY Kernel FFFFFFFF.805A4890 0100008A SYS$RMSRUNDWN
00000000.7FF43B40 SSENTRY Kernel FFFFFFFF.80B5C3D0 01000028 SYS$DELPRC
00000000.7FF43D30 SSENTRY Kernel FFFFFFFF.80B5AC40 0100018E SYS$EXIT_INT
00000000.7FF43F40 SSENTRY Kernel 00000000.00020080 01000019 SYS$CMKRNL
Troubleshooting a Hung System Integrity (cont)
SDA> sho exception 7FF430A0
Exception Frame at 00000000.7FF430A0
------------------------------------
IPL = 08
TRAP_TYPE = 00000080 External Interrupt
IVT_OFFSET = 00003000 External Interrupt
IIP = FFFFFFFF.80145950 EXE$TIMEOUT_C+00320
IIPA = FFFFFFFF.80145950 EXE$TIMEOUT_C+00320
IFA = FFFFFFFF.7E34A000
IPSR = 00001410.08026010 RT TB LP DB SI DI PP SP DFH DFL DT PK I IC MFH MFL AC BE UP
1 0 0 0 0 0 0 0 0 0 1 0 1 1 0 1 0 0 0
IA BN ED RI SS DD DA ID IT MC IS CPL
0 1 0 2 0 0 0 0 1 0 0 0
PREVSTACK = 00
BSP = 00000000.7FF2F208
BSPSTORE = 00000000.7FF2F208
BSPBASE = 00000000.7FF2F208
RNAT = 00000000.00000000
Troubleshooting a Hung System Integrity (cont)
SDA> pcs sho trace
PC sampler information:
-----------------------
Timestamp CPU PC IPL M Pid Routine Module
---------------------- --- ----------------- --- - -------- -------------------------------- ------
14-MAY 17:27:14.605993 00 FFFFFFFF.8003C200 21 I 34400430 BOO$READB_C SYS$PLATFORM_SUPPORT
14-MAY 17:27:14.594993 00 FFFFFFFF.810B4D90 21 I 34400430 SYS$SRDRIVER+11B90 SYS$SRDRIVER
14-MAY 17:27:14.583994 00 FFFFFFFF.810B4D90 21 I 34400430 SYS$SRDRIVER+11B90 SYS$SRDRIVER
14-MAY 17:27:14.572994 00 FFFFFFFF.810B4D90 21 I 34400430 SYS$SRDRIVER+11B90 SYS$SRDRIVER
.
.
.
14-MAY 17:27:13.483994 00 FFFFFFFF.810B4D90 21 I 34400430 SYS$SRDRIVER+11B90 SYS$SRDRIVER
14-MAY 17:27:13.472994 00 FFFFFFFF.810B4D90 21 I 34400430 SYS$SRDRIVER+11B90 SYS$SRDRIVER
14-MAY 17:27:13.461993 00 FFFFFFFF.801459A0 8 I 34400430 EXE$TIMEOUT_C+00370 SYSTEM_PRIMITIVES_MIN
14-MAY 17:27:13.450993 00 FFFFFFFF.80145980 8 I 34400430 EXE$TIMEOUT_C+00350 SYSTEM_PRIMITIVES_MIN
14-MAY 17:27:13.439993 00 FFFFFFFF.801459B0 8 I 34400430 EXE$TIMEOUT_C+00380 SYSTEM_PRIMITIVES_MIN
14-MAY 17:27:13.428993 00 FFFFFFFF.80145970 8 I 34400430 EXE$TIMEOUT_C+00340 SYSTEM_PRIMITIVES_MIN
14-MAY 17:27:13.417993 00 FFFFFFFF.80145950 8 I 34400430 EXE$TIMEOUT_C+00320 SYSTEM_PRIMITIVES_MIN
14-MAY 17:27:13.406993 00 FFFFFFFF.80145940 8 I 34400430 EXE$TIMEOUT_C+00310 SYSTEM_PRIMITIVES_MIN
14-MAY 17:27:13.395993 00 FFFFFFFF.801459B0 8 I 34400430 EXE$TIMEOUT_C+00380 SYSTEM_PRIMITIVES_MIN
14-MAY 17:27:13.384993 00 FFFFFFFF.80145930 8 I 34400430 EXE$TIMEOUT_C+00300 SYSTEM_PRIMITIVES_MIN
14-MAY 17:27:13.373993 00 FFFFFFFF.80145940 8 I 34400430 EXE$TIMEOUT_C+00310 SYSTEM_PRIMITIVES_MIN
14-MAY 17:27:13.362993 00 FFFFFFFF.80145970 8 I 34400430 EXE$TIMEOUT_C+00340 SYSTEM_PRIMITIVES_MIN
14-MAY 17:27:13.351993 00 FFFFFFFF.80145940 8 I 34400430 EXE$TIMEOUT_C+00310 SYSTEM_PRIMITIVES_MIN
14-MAY 17:27:13.340993 00 FFFFFFFF.80145970 8 I 34400430 EXE$TIMEOUT_C+00340 SYSTEM_PRIMITIVES_MIN
14-MAY 17:27:13.329993 00 FFFFFFFF.801459B0 8 I 34400430 EXE$TIMEOUT_C+00380 SYSTEM_PRIMITIVES_MIN
14-MAY 17:27:13.318993 00 FFFFFFFF.80145940 8 I 34400430 EXE$TIMEOUT_C+00310 SYSTEM_PRIMITIVES_MIN
14-MAY 17:27:13.307993 00 FFFFFFFF.801459D0 8 I 34400430 EXE$TIMEOUT_C+003A0 SYSTEM_PRIMITIVES_MIN
Press RETURN for more.
Troubleshooting a Hung System Integrity (cont)
SDA>
PC sampler information:
-----------------------
Timestamp CPU PC IPL M Pid Routine Module
---------------------- --- ----------------- --- - -------- -------------------------------- ------
14-MAY 17:27:13.296993 00 FFFFFFFF.801459B0 8 I 34400430 EXE$TIMEOUT_C+00380 SYSTEM_PRIMITIVES_MIN
14-MAY 17:27:13.285993 00 FFFFFFFF.80145940 8 I 34400430 EXE$TIMEOUT_C+00310 SYSTEM_PRIMITIVES_MIN
14-MAY 17:27:13.274993 00 FFFFFFFF.80145950 8 I 34400430 EXE$TIMEOUT_C+00320 SYSTEM_PRIMITIVES_MIN
14-MAY 17:27:13.263993 00 FFFFFFFF.801459D0 8 I 34400430 EXE$TIMEOUT_C+003A0 SYSTEM_PRIMITIVES_MIN
14-MAY 17:27:13.252993 00 FFFFFFFF.801459D0 8 I 34400430 EXE$TIMEOUT_C+003A0 SYSTEM_PRIMITIVES_MIN
14-MAY 17:27:13.241993 00 FFFFFFFF.80145930 8 I 34400430 EXE$TIMEOUT_C+00300 SYSTEM_PRIMITIVES_MIN
14-MAY 17:27:13.230993 00 FFFFFFFF.801459D0 8 I 34400430 EXE$TIMEOUT_C+003A0 SYSTEM_PRIMITIVES_MIN
14-MAY 17:27:13.219993 00 FFFFFFFF.80145940 8 I 34400430 EXE$TIMEOUT_C+00310 SYSTEM_PRIMITIVES_MIN
14-MAY 17:27:13.208993 00 FFFFFFFF.801459E0 8 I 34400430 EXE$TIMEOUT_C+003B0 SYSTEM_PRIMITIVES_MIN
14-MAY 17:27:13.197993 00 FFFFFFFF.801459E0 8 I 34400430 EXE$TIMEOUT_C+003B0 SYSTEM_PRIMITIVES_MIN
14-MAY 17:27:13.186993 00 FFFFFFFF.801459A0 8 I 34400430 EXE$TIMEOUT_C+00370 SYSTEM_PRIMITIVES_MIN
14-MAY 17:27:13.175993 00 FFFFFFFF.801459A0 8 I 34400430 EXE$TIMEOUT_C+00370 SYSTEM_PRIMITIVES_MIN
14-MAY 17:27:13.164993 00 FFFFFFFF.80145970 8 I 34400430 EXE$TIMEOUT_C+00340 SYSTEM_PRIMITIVES_MIN
14-MAY 17:27:13.153993 00 FFFFFFFF.801459A0 8 I 34400430 EXE$TIMEOUT_C+00370 SYSTEM_PRIMITIVES_MIN
14-MAY 17:27:13.142993 00 FFFFFFFF.801459A0 8 I 34400430 EXE$TIMEOUT_C+00370 SYSTEM_PRIMITIVES_MIN
14-MAY 17:27:13.131993 00 FFFFFFFF.80145970 8 I 34400430 EXE$TIMEOUT_C+00340 SYSTEM_PRIMITIVES_MIN
14-MAY 17:27:13.120993 00 FFFFFFFF.801459D0 8 I 34400430 EXE$TIMEOUT_C+003A0 SYSTEM_PRIMITIVES_MIN
Troubleshooting a Hung System Integrity (cont)
SDA> pcs sho trace/statistics
PC sampler information:
-----------------------
PC IPL Pid Count Routine Module
----------------- --- -------- ------------ -------------------------------- ------
FFFFFFFF.80145970 8 34400430 575 EXE$TIMEOUT_C+00340 SYSTEM_PRIMITIVES_MIN
FFFFFFFF.801459A0 8 34400430 537 EXE$TIMEOUT_C+00370 SYSTEM_PRIMITIVES_MIN
FFFFFFFF.801459D0 8 34400430 489 EXE$TIMEOUT_C+003A0 SYSTEM_PRIMITIVES_MIN
FFFFFFFF.801459E0 8 34400430 352 EXE$TIMEOUT_C+003B0 SYSTEM_PRIMITIVES_MIN
FFFFFFFF.80145950 8 34400430 343 EXE$TIMEOUT_C+00320 SYSTEM_PRIMITIVES_MIN
FFFFFFFF.801459B0 8 34400430 337 EXE$TIMEOUT_C+00380 SYSTEM_PRIMITIVES_MIN
FFFFFFFF.80145940 8 34400430 274 EXE$TIMEOUT_C+00310 SYSTEM_PRIMITIVES_MIN
FFFFFFFF.80145930 8 34400430 249 EXE$TIMEOUT_C+00300 SYSTEM_PRIMITIVES_MIN
FFFFFFFF.80145980 8 34400430 182 EXE$TIMEOUT_C+00350 SYSTEM_PRIMITIVES_MIN
FFFFFFFF.80145990 8 34400430 144 EXE$TIMEOUT_C+00360 SYSTEM_PRIMITIVES_MIN
FFFFFFFF.810B4D90 21 34400430 102 SYS$SRDRIVER+11B90 SYS$SRDRIVER
FFFFFFFF.805100E0 2 34400430 6 IO_ROUTINES+F10E0 IO_ROUTINES
FFFFFFFF.804682D0 2 34400430 6 IOC$SCAN_IODB_C+00030 IO_ROUTINES
FFFFFFFF.805100A0 2 34400430 4 IO_ROUTINES+F10A0 IO_ROUTINES
FFFFFFFF.80510080 2 34400430 3 IO_ROUTINES+F1080 IO_ROUTINES
FFFFFFFF.80510070 2 34400430 3 IO_ROUTINES+F1070 IO_ROUTINES
FFFFFFFF.80468340 2 34400430 3 IOC$SCAN_IODB_C+000A0 IO_ROUTINES
Press RETURN for more.
SDA> set out delme.out
SDA> pcs sho trace/full
SDA> set out tt
SDA> spawn
CLASS8$ edit/read delme.out
Troubleshooting a Hung System Integrity (cont)
OpenVMS Operating System, Version V8.3 -- System Dump Analysis 14-MAY-2007 17:27:14.60 Page 2
PC sampler information:
Time = 14-MAY 17:27:14.605993
CPU = 00
IPL = 21
Mode = Interrupt
PID = 34400430 MEHLHOP
PC = FFFFFFFF.8003C200 BOO$READB_C SYS$PLATFORM_SUPPORT
B0 = FFFFFFFF.810B4D90 SYS$SRDRIVER+11B90
R2 = 00000000.7FF43040
R3 = 00000000.F4050005
R4 = 00000000.00000000
R5 = FFFFFFFF.88F7C700 UCB (Device TTA0:)
R6 = FFFFFFFF.89008600 DDB
R7 = 00000000.00000038
Time = 14-MAY 17:27:14.594993
CPU = 00
IPL = 21
Mode = Interrupt
PID = 34400430 MEHLHOP
PC = FFFFFFFF.810B4D90 SYS$SRDRIVER+11B90 SYS$SRDRIVER
B0 = FFFFFFFF.810B4D90 SYS$SRDRIVER+11B90
R2 = 00000000.7FF43040
R3 = 00000000.00000060
R4 = 00000000.00000000
R5 = FFFFFFFF.88F7C700 UCB (Device TTA0:)
R6 = FFFFFFFF.89008600 DDB
R7 = 00000000.00000038
.
Troubleshooting a Hung System Integrity (cont)
OpenVMS Operating System, Version V8.3 -- System Dump Analysis 14-MAY-2007 17:27:14.60 Page 32
PC sampler information:
Time = 14-MAY 17:27:13.450993
CPU = 00
IPL = 8
Mode = Interrupt
PID = 34400430 MEHLHOP
PC = FFFFFFFF.80145980 EXE$TIMEOUT_C+00350 SYSTEM_PRIMITIVES_MIN
B0 = FFFFFFFF.80145880 EXE$TIMEOUT_C+00250
R2 = FFFFFFFF.88033228 ECC$REENABLE+00028
R3 = FFFFFFFF.880165C8 IOC$GL_DEVLIST
R4 = 00000000.00000000
R5 = FFFFFFFF.88F7C700 UCB (Device TTA0:)
R6 = FFFFFFFF.89008600 DDB
R7 = 00000000.00000038
Time = 14-MAY 17:27:13.439993
CPU = 00
IPL = 8
Mode = Interrupt
PID = 34400430 MEHLHOP
PC = FFFFFFFF.801459B0 EXE$TIMEOUT_C+00380 SYSTEM_PRIMITIVES_MIN
B0 = FFFFFFFF.80145880 EXE$TIMEOUT_C+00250
R2 = FFFFFFFF.88033228 ECC$REENABLE+00028
R3 = FFFFFFFF.880165C8 IOC$GL_DEVLIST
R4 = 00000000.00000000
R5 = FFFFFFFF.88F7C700 UCB (Device TTA0:)
R6 = FFFFFFFF.89008600 DDB
R7 = 00000000.00000038
Troubleshooting a Hung System Integrity (cont)
Time = 14-MAY 17:27:13.428993
CPU = 00
IPL = 8
Mode = Interrupt
PID = 34400430 MEHLHOP
PC = FFFFFFFF.80145970 EXE$TIMEOUT_C+00340 SYSTEM_PRIMITIVES_MIN
B0 = FFFFFFFF.80145880 EXE$TIMEOUT_C+00250
R2 = FFFFFFFF.88033228 ECC$REENABLE+00028
R3 = FFFFFFFF.880165C8 IOC$GL_DEVLIST
R4 = 00000000.00000000
R5 = FFFFFFFF.88F7C760
R6 = FFFFFFFF.89008600 DDB
R7 = 00000000.00000038
Time = 14-MAY 17:27:13.417993
CPU = 00
IPL = 8
Mode = Interrupt
PID = 34400430 MEHLHOP
PC = FFFFFFFF.80145950 EXE$TIMEOUT_C+00320 SYSTEM_PRIMITIVES_MIN
B0 = FFFFFFFF.80145880 EXE$TIMEOUT_C+00250
R2 = FFFFFFFF.88033228 ECC$REENABLE+00028
R3 = FFFFFFFF.880165C8 IOC$GL_DEVLIST
R4 = 00000000.00000000
R5 = FFFFFFFF.88F7C700 UCB (Device TTA0:)
R6 = FFFFFFFF.89008600 DDB
R7 = 00000000.00000038
Troubleshooting a Hung System AXP
SYS6$ run hang_ddb_corrupt
(Hit the HALT switch instead of ^P)
halted CPU 0
halt code = 1
operator initiated halt
PC = ffffffff80051bac
>>>crash
CPU 0 restarting
**** OpenVMS Alpha Operating System V8.2 - BUGCHECK ****
** Bugcheck code = 0000064C: OPERCRASH, Operator forced system crash
** Crash CPU: 00 Primary CPU: 00 Active CPUs: 00000001
** Current Process = MEHLHOP
** Current PSB ID = 00000001
** Image Name =
**** Starting full memory dump at 30-APR-2007 16:51...
................................................................................
................................................................................
.
.
.
................................................................................
................................................................................
....................
...Complete ****
halted CPU 0
halt code = 5
HALT instruction executed
PC = ffffffff80081b34
CPU 0 booting
Troubleshooting High MP Synch time
CLASS8$ run hispin
CLASS8$ an/sy
OpenVMS system analyzer
SDA> spl sho trace
Spinlock Trace Information:
---------------------------
Timestamp CPU Spin/Forklock/IPL Caller's/Fork PC EPID Operation Trace Buffer
---------------------- --- --------------------- -------------------------------------- -------- ----------------- -----------------
16-MAY 11:54:57.378645 01 881C0B00 MMG 8067DB60 SYS$VM+35660 00000000 Release FFFFFFFF.7DC06E20
16-MAY 11:54:57.378640 01 881C0B00 MMG 80678C10 MMG$PAGEFAULT_C+00190 00000000 Acquire FFFFFFFF.7DC06DF8
16-MAY 11:54:57.378618 01 881C0B00 MMG 80679610 MMG$PAGEFAULT_C+00B90 00000000 Release FFFFFFFF.7DC06DD0
16-MAY 11:54:57.378617 01 881C0B00 MMG 80678C10 MMG$PAGEFAULT_C+00190 00000000 Acquire FFFFFFFF.7DC06DA8
16-MAY 11:54:57.378608 01 881C0B00 MMG 80679610 MMG$PAGEFAULT_C+00B90 00000000 Release FFFFFFFF.7DC06D80
16-MAY 11:54:57.378606 01 881C0B00 MMG 80678C10 MMG$PAGEFAULT_C+00190 00000000 Acquire FFFFFFFF.7DC06D58
16-MAY 11:54:57.378595 01 881C0B00 MMG 8067DB60 SYS$VM+35660 00000000 Release FFFFFFFF.7DC06D30
16-MAY 11:54:57.378591 01 881C0B00 MMG 80678C10 MMG$PAGEFAULT_C+00190 00000000 Acquire FFFFFFFF.7DC06D08
16-MAY 11:54:57.378589 01 881C0B00 MMG 8067BA70 SYS$VM+33570 00000000 Release FFFFFFFF.7DC06CE0
16-MAY 11:54:57.378586 01 881C0B00 MMG 80678C10 MMG$PAGEFAULT_C+00190 00000000 Acquire FFFFFFFF.7DC06CB8
16-MAY 11:54:57.378528 01 881C0B00 MMG 80679610 MMG$PAGEFAULT_C+00B90 00000000 Release FFFFFFFF.7DC06C90
16-MAY 11:54:57.378527 01 881C0B00 MMG 80678C10 MMG$PAGEFAULT_C+00190 00000000 Acquire FFFFFFFF.7DC06C68
16-MAY 11:54:57.378523 01 881C0B00 MMG 80673A70 MMG_STD$IN_REGION_64_C+00830 00000000 Release FFFFFFFF.7DC06C40
16-MAY 11:54:57.378523 01 881C0B00 MMG 80673900 MMG_STD$IN_REGION_64_C+006C0 00000000 Acquire FFFFFFFF.7DC06C18
16-MAY 11:54:57.378520 01 881C0B00 MMG 806E8D70 MMG_STD$CLONE_REGIONS_64_C+00 00000000 Release FFFFFFFF.7DC06BF0
16-MAY 11:54:57.378520 01 881C0B00 MMG 806E8D00 MMG_STD$CLONE_REGIONS_64_C+00 00000000 Acquire FFFFFFFF.7DC06BC8
16-MAY 11:54:57.378506 01 881C0B00 MMG 80679610 MMG$PAGEFAULT_C+00B90 00000000 Release FFFFFFFF.7DC06BA0
Press RETURN for more.
Troubleshooting High MP Synch time (cont)
SDA>
Spinlock Trace Information:
---------------------------
Timestamp CPU Spin/Forklock/IPL Caller's/Fork PC EPID Operation Trace Buffer
---------------------- --- --------------------- -------------------------------------- -------- ----------------- -----------------
16-MAY 11:54:57.378504 01 881C0B00 MMG 80678C10 MMG$PAGEFAULT_C+00190 00000000 Acquire FFFFFFFF.7DC06B78
16-MAY 11:54:57.378500 01 881C0B00 MMG 8067DB60 SYS$VM+35660 00000000 Release FFFFFFFF.7DC06B50
16-MAY 11:54:57.378491 01 881C0B00 MMG 80678C10 MMG$PAGEFAULT_C+00190 00000000 Acquire FFFFFFFF.7DC06B28
16-MAY 11:54:57.378487 01 881C0A00 SCHED 8052C250 PROCESS_MANAGEMENT+1A550 00000000 Release FFFFFFFF.7DC06B00
16-MAY 11:54:57.378484 01 881C0A00 SCHED 8052BE80 PROCESS_MANAGEMENT+1A180 00000000 Acquire FFFFFFFF.7DC06AD8
16-MAY 11:54:57.378483 01 881C0A00 SCHED 80586AD0 SCH$QEND_C+00AF0 00000000 Release FFFFFFFF.7DC06AB0
16-MAY 11:54:57.378482 01 881C0A00 SCHED 80586500 SCH$QEND_C+00520 00000000 Acquire FFFFFFFF.7DC06A88
16-MAY 11:54:57.378482 00 881C1100 LCKMGR 00020200 SYS$K_VERSION_08+001E0 00000000 Acquire FFFFFFFF.7DC06A60
16-MAY 11:54:57.378481 01 881C1100 LCKMGR 000202E0 SYS$K_VERSION_08+002C0 34600442 Release FFFFFFFF.7DC06A38
16-MAY 11:54:57.377999 00 881C0200 HWCLK 80149960 SYSTEM_PRIMITIVES_MIN+0012996 00000000 Release FFFFFFFF.7DC06A10
16-MAY 11:54:57.377999 00 881C0200 HWCLK 801496B0 SYSTEM_PRIMITIVES_MIN+001296B 00000000 Acqnoipl FFFFFFFF.7DC069E8
16-MAY 11:54:57.376999 00 881C0200 HWCLK 80149960 SYSTEM_PRIMITIVES_MIN+0012996 00000000 Release FFFFFFFF.7DC069C0
16-MAY 11:54:57.376999 00 881C0200 HWCLK 801496B0 SYSTEM_PRIMITIVES_MIN+001296B 00000000 Acqnoipl FFFFFFFF.7DC06998
16-MAY 11:54:57.376233 00 881C1100 LCKMGR 00020200 SYS$K_VERSION_08+001E0 00000000 Acquire (spin) FFFFFFFF.7DC06970
16-MAY 11:54:57.376232 01 881C1100 LCKMGR 00020200 SYS$K_VERSION_08+001E0 34600442 Acquire FFFFFFFF.7DC06948
16-MAY 11:54:57.376232 00 881C1100 LCKMGR 000202E0 SYS$K_VERSION_08+002C0 00000000 Release FFFFFFFF.7DC06920
16-MAY 11:54:57.375999 00 881C0200 HWCLK 80149960 SYSTEM_PRIMITIVES_MIN+0012996 00000000 Release FFFFFFFF.7DC068F8
16-MAY 11:54:57.375999 00 881C0200 HWCLK 801496B0 SYSTEM_PRIMITIVES_MIN+001296B 00000000 Acqnoipl FFFFFFFF.7DC068D0
16-MAY 11:54:57.374999 00 881C0200 HWCLK 80149960 SYSTEM_PRIMITIVES_MIN+0012996 00000000 Release FFFFFFFF.7DC068A8
16-MAY 11:54:57.374999 00 881C0200 HWCLK 801496B0 SYSTEM_PRIMITIVES_MIN+001296B 00000000 Acqnoipl FFFFFFFF.7DC06880
16-MAY 11:54:57.373999 00 881C0200 HWCLK 80149960 SYSTEM_PRIMITIVES_MIN+0012996 00000000 Release FFFFFFFF.7DC06858
16-MAY 11:54:57.373999 00 881C0200 HWCLK 801496B0 SYSTEM_PRIMITIVES_MIN+001296B 00000000 Acqnoipl FFFFFFFF.7DC06830
16-MAY 11:54:57.373980 01 881C1100 LCKMGR 00020200 SYS$K_VERSION_08+001E0 34600442 Acquire (spin) FFFFFFFF.7DC06808
16-MAY 11:54:57.373981 00 881C1100 LCKMGR 00020200 SYS$K_VERSION_08+001E0 00000000 Acquire FFFFFFFF.7DC067E0
16-MAY 11:54:57.373980 01 881C1100 LCKMGR 000202E0 SYS$K_VERSION_08+002C0 34600442 Release FFFFFFFF.7DC067B8
16-MAY 11:54:57.372999 00 881C0200 HWCLK 80149960 SYSTEM_PRIMITIVES_MIN+0012996 00000000 Release FFFFFFFF.7DC06790
16-MAY 11:54:57.372999 00 881C0200 HWCLK 801496B0 SYSTEM_PRIMITIVES_MIN+001296B 00000000 Acqnoipl FFFFFFFF.7DC06768
16-MAY 11:54:57.371999 00 881C0200 HWCLK 80149960 SYSTEM_PRIMITIVES_MIN+0012996 00000000 Release FFFFFFFF.7DC06740
16-MAY 11:54:57.371999 00 881C0200 HWCLK 801496B0 SYSTEM_PRIMITIVES_MIN+001296B 00000000 Acqnoipl FFFFFFFF.7DC06718
16-MAY 11:54:57.371731 00 881C1100 LCKMGR 00020200 SYS$K_VERSION_08+001E0 00000000 Acquire (spin) FFFFFFFF.7DC066F0
16-MAY 11:54:57.371730 01 881C1100 LCKMGR 00020200 SYS$K_VERSION_08+001E0 34600442 Acquire FFFFFFFF.7DC066C8
16-MAY 11:54:57.371730 00 881C1100 LCKMGR 000202E0 SYS$K_VERSION_08+002C0 00000000 Release FFFFFFFF.7DC066A0
16-MAY 11:54:57.370999 00 881C0200 HWCLK 80149960 SYSTEM_PRIMITIVES_MIN+0012996 00000000 Release FFFFFFFF.7DC06678
16-MAY 11:54:57.370999 00 881C0200 HWCLK 801496B0 SYSTEM_PRIMITIVES_MIN+001296B 00000000 Acqnoipl FFFFFFFF.7DC06650
Troubleshooting High MP Synch time (cont)
SDA>
Spinlock Trace Information:
---------------------------
Timestamp CPU Spin/Forklock/IPL Caller's/Fork PC EPID Operation Trace Buffer
---------------------- --- --------------------- -------------------------------------- -------- ----------------- -----------------
16-MAY 11:54:57.362728 00 TIMER 80143560 SYSTEM_PRIMITIVES_MIN+0012356 Frk Dsp Start FFFFFFFF.7DC060D8
16-MAY 11:54:57.362728 00 881C0D00 TIMER 80143140 EXE_STD$IOFORK_CPU_C+00EC0 00000000 Acquire FFFFFFFF.7DC060B0
16-MAY 11:54:57.362727 01 881C1100 LCKMGR 00020200 SYS$K_VERSION_08+001E0 34600442 Acquire FFFFFFFF.7DC06088
16-MAY 11:54:57.362728 00 881C1100 LCKMGR 000202E0 SYS$K_VERSION_08+002C0 00000000 Release FFFFFFFF.7DC06060
16-MAY 11:54:57.361999 00 881C0200 HWCLK 80149960 SYSTEM_PRIMITIVES_MIN+0012996 00000000 Release FFFFFFFF.7DC06038
16-MAY 11:54:57.361999 00 881C0200 HWCLK 801496B0 SYSTEM_PRIMITIVES_MIN+001296B 00000000 Acqnoipl FFFFFFFF.7DC06010
16-MAY 11:54:57.361000 00 881C0200 HWCLK 80149960 SYSTEM_PRIMITIVES_MIN+0012996 00000000 Release FFFFFFFF.7DC05FE8
16-MAY 11:54:57.360999 00 881C0200 HWCLK 801496B0 SYSTEM_PRIMITIVES_MIN+001296B 00000000 Acqnoipl FFFFFFFF.7DC05FC0
16-MAY 11:54:57.360512 01 881C1100 LCKMGR 00020200 SYS$K_VERSION_08+001E0 34600442 Acquire (spin) FFFFFFFF.7DC05F98
16-MAY 11:54:57.360511 01 IPL 08 Frk Dsp End FFFFFFFF.7DC05F70
16-MAY 11:54:57.360511 01 88E09C80 EWA$PORT 80295960 SYS$LAN_CSMACD+32360 00000000 Restorel FFFFFFFF.7DC05F48
16-MAY 11:54:57.360511 01 88E09C80 EWA$PORT 802957B0 SYS$LAN_CSMACD+321B0 00000000 Acqnoipl FFFFFFFF.7DC05F20
16-MAY 11:54:57.360511 01 88E4D540 PEA$PORT 81128000 SYS$PEDRIVER+53A00 00000000 Restorel FFFFFFFF.7DC05EF8
16-MAY 11:54:57.360511 01 88E4D540 PEA$PORT 81127EE0 SYS$PEDRIVER+538E0 00000000 Acqnoipl FFFFFFFF.7DC05ED0
16-MAY 11:54:57.360510 01 IPL 08 80295630 LAN$COMPLETE_XMT_CSMACD_C+00E Frk Dsp Start FFFFFFFF.7DC05EA8
16-MAY 11:54:57.360510 01 IPL 08 Frk Dsp End FFFFFFFF.7DC05E80
16-MAY 11:54:57.360510 01 88E09C80 EWA$PORT 80295960 SYS$LAN_CSMACD+32360 00000000 Restorel FFFFFFFF.7DC05E58
16-MAY 11:54:57.360510 01 88E09C80 EWA$PORT 802957B0 SYS$LAN_CSMACD+321B0 00000000 Acqnoipl FFFFFFFF.7DC05E30
16-MAY 11:54:57.360510 01 88E4D540 PEA$PORT 81128000 SYS$PEDRIVER+53A00 00000000 Restorel FFFFFFFF.7DC05E08
16-MAY 11:54:57.360509 01 88E4D540 PEA$PORT 81127EE0 SYS$PEDRIVER+538E0 00000000 Acqnoipl FFFFFFFF.7DC05DE0
16-MAY 11:54:57.360509 01 IPL 08 80295630 LAN$COMPLETE_XMT_CSMACD_C+00E Frk Dsp Start FFFFFFFF.7DC05DB8
16-MAY 11:54:57.360508 01 IPL 08 Frk Dsp End FFFFFFFF.7DC05D90
16-MAY 11:54:57.360508 01 88E09C80 EWA$PORT 810C5220 SYS$EW5700+1E320 00000000 Restorel FFFFFFFF.7DC05D68
16-MAY 11:54:57.360505 01 88E4D540 PEA$PORT 811284C0 SYS$PEDRIVER+53EC0 00000000 Restorel FFFFFFFF.7DC05D40
16-MAY 11:54:57.360505 01 88E09C80 EWA$PORT 8026B760 LAN$RETURN_RCV_VCRP_C+00240 00000000 Restorel FFFFFFFF.7DC05D18
16-MAY 11:54:57.360505 01 88E09C80 EWA$PORT 8026B5E0 LAN$RETURN_RCV_VCRP_C+000C0 00000000 Acqnoipl (own) FFFFFFFF.7DC05CF0
16-MAY 11:54:57.360503 01 88E4D540 PEA$PORT 811283D0 SYS$PEDRIVER+53DD0 00000000 Acquire (nospin) FFFFFFFF.7DC05CC8
16-MAY 11:54:57.360501 01 88E4D540 PEA$PORT 811284C0 SYS$PEDRIVER+53EC0 00000000 Restorel FFFFFFFF.7DC05CA0
16-MAY 11:54:57.360501 01 88E09C80 EWA$PORT 8026B760 LAN$RETURN_RCV_VCRP_C+00240 00000000 Restorel FFFFFFFF.7DC05C78
16-MAY 11:54:57.360500 01 88E09C80 EWA$PORT 8026B5E0 LAN$RETURN_RCV_VCRP_C+000C0 00000000 Acqnoipl (own) FFFFFFFF.7DC05C50
16-MAY 11:54:57.360491 01 88E4D540 PEA$PORT 811283D0 SYS$PEDRIVER+53DD0 00000000 Acquire (nospin) FFFFFFFF.7DC05C28
16-MAY 11:54:57.360486 01 88E09C80 EWA$PORT 810C4960 SYS$EW5700+1DA60 00000000 Acqnoipl FFFFFFFF.7DC05C00
16-MAY 11:54:57.360486 01 IPL 08 810C48C0 SYS$EW5700+1D9C0 Frk Dsp Start FFFFFFFF.7DC05BD8
Troubleshooting High MP Synch time (cont)
SDA> spl sho trace/sum
Spinlock Trace Information: (at 16-MAY-2007 13:13:42.55, trace time 00:00:42.377022)
------------------------------------------------------------------------------------
Events Acquires Releases Acq Own Acq NoSpin Spinwaits %
Spinlock /sec /sec /sec /sec /sec /sec Spinwait
---------- ---------- ---------- ---------- ---------- ---------- ---------- ----------
MEGA 2.0 1.0 1.0 0.0 0.0 0.0 0.0
HWCLK 2000.0 1000.0 1000.0 0.0 0.0 0.0 0.0
INVALIDATE 4.9 2.4 2.4 0.0 0.0 0.0 0.0
PERFMON 10.7 5.3 5.3 0.0 0.0 0.0 0.0
SCHED 1872.0 935.6 935.6 0.0 0.0 0.7 0.1
MMG 169.5 83.3 84.6 0.1 1.3 0.2 0.2
TIMER 617.5 306.5 306.5 0.0 0.0 4.4 1.4
TX_SYNCH 0.9 0.5 0.5 0.0 0.0 0.0 0.0
IOLOCK8 147.7 73.6 73.8 0.3 0.0 0.0 0.1
LCKMGR 372.7 130.8 131.5 0.0 0.7 109.8 83.5
FILSYS 4.6 2.3 2.3 0.0 0.0 0.0 0.0
Dynamic 631.5 279.0 315.1 22.3 14.8 0.2 0.1
---------- ---------- ---------- ---------- ---------- ---------- ---------- ----------
5833.9 2820.3 2858.8 22.8 16.7 115.3 4.1
Press RETURN for more.
Troubleshooting High MP Synch time (cont)
SDA>
Spinlock Trace Information:
---------------------------
Events Acquires or Spins % Own
Spinlock /sec Releases/sec /sec Spin /sec Caller's PC Module Offset
---------- ---------- --------------- ------- ----- ------ -------------------------------------- ----------------------------------
MEGA 1.0 1.0 Rel/s 0.0 0.0 0.0 80147620 SYSTEM_PRIMITIVES_MIN+001276 SYSTEM_PRIMITIVES_MIN 00127620
MEGA 1.0 1.0 Acq/s 0.0 0.0 0.0 80146BB0 SYSTEM_PRIMITIVES_MIN+00126B SYSTEM_PRIMITIVES_MIN 00126BB0
HWCLK 1000.0 1000.0 Acq/s 0.0 0.0 0.0 801496B0 SYSTEM_PRIMITIVES_MIN+001296 SYSTEM_PRIMITIVES_MIN 001296B0
HWCLK 1000.0 1000.0 Rel/s 0.0 0.0 0.0 80149960 SYSTEM_PRIMITIVES_MIN+001299 SYSTEM_PRIMITIVES_MIN 00129960
INVALIDATE 1.7 1.7 Rel/s 0.0 0.0 0.0 80725F30 SYS$VM+DDA30 SYS$VM 000DDA30
INVALIDATE 1.7 1.7 Acq/s 0.0 0.0 0.0 80724E50 MMG$TBI_DATA_64_THREADS_C+00 SYS$VM 000DC950
INVALIDATE 0.7 0.7 Acq/s 0.0 0.0 0.0 80728030 SYS$VM+DFB30 SYS$VM 000DFB30
INVALIDATE 0.7 0.7 Rel/s 0.0 0.0 0.0 80729110 SYS$VM+E0C10 SYS$VM 000E0C10
PERFMON 2.1 2.1 Acq/s 0.0 0.0 0.0 80440E40 PMS$END_RQ_C+001B0 IO_ROUTINES 00021E40
PERFMON 2.1 2.1 Acq/s 0.0 0.0 0.0 80441F10 PMS$START_RQ_C+00230 IO_ROUTINES 00022F10
PERFMON 2.1 2.1 Rel/s 0.0 0.0 0.0 80442230 PMS$START_RQ_C+00550 IO_ROUTINES 00023230
PERFMON 1.4 1.4 Rel/s 0.0 0.0 0.0 804414E0 PMS$END_RQ_C+00850 IO_ROUTINES 000224E0
Troubleshooting High MP Synch time (cont)
Press RETURN for more.
SDA>
Spinlock Trace Information:
---------------------------
Events Acquires or Spins % Own
Spinlock /sec Releases/sec /sec Spin /sec Caller's PC Module Offset
---------- ---------- --------------- ------- ----- ------ -------------------------------------- ----------------------------------
PERFMON 0.7 0.7 Rel/s 0.0 0.0 0.0 80440F70 PMS$END_RQ_C+002E0 IO_ROUTINES 00021F70
PERFMON 0.7 0.7 Acq/s 0.0 0.0 0.0 80440820 PMS$END_IO_C+00190 IO_ROUTINES 00021820
PERFMON 0.7 0.7 Rel/s 0.0 0.0 0.0 80440930 PMS$END_IO_C+002A0 IO_ROUTINES 00021930
PERFMON 0.3 0.3 Acq/s 0.0 0.0 0.0 80441840 PMS$START_IO_C+00190 IO_ROUTINES 00022840
PERFMON 0.3 0.3 Rel/s 0.0 0.0 0.0 80441980 PMS$START_IO_C+002D0 IO_ROUTINES 00022980
PERFMON 0.0 0.0 Rel/s 0.0 0.0 0.0 011B8720
SCHED 686.8 686.8 Acq/s 0.0 0.0 0.0 011A8CE0
SCHED 686.8 686.8 Rel/s 0.0 0.0 0.0 011A8E80
SCHED 136.0 136.0 Rel/s 0.0 0.0 0.0 011AD290
SCHED 136.0 136.0 Acq/s 0.0 0.0 0.0 011ACFB0
SCHED 32.7 32.7 Acq/s 0.0 0.0 0.0 815B89E0 PSDC$LOADABLE_IMAGE+147E0 PSDC$LOADABLE_IMAGE 000147E0
SCHED 32.7 32.7 Rel/s 0.0 0.0 0.0 815B8770 PSDC$LOADABLE_IMAGE+14570 PSDC$LOADABLE_IMAGE 00014570
SCHED 23.1 23.1 Rel/s 0.0 0.0 0.0 8052C250 PROCESS_MANAGEMENT+1A550 PROCESS_MANAGEMENT 0001A550
SCHED 17.0 16.8 Acq/s 0.2 1.1 0.0 805CF370 PROCESS_MANAGEMENT+BD670 PROCESS_MANAGEMENT 000BD670
SCHED 14.1 13.8 Acq/s 0.3 2.2 0.0 805654E0 SCH$CALC_CPU_LOAD_C+00E20 PROCESS_MANAGEMENT 000537E0
SCHED 13.8 13.8 Rel/s 0.0 0.0 0.0 80564FB0 SCH$CALC_CPU_LOAD_C+008F0 PROCESS_MANAGEMENT 000532B0
Troubleshooting High MP Synch time (cont)
Press RETURN for more.
SDA>
Spinlock Trace Information:
---------------------------
Events Acquires or Spins % Own
Spinlock /sec Releases/sec /sec Spin /sec Caller's PC Module Offset
---------- ---------- --------------- ------- ----- ------ -------------------------------------- ----------------------------------
MMG 53.2 53.1 Acq/s 0.1 0.2 0.0 800C0E40 MMG$READ_DATA_PHYS_C+00280 SYSTEM_PRIMITIVES_MIN 000A0E40
MMG 53.1 53.1 Rel/s 0.0 0.0 0.0 800C1AE0 MMG$READ_DATA_PHYS_C+00F20 SYSTEM_PRIMITIVES_MIN 000A1AE0
MMG 19.8 19.7 Acq/s 0.1 0.5 0.0 800C1D50 MMG$WRITE_DATA_PHYS_C+001E0 SYSTEM_PRIMITIVES_MIN 000A1D50
MMG 19.7 19.7 Rel/s 0.0 0.0 0.0 800C2AC0 MMG$WRITE_DATA_PHYS_C+00F50 SYSTEM_PRIMITIVES_MIN 000A2AC0
MMG 2.0 2.0 Acq/s 0.0 0.0 0.0 80678C10 MMG$PAGEFAULT_C+00190 SYS$VM 00030710
MMG 1.6 1.6 Acq/s 0.0 0.0 0.0 8017D660 MMG$PTEREF_64_C+002C0 SYSTEM_PRIMITIVES_MIN 0015D660
MMG 1.1 1.1 Rel/s 0.0 0.0 0.0 80673A70 MMG_STD$IN_REGION_64_C+00830 SYS$VM 0002B570
MMG 1.1 1.1 Rel/s 0.0 0.0 0.0 80679610 MMG$PAGEFAULT_C+00B90 SYS$VM 00031110
MMG 1.1 1.1 Rel/s 0.0 0.0 0.0 806A6950 SYS$VM+5E450 SYS$VM 0005E450
MMG 0.9 0.9 Acq/s 0.0 0.0 0.0 80673900 MMG_STD$IN_REGION_64_C+006C0 SYS$VM 0002B400
TIMER 118.6 116.2 Acq/s 2.4 2.1 0.0 80143D90 SYSTEM_PRIMITIVES_MIN+00123D SYSTEM_PRIMITIVES_MIN 00123D90
TIMER 103.6 101.6 Acq/s 2.0 2.0 0.0 80157B80 EXE$INSTIMQ_C+000C0 SYSTEM_PRIMITIVES_MIN 00137B80
TIMER 103.6 103.6 Rel/s 0.0 0.0 0.0 801440E0 SYSTEM_PRIMITIVES_MIN+001240 SYSTEM_PRIMITIVES_MIN 001240E0
TIMER 101.6 101.6 Rel/s 0.0 0.0 0.0 80157D00 EXE$INSTIMQ_C+00240 SYSTEM_PRIMITIVES_MIN 00137D00
TIMER 72.2 72.2 Acq/s 0.0 0.0 0.0 80143140 EXE_STD$IOFORK_CPU_C+00EC0 SYSTEM_PRIMITIVES_MIN 00123140
TIMER 72.2 72.2 Rel/s 0.0 0.0 0.0 80143450 SYSTEM_PRIMITIVES_MIN+001234 SYSTEM_PRIMITIVES_MIN 00123450
TIMER 12.6 12.6 Rel/s 0.0 0.0 0.0 80144590 SYSTEM_PRIMITIVES_MIN+001245 SYSTEM_PRIMITIVES_MIN 00124590
TIMER 12.6 12.6 Acq/s 0.0 0.0 0.0 805E00A0 PROCESS_MANAGEMENT+CE3A0 PROCESS_MANAGEMENT 000CE3A0
TIMER 12.6 12.6 Rel/s 0.0 0.0 0.0 805E02F0 PROCESS_MANAGEMENT+CE5F0 PROCESS_MANAGEMENT 000CE5F0
TIMER 3.0 3.0 Acq/s 0.0 0.0 0.0 805DF180 PROCESS_MANAGEMENT+CD480 PROCESS_MANAGEMENT 000CD480
Troubleshooting High MP Synch time (cont)
Press RETURN for more.
SDA>
Spinlock Trace Information:
---------------------------
Events Acquires or Spins % Own
Spinlock /sec Releases/sec /sec Spin /sec Caller's PC Module Offset
---------- ---------- --------------- ------- ----- ------ -------------------------------------- ----------------------------------
TX_SYNCH 0.5 0.5 Rel/s 0.0 0.0 0.0 80C45790 EXE$GENERATE_UID_C+004C0 SYS$TRANSACTION_SERVICES 00074A90
TX_SYNCH 0.5 0.5 Acq/s 0.0 0.0 0.0 80C45390 EXE$GENERATE_UID_C+000C0 SYS$TRANSACTION_SERVICES 00074690
IOLOCK8 48.6 48.6 Rel/s 0.0 0.0 0.0 810EF680 SYS$PEDRIVER+1B080 SYS$PEDRIVER 0001B080
IOLOCK8 48.6 48.6 Acq/s 0.0 0.0 0.0 810EF200 SYS$PEDRIVER+1AC00 SYS$PEDRIVER 0001AC00
IOLOCK8 9.9 9.9 Acq/s 0.0 0.0 0.0 812B5D10 SYS$LTDRIVER+17810 SYS$LTDRIVER 00017810
IOLOCK8 8.9 8.9 Rel/s 0.0 0.0 0.0 812B60C0 SYS$LTDRIVER+17BC0 SYS$LTDRIVER 00017BC0
IOLOCK8 6.9 6.9 Acq/s 0.0 0.0 0.0 80145B00 EXE$TIMEOUT_C+004D0 SYSTEM_PRIMITIVES_MIN 00125B00
IOLOCK8 6.9 6.9 Rel/s 0.0 0.0 0.0 80145D60 EXE$TIMEOUT_C+00730 SYSTEM_PRIMITIVES_MIN 00125D60
IOLOCK8 2.0 1.9 Acq/s 0.0 1.2 0.0 810F5B20 SYS$PEDRIVER+21520 SYS$PEDRIVER 00021520
IOLOCK8 1.9 1.9 Rel/s 0.0 0.0 0.0 810F5C90 SYS$PEDRIVER+21690 SYS$PEDRIVER 00021690
IOLOCK8 1.0 1.0 Rel/s 0.0 0.0 0.0 8125E610 NETDRIVER+32D10 NETDRIVER 00032D10
IOLOCK8 1.0 1.0 Rel/s 0.0 0.0 0.0 812B7570 SYS$LTDRIVER+19070 SYS$LTDRIVER 00019070
LCKMGR 232.4 124.7 Acq/s 107.7 86.4 0.0 00020200 SYS$K_VERSION_08+001E0
LCKMGR 124.7 124.7 Rel/s 0.0 0.0 0.0 000202E0 SYS$K_VERSION_08+002C0
LCKMGR 2.2 2.0 Acq/s 0.2 11.9 0.0 807A87A0 LCK$CALCULATE_LOCKIDTBL_MAX_ LOCKING 0004FBA0
LCKMGR 2.0 2.0 Rel/s 0.0 0.0 0.0 807A8820 LCK$CALCULATE_LOCKIDTBL_MAX_ LOCKING 0004FC20
LCKMGR 1.2 1.0 Acq/s 0.2 23.8 0.0 80146A90 SYSTEM_PRIMITIVES_MIN+00126A SYSTEM_PRIMITIVES_MIN 00126A90
LCKMGR 1.2 1.0 Acq/s 0.2 19.0 0.0 80DA3580 SYS$CLUSTER+98B80 SYS$CLUSTER 00098B80
LCKMGR 1.0 0.6 Acq/s 0.4 59.3 0.0 8077DA50 LCK$ENQUE_C+00B90 LOCKING 00024E50
LCKMGR 1.0 1.0 Rel/s 0.0 0.0 0.0 80147820 SYSTEM_PRIMITIVES_MIN+001278 SYSTEM_PRIMITIVES_MIN 00127820
LCKMGR 1.0 1.0 Rel/s 0.0 0.0 0.0 80DA37A0 SYS$CLUSTER+98DA0 SYS$CLUSTER 00098DA0
LCKMGR 0.8 0.8 Rel/s 0.0 0.0 0.0 8076FD70 LOCKING+17170 LOCKING 00017170
Troubleshooting High MP Synch time (cont)
Press RETURN for more.
SDA>
Spinlock Trace Information:
---------------------------
Events Acquires or Spins % Own
Spinlock /sec Releases/sec /sec Spin /sec Caller's PC Module Offset
---------- ---------- --------------- ------- ----- ------ -------------------------------------- ----------------------------------
FILSYS 0.4 0.4 Rel/s 0.0 0.0 0.0 806BCB50 SYS$VM+74650 SYS$VM 00074650
FILSYS 0.4 0.4 Acq/s 0.0 0.0 0.0 806BCA10 SYS$VM+74510 SYS$VM 00074510
FILSYS 0.4 0.4 Acq/s 0.0 0.0 0.0 8082CD10 F11BXQP+2EF10 F11BXQP 0002EF10
FILSYS 0.4 0.4 Rel/s 0.0 0.0 0.0 8082CD80 F11BXQP+2EF80 F11BXQP 0002EF80
FILSYS 0.3 0.3 Acq/s 0.0 0.0 0.0 80841CF0 F11BXQP+43EF0 F11BXQP 00043EF0
FILSYS 0.3 0.3 Rel/s 0.0 0.0 0.0 808426D0 F11BXQP+448D0 F11BXQP 000448D0
FILSYS 0.2 0.2 Acq/s 0.0 0.0 0.0 80819D20 F11BXQP+1BF20 F11BXQP 0001BF20
FILSYS 0.2 0.2 Rel/s 0.0 0.0 0.0 808196D0 F11BXQP+1B8D0 F11BXQP 0001B8D0
FILSYS 0.2 0.2 Acq/s 0.0 0.0 0.0 80819570 F11BXQP+1B770 F11BXQP 0001B770
FILSYS 0.2 0.2 Rel/s 0.0 0.0 0.0 80819E50 F11BXQP+1C050 F11BXQP 0001C050
Dynamic 50.8 50.8 Acq/s 0.0 0.0 0.0 810EC540 SYS$PEDRIVER+17F40 SYS$PEDRIVER 00017F40
Dynamic 50.8 50.8 Rel/s 0.0 0.0 0.0 810EC730 SYS$PEDRIVER+18130 SYS$PEDRIVER 00018130
Dynamic 48.7 48.6 Acq/s 0.0 0.0 0.0 810EF390 SYS$PEDRIVER+1AD90 SYS$PEDRIVER 0001AD90
Dynamic 48.6 48.6 Rel/s 0.0 0.0 0.0 810EF550 SYS$PEDRIVER+1AF50 SYS$PEDRIVER 0001AF50
Dynamic 23.0 23.0 Acq/s 0.0 0.0 0.0 805CEE30 PROCESS_MANAGEMENT+BD130 PROCESS_MANAGEMENT 000BD130
Dynamic 20.8 20.8 Rel/s 0.0 0.0 0.0 81348B00 TCPIP$INTERNET_SERVICES+1C80 TCPIP$INTERNET_SERVICES 0001C800
Dynamic 20.8 20.8 Acq/s 0.0 0.0 0.0 81348670 TCPIP$INTERNET_SERVICES+1C37 TCPIP$INTERNET_SERVICES 0001C370
Dynamic 16.8 16.8 Rel/s 0.0 0.0 0.0 805CF3F0 PROCESS_MANAGEMENT+BD6F0 PROCESS_MANAGEMENT 000BD6F0
Dynamic 16.6 16.6 Rel/s 0.0 0.0 0.0 80625DE0 PROCESS_MANAGEMENT+001140E0 PROCESS_MANAGEMENT 001140E0
Dynamic 16.6 3.6 Acq/s 0.0 0.0 13.0 80624980 SCH_STD$QAST_C+00300 PROCESS_MANAGEMENT 00112C80
Troubleshooting High MP Synch time (cont)
Press RETURN for more.
SDA>
Forklock Trace Information: (at 16-MAY-2007 13:13:42.55, trace time 00:00:42.377022)
------------------------------------------------------------------------------------
Total CPU ID
Forklock Events/sec 0 1
---------- ---------- ------------
IPL 06 0.1 0.1 0.0
IPL 08 103.3 54.5 48.8
TIMER 72.2 72.2 0.0
IOLOCK8 0.2 0.0 0.2
LCKMGR 0.3 0.3 0.0
---------- ------------
Totals 176.0 127.0 49.0
Press RETURN for more.
SDA>
Forklock Trace Information:
---------------------------
Average Minimum Maximum
Forklock Event/sec % Time Held (nsec) (nsec) (nsec) Fork PC
---------- --------- --------------- --------- --------- --------- ----------------------------------------
IPL 06 0.1 0.0 114915 43565 153208 80185C10 EXE$KP_DEALLOCATE_KPB_C+00680
---------- --------- ---------------
Totals 0.1 0.0
IPL 08 50.8 0.0 2496 1523 41758 810EC410 SYS$PEDRIVER+17E10
IPL 08 20.8 0.0 6701 2216 63843 81348580 TCPIP$INTERNET_SERVICES+1C280
IPL 08 16.3 0.0 6056 1819 22713 810C48C0 SYS$EW5700+1D9C0
IPL 08 6.5 0.0 1613 886 4989 80295630 LAN$COMPLETE_XMT_CSMACD_C+00E80
IPL 08 3.9 0.0 695 554 1149 813C2200 TCPIP$INTERNET_SERVICES+95F00
IPL 08 2.0 0.0 6275 4036 10754 8024D320 SYS$EIDRIVER+1AD20
IPL 08 1.2 0.0 5305 3159 10131 81120360 SYS$PEDRIVER+4BD60
IPL 08 1.1 0.0 11958 2144 46351 8151D030 TCPIP$INTERNET_SERVICES+001F0D3
IPL 08 0.4 0.0 9222 5728 15571 80FC1240 SYS$TTDRIVER+2AF40
IPL 08 0.3 0.0 6576 4236 11479 80FC02C0 SYS$TTDRIVER+29FC0
---------- --------- ---------------
Troubleshooting High MP Synch time (cont)
Press RETURN for more.
SDA>
Forklock Trace Information:
---------------------------
Average Minimum Maximum
Forklock Event/sec % Time Held (nsec) (nsec) (nsec) Fork PC
---------- --------- --------------- --------- --------- --------- ----------------------------------------
Totals 103.3 0.0
TIMER 72.2 0.1 17443 172 4664164 80143560 SYSTEM_PRIMITIVES_MIN+00123560
---------- --------- ---------------
Totals 72.2 0.1
IOLOCK8 0.2 0.0 2812 1237 5833 80297130 SYS$LAN_CSMACD+33B30
IOLOCK8 0.0 0.0 2555 2538 2573 80295630 LAN$COMPLETE_XMT_CSMACD_C+00E80
---------- --------- ---------------
Totals 0.2 0.0
LCKMGR 0.3 0.0 13346 2586 30121 80D45030 SYS$CLUSTER+3A630
---------- --------- ---------------
Totals 0.3 0.0
========== ========= ===============
Totals 176.0 0.2
Troubleshooting High MP Synch time (cont)
After looking at the data I wanted to know WHAT was accessing the SCHED spinlock.
Ran problem program again
Forced a crash
Analyzed that crash
SDA> set output hispin.out
SDA> spl show trace
$ search hispin.out pc
SDA> show process/id=pid from search
Troubleshooting High MP Synch time (cont)
SDA> set out delme_hispin.out
SDA> spl sho trace
CLASS8::MEHLHOP 21:36:33 SDA CPU=00:01:02.53 PF=1554 IO=3376 MEM=1257
CLASS8::MEHLHOP 21:36:41 SDA CPU=00:01:09.29 PF=1554 IO=3648 MEM=1257
CLASS8::MEHLHOP 21:38:11 SDA CPU=00:02:23.72 PF=1554 IO=6720 MEM=1257
CLASS8::MEHLHOP 21:39:17 SDA CPU=00:03:16.56 PF=1554 IO=9079 MEM=1257
Interrupt
CLASS8$ exit
CLASS8$ search delme_hispin.out 011A8CE0
20-MAY 21:29:35.000984 00 881C0A00 SCHED 011A8CE0 34E00430 Acquire FFFFFFFF.7D53CF40
20-MAY 21:29:35.000983 00 881C0A00 SCHED 011A8CE0 34E00430 Acquire FFFFFFFF.7D53CEF0
20-MAY 21:29:35.000983 00 881C0A00 SCHED 011A8CE0 34E00430 Acquire FFFFFFFF.7D53CEA0
20-MAY 21:29:35.000982 00 881C0A00 SCHED 011A8CE0 34E00430 Acquire FFFFFFFF.7D53CE50
20-MAY 21:29:35.000982 00 881C0A00 SCHED 011A8CE0 34E00430 Acquire FFFFFFFF.7D53CE00
20-MAY 21:29:35.000981 00 881C0A00 SCHED 011A8CE0 34E00430 Acquire FFFFFFFF.7D53CDB0
20-MAY 21:29:35.000981 00 881C0A00 SCHED 011A8CE0 34E00430 Acquire FFFFFFFF.7D53CD60
20-MAY 21:29:35.000980 00 881C0A00 SCHED 011A8CE0 34E00430 Acquire FFFFFFFF.7D53CD10
20-MAY 21:29:35.000979 00 881C0A00 SCHED 011A8CE0 34E00430 Acquire FFFFFFFF.7D53CCC0
20-MAY 21:29:35.000979 00 881C0A00 SCHED 011A8CE0 34E00430 Acquire FFFFFFFF.7D53CC70
20-MAY 21:29:35.000978 00 881C0A00 SCHED 011A8CE0 34E00430 Acquire FFFFFFFF.7D53CC20
20-MAY 21:29:35.000978 00 881C0A00 SCHED 011A8CE0 34E00430 Acquire FFFFFFFF.7D53CBD0
20-MAY 21:29:35.000977 00 881C0A00 SCHED 011A8CE0 34E00430 Acquire FFFFFFFF.7D53CB80
20-MAY 21:29:35.000977 00 881C0A00 SCHED 011A8CE0 34E00430 Acquire FFFFFFFF.7D53CB30
CLASS8$ an/cr sys$system
Troubleshooting High MP Synch time (cont)
SDA> sho proc/id=34E00430
Process index: 0030 Name: PSDC$DC_SERVER Extended PID: 34E00430
--------------------------------------------------------------------
Process status: 00140011 RES,PSWAPM,PHDRES,LOGIN
status2: 00000000
PCB address 89126980 JIB address 89122A40
PHD address 8DA92000 Swapfile disk address 00000000
KTB vector address 89126CA8 HWPCB address FFFFFFFF.8DA92080
Callback vector address 00000000 Termination mailbox 0034
Master internal PID 00010030 Subprocess count 0
Creator extended PID 00000000 Creator internal PID 00000000
Previous CPU Id 00000000 Current CPU Id 00000000
Previous ASNSEQ 0000000000000001 Previous ASN 0000000000FD6DEB
Initial process priority 15 # open files remaining 125/128
Delete pending count 0 Direct I/O count/limit 100/100
UIC [00001,000004] Buffered I/O count/limit 100/100
Abs time of last event 0134E8B6 BUFIO byte count/limit 261440/261440
# of threads 1 ASTs remaining 98/100
Swapped copy of LEFC0 00000000 Timer entries remaining 14/16
Swapped copy of LEFC1 00000000 Active page table count 0
Global cluster 2 pointer 00000000 Process WS page count 2385
Global cluster 3 pointer 00000000 Global WS page count 21
PCB Specific Spinlock 89127280 Subprocesses in job 0
Troubleshooting High MP Synch time (cont)
Press RETURN for more.
SDA>
Process index: 0030 Name: PSDC$DC_SERVER Extended PID: 34E00430
--------------------------------------------------------------------
Thread index: 0000
------------------
Current capabilities: System: 000C QUORUM,RUN
User: 0000.00000000
Permanent capabilities: System: 000C QUORUM,RUN
User: 0000.00000000
Current affinities: 00000000.00000000
Permanent affinities: 00000000.00000000
Thread status: 00140011
status2: 00000000
KTB address 89126980 HWPCB address FFFFFFFF.8DA92080
PKTA address 7FFEFF98 Callback vector address 00000000
Internal PID 00010030 Callback error 00000000
Extended PID 34E00430 Current CPU id 00000000
State HIB Flags 00000000
Base priority 15 Current priority 15
Waiting EF cluster 4 Event flag wait mask 0000000D
CPU since last quantum FFFFFFFE Mutex count 0
ASTs active NONE
SDA>
Troubleshooting High Alignment Faults
CLA
top related