Dm 1209operationsconsole PDF

Copyright IBM Corporation 2012 TrademarksInterpret IBM InfoSphere DataStage and QualityStage OperationsConsole metricsPage 1 of 31Interpret IBM InfoSphere DataStage and QualityStageOperations Console metricsA deep dive into the metrics and statistics presented in theOperations ConsoleArron Harden ([email protected])Senior Software EngineerIBMLen Greenwood ([email protected])DataStage Core ArchitectIBMGeoff McClean ([email protected])Senior Software DeveloperIBMJulian Vizor ([email protected])Senior Development ManagerIBMEric Jacobson ([email protected])Senior Software EngineerIBM13 September 2012This article looks into the metrics and statistics presented in the IBM InfoSphereDataStage and QualityStage Operations Console. This includes descriptions of how thevalues are generated (or where they are read from) and how the calculations are made. Whereappropriate, example SQL queries are shown to demonstrate how the information is read fromthe operations database.IntroductionThe IBM InfoSphere DataStage and QualityStage Operations Console is a web application thatallows the DataStage engine components of an InfoSphere Information Server installation to bemonitored in real time. This provides a complete view of all DataStage job runs on that system,developerWorks ibm.com/developerWorks/Interpret IBM InfoSphere DataStage and QualityStage OperationsConsole metricsPage 2 of 31both present time and historic. It also includes the monitoring of key operating system metrics suchas CPU usage, free memory and disk space.The following article describes in detail the metrics and properties displayed in the OperationsConsole, and provides information about where the data is collected from. It assumes the reader isfamiliar with the Operations Console and DataStage in general. The information presented here isrelated to InfoSphere Information Server version 8.7.Data sourcesThe information displayed in the Operations Console is read from the following sources.Metadata repository (XMETA) This is the design time metadata repository, used primarily bythe DataStage Designer to store table definitions, job designs, and so on.User registry Depending on how the system is configured, this can be an internal registrywhere all the user information is stored in the metadata repository, or it can be an externalregistry such as LDAP, where the user information is stored in an external directory system.DSEngine This is the run time engine and repository, used primarily by the DataStageDirector and jobs when they are run. It is also used to store the executable version of a jobdesign when it is compiled by the DataStage Designer.Operations database (DSODB) This is the operational repository that stores informationabout job runs. Once enabled, whenever a job is run, data about that run is stored here.The operations database uses a publicly documented open schema, and since it is an openschema, the relevant SQL queries that extracts the information is shown in this article (whereapplicable). Note that for clarity, only DB2 SQL queries have been shown, without paginationor sorting.Dates and timesAll dates and times displayed in the Operations Console are converted and displayed in thetime zone of the browser. For example, if the engine tier system time zone is UTC-0500, and thebrowser is UTC+0000, an event that happens on 3rd November 2011 at 4:00pm local time onthe engine is shown in the browser as occurring on 3rd November 2011 at 9:00pm. Similarly, if adifferent browser is used and is UTC+0500, that same event is shown on 4th November 2011 at2:00am on that browser.The home page of the Operations Console shows the time zone, including the UTC offset, for boththe engine tier system and the browser, shown as local in Table 1.Table 1. Time zonesGeneral propertiesEngine started: Oct 25, 2011 15:46:05 (local time zone). Engine is UTC+0000 (GMT),local is UTC+0000Operating System: Linux: 2.6.32.45-0.3-paeCPU: 2x Intel(R) Pentium(R)4 CPU 3.20GHzMemory: 2.8GB Physical - 2.01GB Virtualibm.com/developerWorks/ developerWorksInterpret IBM InfoSphere DataStage and QualityStage OperationsConsole metricsPage 3 of 31Within the operations database itself all timestamps are stored in UTC, regardless of the time zoneof the engine system.It is important to note that some timestamps are created using the clock on the engine tier systemand some are created using the clock on the services tier system. If these are different physicalsystems and the clocks are significantly out of synchronization, there may be discrepancies whenexamining and comparing timestamps.Operating system monitoringThe Operations Console displays information about various operating system metrics such asCPU usage, and free memory. The attributes associated with each of these metrics, and where theinformation is ultimately read from, is explained later in this article.Note that in the following SQL query examples, if the engine has been configured with remotenode information (via the ResourceNode property in DSODBConfig.cfg), the information read fromthe operations database can either be the engine itself (that is = ),or for a particular remote node.CPU total usage chartThe CPU total usage chart is displayed on the home page, and can also be optionally displayedon the job run details window and the activity page. For example, total CPU usage from the activitypage is shown in Figure 1.Figure 1. CPU total usage chartThe base information for total CPU usage is read from the operations database using the SQLquery shown in Listing 1.developerWorks ibm.com/developerWorks/Interpret IBM InfoSphere DataStage and QualityStage OperationsConsole metricsPage 4 of 31Listing 1. SQL example: CPU total usageSELECTX.CPUPCTIDLEAVG, X.STARTTIMESTAMP, X.ENDTIMESTAMPFROMDSODB.RESOURCEUSAGESYSTEM X,DSODB.HOST Y1,DSODB.HOST Y2WHERE(X.HOSTID = Y1.HOSTID AND Y1.HOSTNAME = ) AND(X.HEAD_HOSTID = Y2.HOSTID AND Y2.HOSTNAME = ) AND((X.STARTTIMESTAMP >= ANDX.STARTTIMESTAMP = ANDX.ENDTIMESTAMP ))The CPU idle percentage returned by the previous SQL query is converted to CPU usagepercentage by using the calculation 100% - CPU idle time. For this field, 100% represents thetotal possible across all processing units on the system. So for example if a system has 4 CPUs,and 1 of those CPUs was completely in use, the idle time would be 75% (and therefore the totalusage would be 25%).The information in the operations database returned by the previous query is written by theResMonApp and ResTrackApp services, which obtain it using the following methods.Linux - The CPU idle percentage is calculated using the idle entry in the CPU field in /proc/stat. This idle value is subtracted from the previously recorded value, divided by the sum of allthe other entries in the CPU field and multiplied by 100 to obtain the percentage value.AIX - The CPU idle percentage is calculated using the sum of the idle entries returned by theperfstat_cpu() system call. This idle value is subtracted from the previously recorded value,divided by the sum of all the other entries in the CPU field and multiplied by 100 to obtain thepercentage value.Solaris - The CPU idle percentage is calculated using the sum of all the idle entries in thecpu_stat modules in the kstat control structure returned by the kstat_*() system calls. Thisidle value is subtracted from the previously recorded value, divided by the sum of all the otherentries in the CPU field and multiplied by 100 to obtain the percentage value.Windows - The CPU idle percentage is calculated using the idle value returned by theWindows PerformanceCounter class with a counter class of Processor and counter name of %Idle Time.CPU system usage chartThe CPU system usage chart can optionally be displayed on the job run details window, as shownin Figure 2.ibm.com/developerWorks/ developerWorksInterpret IBM InfoSphere DataStage and QualityStage OperationsConsole metricsPage 5 of 31Figure 2. CPU system usage chartThe base information for system or privileged CPU usage is read from the operations databaseusing the SQL query shown in Listing 2.Listing 2. SQL example: CPU system usageSELECTX.CPUPCTPRIVILEGEDAVG, X.STARTTIMESTAMP, X.ENDTIMESTAMPFROMDSODB.RESOURCEUSAGESYSTEM X,DSODB.HOST Y1,DSODB.HOST Y2WHERE(X.HOSTID = Y1.HOSTID AND Y1.HOSTNAME = ) AND(X.HEAD_HOSTID = Y2.HOSTID AND Y2.HOSTNAME = ) AND((X.STARTTIMESTAMP >= ANDX.STARTTIMESTAMP = ANDX.ENDTIMESTAMP ))For CPU system usage percentage, 100% represents the total possible across all processing unitson the system, so for example if a system has 4 CPUs, and 1 of those CPUs was completely inuse by the operating system kernel, the system usage would be 25%.The information in the operations database returned by the above query is written by theResMonApp and ResTrackApp services, which obtain it using the following methods.Linux - The CPU privileged percentage is calculated using the privileged entry in the CPUfield in /proc/stat. This privileged value is subtracted from the previously recorded value,divided by the sum of all the other entries in the CPU field and multiplied by 100 to obtain thepercentage value.IX - The CPU privileged percentage is calculated using the sum of the privileged entriesreturned by the perfstat_cpu() system call. This privileged value is subtracted from thedeveloperWorks ibm.com/developerWorks/Interpret IBM InfoSphere DataStage and QualityStage OperationsConsole metricsPage 6 of 31previously recorded value, divided by the sum of all the other entries in the CPU field andmultiplied by 100 to obtain the percentage value.Solaris - The CPU privileged percentage is calculated using the sum of all the privilegedentries in the cpu_stat modules in the kstat control structure returned by the kstat_*() systemcalls. This privileged value is subtracted from the previously recorded value, divided by thesum of all the other entries in the CPU field and multiplied by 100 to obtain the percentagevalue.Windows - The CPU privileged percentage is calculated using the value returned by theWindows PerformanceCounter class with a counter class of Processor, and counter name of% Privileged Time.Memory - physical free chartThe free physical memory chart is displayed on the home page, and can also be optionallydisplayed on the job run details window and the activity page. For example, from the activity page,physical memory is the blue dashed line, as shown in Figure 3.Figure 3. Free physical memory chartThe base information for free physical memory is read from the operations database using theSQL query shown in Listing 3.Listing 3. SQL example: Free physical memorySELECTX.MEMFREEKBPHYSICALAVG, X.STARTTIMESTAMP, X.ENDTIMESTAMPFROMDSODB.RESOURCEUSAGESYSTEM X,DSODB.HOST Y1,DSODB.HOST Y2WHERE(X.HOSTID = Y1.HOSTID AND Y1.HOSTNAME = ) AND(X.HEAD_HOSTID = Y2.HOSTID AND Y2.HOSTNAME = ) AND((X.STARTTIMESTAMP >= ANDX.STARTTIMESTAMP = ANDX.ENDTIMESTAMP ))ibm.com/developerWorks/ developerWorksInterpret IBM InfoSphere DataStage and QualityStage OperationsConsole metricsPage 7 of 31The information in the operations database returned by the above query is written by theResMonApp and ResTrackApp services, which obtain it using the following methods.Linux - Physical free memory is calculated using PhysicalFree = MemFree + Cached, whereMemFree and Cached are fields from /proc/meminfo. This represents the physical memorythat could be made available without requiring anything to be swapped out to disk (thecontents of the cache component would be discarded).AIX - Physical free memory is calculated using PhysicalFree = (MemFree + Cached) *4k, where MemFree is the real_free value returned by the perfstat_memory_total call, andCached is the numperm value returned by the same call. This represents the physicalmemory that could be made available without requiring anything to be swapped out to disk(the contents of the cache component would be discarded).Solaris - Physical free memory is calculated using the value of the freemem entry in the kstatcontrol structure module vminfo returned by the kstat_*() system calls.Windows - Physical free memory is calculated using PhysicalFree = (MemFree / 1024),where MemFree is the value returned by the Windows PerformanceCounter class with acounter class of Memory, and counter name of Available Bytes.Memory - virtual free chartThe free virtual memory chart is displayed on the home page, and can also be optionally displayedon the job run details window and the activity page. For example, from the activity page (virtualmemory is the green solid line) as shown in Figure 4.Figure 4. Free virtual memory chartThe base information for free virtual memory is read from the operations database using the SQLquery shown in Listing 4.developerWorks ibm.com/developerWorks/Interpret IBM InfoSphere DataStage and QualityStage OperationsConsole metricsPage 8 of 31Listing 4. SQL example: Free virtual memorySELECTX.MEMFREEKBVIRTUALAVG, X.STARTTIMESTAMP, X.ENDTIMESTAMPFROMDSODB.RESOURCEUSAGESYSTEM X,DSODB.HOST Y1,DSODB.HOST Y2WHERE(X.HOSTID = Y1.HOSTID AND Y1.HOSTNAME = ) AND(X.HEAD_HOSTID = Y2.HOSTID AND Y2.HOSTNAME = ) AND((X.STARTTIMESTAMP >= ANDX.STARTTIMESTAMP = ANDX.ENDTIMESTAMP ))The information in the operations database returned by the above query is written by theResMonApp and ResTrackApp services, which obtain it using the following methods.Linux - Virtual free memory is read from the SwapFree field in /proc/meminfo.AIX - Virtual free memory is calculated using VirtualFree = PagesFree * 4k, wherePagesFree is the pgsp_free value returned by the perfstat_memory_total call.Solaris - Virtual free memory is calculated using the value of the swap_free entry in the kstatcontrol structure module vminfo returned by the kstat_*() system calls.Windows - Virtual free memory is calculated using VirtualFree = ((CommitLimit- Committed) / 1024), where CommitLimit is the value returned by the WindowsPerformanceCounter class with a counter class of Memory, and counter name of CommitLimit. Committed is the value returned by the Windows PerformanceCounter class with acounter class of Memory and counter name of Committed Bytes.Processes - count chartThe process count chart can optionally be displayed on the job run details window and the activitypage. For example, from the activity page, as shown in Figure 5.Figure 5. Free virtual memory chartThe base information for the number of processes (across the entire operating system - not justDataStage related) is read from the operations database using the SQL query shown in Listing 5.ibm.com/developerWorks/ developerWorksInterpret IBM InfoSphere DataStage and QualityStage OperationsConsole metricsPage 9 of 31Listing 5. SQL example: Processes countSELECTX.PROCNUMWAITINGAVG , X.PROCNUMSLEEPINGAVG , X.PROCNUMZOMBIEDAVG ,X.PROCNUMBLOCKEDAVG , X.PROCNUMRUNNINGAVG , X.STARTTIMESTAMP,X.ENDTIMESTAMPFROMDSODB.RESOURCEUSAGESYSTEM X,DSODB.HOST Y1,DSODB.HOST Y2WHERE(X.HOSTID = Y1.HOSTID AND Y1.HOSTNAME = ) AND(X.HEAD_HOSTID = Y2.HOSTID AND Y2.HOSTNAME = ) AND((X.STARTTIMESTAMP >= ANDX.STARTTIMESTAMP = ANDX.ENDTIMESTAMP ))The total process count is then calculated by summing the fields together. The information in theoperations database returned by the above query is written by the ResMonApp and ResTrackAppservices, which obtain it using the following methods.Linux - Process counts are computed by looping through all the process ID directoriesunder /proc. Within each process ID directory, the stat file is processed to find the state ofthe process. The total process count is calculated using ProcessCount = ProcRunning +ProcSleeping + ProcZombied + ProcBlocked, where ProcRunning is the sum of stat files witha state of "R", ProcSleeping is the sum of stat files with a state of "S" or "D", ProcZombied isthe sum of stat files with a state of "Z", and ProcBlocked is the sum of stat files with a state of"T" or "X".AIX - Process counts are computed by looping through all the process ID directoriesunder /proc. Within each process ID directory the psinfo file is processed to find the stateof the process. The total process count is calculated using ProcessCount = ProcRunning+ ProcSleeping + ProcZombied + ProcBlocked, where ProcRunning is the sum of psinfofiles with a state of "R" or "O", ProcSleeping is the sum of psinfo files with a state of "S",ProcZombied is the sum of psinfo files with a state of "Z", and ProcBlocked is the sum ofpsinfo files with a state of "T".Solaris - Root access is required to read /proc, so instead process counts are computedby issuing the command /usr/bin/ps -eo s and parsing the result. The total process countis calculated using ProcessCount = ProcRunning + ProcSleeping + ProcZombied +ProcBlocked, where ProcRunning is the sum of processes with a state of "R" or "O",ProcSleeping is the sum of processes with a state of "S", ProcZombied is the sum ofprocesses with a state of "Z", and ProcBlocked is the sum of processes with a state of "T".Windows - The process count is computed using the value returned by the WindowsPerformanceCounter class with a counter class of System, and counter name of Processes.File system - free space chartThe file system free space chart can optionally be displayed on the job run details window and theactivity page. For example, from the activity page as shown in Figure 6.developerWorks ibm.com/developerWorks/Interpret IBM InfoSphere DataStage and QualityStage OperationsConsole metricsPage 10 of 31Figure 6. Free disk spaceFree disk space information is only recorded if the DSODBConfig.cfg file has been set up with therelevant ResourceLocalFS and ResourceRemoveFS properties. The base information for the freedisk space is read from the operations database using the SQL query shown in Listing 6.Listing 6. SQL example: Free disk space countSELECTX.DISKPATHMONITORED, X.DISKTOTALKB, X.DISKFREEKBAVG, X.STARTTIMESTAMP, X.ENDTIMESTAMPFROMDSODB.RESOURCEUSAGEDISKS X,DSODB.HOST Y1,DSODB.HOST Y2WHERE(X.HOSTID = Y1.HOSTID AND Y1.HOSTNAME = ) AND(X.HEAD_HOSTID = Y2.HOSTID AND Y2.HOSTNAME = ) AND((X.STARTTIMESTAMP >= ANDX.STARTTIMESTAMP = ANDX.ENDTIMESTAMP ))The information in the operations database returned by the previous query is written by theResMonApp and ResTrackApp services, which obtain it using the following methods.Linux, AIX and Solaris - Free disk space information is computed using the disk block sizeand available block count returned by the statvfs64() system call. Specifically the block size isread from the f_frsize element and available block count is read from the f_bavail element.Windows - Free disk space information is computed using the disk block size and availableblock count returned by the GetDiskFreeSpace() system call. Specifically the block size isread from the lpBytesPerSector return argument and available block count is calculated bymultiplying the lpNumberOfFreeClusters return argument by the lpSectorsPerCluster returnargument.General propertiesThe general properties on the home page show the latest recorded information about the currentlyselected engine system, as shown in Table 2.ibm.com/developerWorks/ developerWorksInterpret IBM InfoSphere DataStage and QualityStage OperationsConsole metricsPage 11 of 31Table 2. Time zonesGeneral propertiesEngine started: Oct 24, 2011 15:46:05 (local time zone). Engine is UTC+0100 (BST),local is UTC+0100Operating System: Linux: 2.6.32.45-0.3-paeCPU: 2x Intel(R) Pentium(R)4 CPU 3.20GHzMemory: 2.8GB Physical - 2.01GB VirtualThe engine start time is read by calling the engine system through ASBAgent and using aUniVerse Basic system call in DSEngine itself, for example:engineStartTime = SYSTEM(12001)The local UTC offset is read by running the following JavaScript on the local web browser, forexample:offset = new Date().getTimezoneOffset();The remainder of this information is read from the operations database using the following SQLquery. The sub-select is required because the operations database records all historical enginesystem information, but only requires the latest, hence the MAX(LASTCHECKEDTIMESTAMP), as shownin Listing 7.Listing 7. SQL Example: Engine propertiesSELECTHD.CREATEDTIMESTAMP, HD.LASTCHECKEDTIMESTAMP, HD.CPUMODEL, HD.NUMCPUS,HD.PHYSICALMEMORYKB, HD.PLATFORMNAME, HD.PLATFORMVERSION,HD.UTCOFFSETMINS, HD.VIRTUALMEMORYKB, HD.TIMEZONENAMEFROMDSODB.HOSTDETAIL HD,DSODB.HOST H1,DSODB.HOST H2,(SELECTMAX(LASTCHECKEDTIMESTAMP) AS MAXTIMESTAMPFROMDSODB.HOSTDETAIL HD, DSODB.HOST H1, DSODB.HOST H2WHEREHD.HOSTID = H1.HOSTID ANDH1.HOSTNAME = ANDHD.HEAD_HOSTID = H2.HOSTID ANDH2.HOSTNAME = ) MWHEREHD.LASTCHECKEDTIMESTAMP = M.MAXTIMESTAMP ANDHD.HOSTID = H1.HOSTID ANDH1.HOSTNAME = ANDHD.HEAD_HOSTID = H2.HOSTID ANDH2.HOSTNAME = The information in the operations database returned by the previous query is written by theResMonApp and ResTrackApp services, which obtain it using the following methods.developerWorks ibm.com/developerWorks/Interpret IBM InfoSphere DataStage and QualityStage OperationsConsole metricsPage 12 of 31LinuxCPUMODEL and NUMCPUS ultimately come from the model name attribute in /proc/cpuinfo.PHYSICALMEMORYKB is read from the MemTotal field in /proc/meminfo.VIRTUALMEMORYKB is read from the SwapTotal field in /proc/meminfo.PLATFORMNAME and PLATFORMVERSION are read from /proc/version.TIMEZONENAME and UTCOFFSETMINS are read from the tm_zone and tm_gmtoffrespectively returned by the localtime() system call.AIXCPUMODEL and NUMCPUS ultimately come from the processor information returned by theperfstat_cpu_total () system call.PHYSICALMEMORYKB is calculated by multiplying the real_total value returned by theperfstat_memory_total() system call by the page size of 4.VIRTUALMEMORYKB is calculated by multiplying the pgsp_total value returned by theperfstat_memory_total() system call by the page size of 4.PLATFORMNAME and PLATFORMVERSION are read using the sysname and version valuesrespectively returned by the uname() system call.TIMEZONENAME is read from the "TZ" environment variable.UTCOFFSETMINS is calculated using the tz_minuteswest value returned by thegettimeofday() system call.SolarisCPUMODEL is read by parsing the information returned by the /usr/sbin/psrinfo -v command.NUMCPUS is read from the cpu_count value returned by thesysconf(_SC_NPROCESSORS_CONF) system call.PHYSICALMEMORYKB is calculated by multiplying the Memory value returned by the /usr/sbin/prtconf command by 1024.VIRTUALMEMORYKB is calculated by multiplying the ani_max value returned by theswapctl() system call by the system page size.PLATFORMNAME and PLATFORMVERSION are read using the values returned by thesysinfo(SI_SYSNAME,...) and sysinfo(SI_RELEASE,...) system calls.TIMEZONENAME is read from the "TZ" environment variable.UTCOFFSETMINS is calculated by computing the difference of the times returned by thelocaltime() and gmtime() system calls.WindowsCPUMODEL is read from the HKEY_LOCAL_MACHINE\\HARDWARE\\DESCRIPTION\\System\CentralProcessor\\0\ \ProcessorNameString registry key.NUMCPUS is read from the value of the dwNumberOfProcessors element returned by theGetSystemInfo() system call.PHYSICALMEMORYKB is calculated by dividing the ullTotalPhys value returned by theGlobalMemoryStatusEx() system call by 1024.ibm.com/developerWorks/ developerWorksInterpret IBM InfoSphere DataStage and QualityStage OperationsConsole metricsPage 13 of 31VIRTUALMEMORYKB is calculated by dividing the value returned by the WindowsPerformanceCounter class with a counter class of Memory and counter name of Commit Limitby 1024.PLATFORMNAME is always set to Windows.PLATFORMVERSION is calculated by looking at the values for dwMajorVersion,dwMinorVersion, and wProductType returned by the GetVersionEx() system call.TIMEZONENAME is read from the value returned by the _get_tzname() system call.UTCOFFSETMINS is calculated by using the value returned by the _get_timezone() systemcall and adding the value returned by the _get_dstbias() system call.DataStage engine monitoringThe Operations Console displays information about various DataStage engine metrics such as jobrun counts and service status. The attributes associated with each of these metrics, and where theinformation is read from, is explained as follows.Job runs chartThe job runs chart is displayed on the home page, and can also be optionally displayed the activitypage as well. For example, from the home page, as shown in Figure 7.Figure 7. Job runs chartIt shows the number of job runs that were active over a given period of time for the currentlyselected engine system. This is defined as any job run that satisfies one of the following criteria.Started in the time periodEnded in the time periodRunning throughout the time periodThe base information is read from the operations database using the SQL query (to keep the SQLquery simple, project filtering is not included here) as shown in Listing 8.developerWorks ibm.com/developerWorks/Interpret IBM InfoSphere DataStage and QualityStage OperationsConsole metricsPage 14 of 31Listing 8. SQL example: Job runs activitySELECTX.RUNMAJORSTATUS, X.RUNENDTIMESTAMP, X.RUNSTARTTIMESTAMPFROMDSODB.JOBRUN X, DSODB.JOBEXEC Y, DSODB.HOST ZWHEREX.JOBID = Y.JOBID ANDY.HOSTID = Z.HOSTID ANDZ.HOSTNAME = AND((X.CREATIONTIMESTAMP >= ANDX.CREATIONTIMESTAMP = ANDX.RUNENDTIMESTAMP