Sybase Statistics Demystified - Software Gems - Database ... GEM Documents/Syba… · Sybase Statistics Demystiﬁed Sybase Statistics Demystiﬁed/1 Introduction/1 of 16 Derek Asirvadem

Sybase Statistics Demystified

Derek Asirvadem • V2.1 • DRAFT • 20 Nov 11Sybase Statistics Demystified/1 Introduction/1 of 16

1 Introduction

Copyright © 1993-2011 Software Gems Pty Ltd

There are three types of Stats that Sybase can capture:

Of course Sybase actually captures only those Stats that you explicitly or implicitly request via the various flavours of UPDATE STATS command. That is where much of the confusion lies, and that is explained later.There are three kinds of what may be loosely called 'statistics' stored in the catalogue: • Column & ColumnGroup Stats are stored in sysstatistics; that is what is usually meant when the term Statistics is used, and that is the focus of this document• Index metrics (with two exceptions) are stored in systabstats. Some of these values are updated only by UPDATE STATISTICS and that is included.• Query metrics are stored in sysquerymetrics; that is excluded.

Also known asFor Each Sybase Can StoreColumn Density

ColumnGroup DensityColumn Histogram

Column StatsColumn

ColumnGroup ColumnGroup Stats

The output of the optdiag utility is used here, because most administrators are familiar with it. It is single-object oriented, or vertical; therefore, it is hard to navigate and follow, and voluminous.

2.1 Statistic Type

Statistics are very important. They form the basis of decisions that the Optimiser makes, when determining the Query Plan, and thus affect the performance of every query. The new Optimiser in Sybase ASE 15 is much more sensitive to Stats issues. More important, it uses Stats in many more places, to make many more decisions than before. Therefore it is now more important than ever, to understand Stats, and to implement regular UPDATE STATS as an integral part of database maintenance. The key criteria of Stats are:• they must be accurate, up-to-date, identifying the current content of the tables and columns• they must be relevant: maintaining Stats that are rarely used lengthens the maintenance window; missing

Stats causes the Optimiser to make incorrect decisions, which typically affect performance• they must be 'deep' enough to be useful• the decreasing size of the maintenance windowThis document is intended to assist any capable Administrator in covering all these requirements, efficiently. The Sybase manuals are poor, but in the area of Stats, they are abysmal; this document intends to overcome that obstacle. It is divided roughly into five parts:• which every DBA should become thoroughly familiar with• required for anyone coding their own reports, to inspect and manage their Stats• provides a detailed explanation of each flavour of the command• to work towards, rather than simple instructions, in order to produce an efficient and

relevant Stats updating in the shortest maintenance window (Customers only)•

Not so much for marketing purposes, but to identify how to structure your own reports, to obtain maximum benefit

Statistics for column: "Component"Last update of column statistics: Aug 11 2011 11:08:31:346AM

Range cell density: 0.0000000000000000 Total density: 0.1025191733659713 Range selectivity: default used (0.33) In between selectivity: default used (0.25) Unique range values: 0.0000000000000000 Unique total values: 0.0833333333333333 Average column width: default used (3.00)

2.2 Column Stats

Histogram for column: "Component"Column datatype: char(3)Requested step count: 20Actual step count: 24Sampling Percent: 10Out of range Histogram Adjustment is ON.

Step Weight Value

1 0.00000000 < "Cac" 2 0.15133531 = "Cac" 3 0.00000000 < "Ctx" 4 0.08902077 = "Ctx" 5 0.00000000 < "Dsk" 6 0.04154303 = "Dsk" 7 0.00000000 < "Eng" 8 0.10089020 = "Eng" 9 0.00000000 < "Hkp" 10 0.08011869 = "Hkp" 11 0.00000000 < "Log" 12 0.13946587 = "Log" 13 0.00000000 < "Net" 14 0.05044510 = "Net" 15 0.00000000 < "Upr" 16 0.06231454 = "Upr" 17 0.00000000 < "Usy" 18 0.02670623 = "Usy" 19 0.00000000 < "Uth" 20 0.03560831 = "Uth" 21 0.00000000 < "Uvm" 22 0.09198813 = "Uvm" 23 0.00000000 < "Xct" 24 0.13056380 = "Xct"

2.3 Histogram

For each Column, ASE can store density, basic data distribution, and selectivity Stats.

may have

For each Column, ASE can store a Histogram.

2 Concepts

More, later.

Colour is used consistently throughout the document.

Continued ...

ConceptsCatalogueUPDATE STATS

Goals

Help Statistics & Help Histogram

http://www.softwaregems.com.au



Derek Asirvadem • V2.1 • DRAFT • 20 Nov 11Sybase Statistics Demystified/2 Concepts /2 of 16

2 Concepts


Stats can be rendered for Indices. Where an Index consists of many Columns, as is normal for Relational databases, Stats can be rendered for: the set of Columns, taken together; and for each step in the set. For example, where an Index such as:

is implemented, the columns and sequence are chosen purposely, on the basis that queries using:ServerServer, ComponentServer, Component, MetricServer, Component, Metric, DateTime

are expected. These are the Steps. Of the four steps, the first step is of course a single column (Column Stats and a Histogram are possible), and the next three steps are ColumnGroups (ColumnGroup Stats are possible). There are no Histograms per ColumnGroup, because they are more relevant at the Column, not ColumnGroup, level; ColumnGroup Densities are deemed adequate; and additional Histograms would require even more time to build. For the purpose of understanding, as a starting point, take it that a ColumnGroup is an Index.

2.5 ColumnGroup/Not IndexedAs Sybase progressed, two things were realised:a. that Stats should not be tightly related to Indices (which may be dropped and recreated, or replaced,

whereas Columns and their Stats are more permanent, and should remain, even if the related Index is not recreated)

b. ColumnGroups that are not Indexed are used in queries, and Stats are required for them.Therefore, for decades, Indices have not been tightly related to Stats, and ColumnGroups are supported either with an index that identifies them, or without. There is a relation between Columns (for which Stats are collected, and a Histogram is constructed) and Indices, but it is fluid, not a direct reference via indid.It is important to understand and use ColumnGroups (which are not part of Index ColumnGroup Steps; and not members of an Index), and to set up Stats for them. Any group of Columns that are frequently used in queries can be identified to ASE as a ColumnGroup, and Stats can be captured for them.Therefore, ColumnGroups are either implicitly derived from Indices or explicitly declared.

Statistics for column group: "Value", "Metric"Last update of column statistics: Aug 12 2011 8:51:12:780PM


2.5.1 ColumnGroup Stats/Not IndexedFor each second and subsequent Step in an Indexed ColumnGroup, Sybase ASE can store density and selectivity Stats. The first Step is covered as a single Column.

2.4 ColumnGroup

Statistics for column group: "Server", "Component"Last update of column statistics: Aug 12 2011 7:14:13:426PM


Statistics for column group: "Server", "Component", "Metric"Last update of column statistics: Aug 12 2011 7:14:13:426PM


2.4.1 ColumnGroup Stats

Statistics for column group: "Server", "Component", "Metric", "DateTime"Last update of column statistics: Aug 12 2011 7:14:13:426PM


• Actually, there are a few more Column and ColumnGroup Stats, but we will leave that for now• Index metrics and derived stats are not shown here, as they are not really Stats; they belong with the

Index.

U_ComponentMetric ( Server, Component, Metric, DateTime )

Also known asFor Each Sybase Can StoreColumn Density

ColumnGroup DensityColumn Histogram

Column StatsColumn

ColumnGroup ColumnGroup Stats

For each second and subsequent Step in any identified ColumnGroup, Sybase ASE can store density and selectivity Stats. The first Step is covered as a single Column.

Continued ...




Derek Asirvadem • V2.1 • DRAFT • 20 Nov 11Sybase Statistics Demystified/2 Concepts/3 of 16

2 Concepts


The Software Gems Help Histogram report may be a little clearer: • The Cells, Weight, Operator and Values remain unchanged, so that the

report can be compared against optdiag without issue• Range Cells are unchanged• Frequency Cells, including the Null Cell, remain empty (as per storage in

sysstatistics), and are thus differentiated. Missing Frequency Cells are more obvious.

• the special Weights (or ratios) of 1.0 and 0.0, have their trailing zeroes truncated, allowing non-zero Weights to be read more easily

2.6.3 Help Histogram

2.6.1 optdiag

2.6.2 Help Statistic

Table owner: "dbo"Table name: "SM_ComponentValue"

Statistics for column: "DateTime"Last update of column statistics: Aug 11 2011 11:08:31:346AM


Histogram for column: "DateTime"Column datatype: smalldatetimeRequested step count: 20Actual step count: 28Sampling Percent: 10Out of range Histogram Adjustment is ON.

Step Weight Value

1 0.00000000 < "Dec 21 2010 5:00AM" 2 0.12462908 = "Dec 21 2010 5:00AM" 3 0.00000000 < "Dec 21 2010 12:00PM" 4 0.12462908 = "Dec 21 2010 12:00PM" 5 0.00000000 < "Dec 21 2010 6:00PM" 6 0.06528190 = "Dec 21 2010 6:00PM" 7 0.00000000 < "Dec 21 2010 10:00PM" 8 0.06528190 = "Dec 21 2010 10:00PM" 9 0.00000000 < "Feb 7 2011 3:00AM" 10 0.12462908 = "Feb 7 2011 3:00AM" 11 0.00000000 < "Feb 7 2011 4:00AM" 12 0.04747774 = "Feb 7 2011 4:00AM" 13 0.01186944 <= "Feb 7 2011 5:00AM" 14 0.00000000 < "Feb 7 2011 9:00AM" 15 0.06231454 = "Feb 7 2011 9:00AM" 16 0.00000000 < "Feb 7 2011 11:00AM" 17 0.12462908 = "Feb 7 2011 11:00AM" 18 0.00000000 < "Feb 7 2011 3:00PM" 19 0.01780416 = "Feb 7 2011 3:00PM" 20 0.00000000 < "Feb 7 2011 4:00PM" 21 0.04154303 = "Feb 7 2011 4:00PM" 22 0.00000000 < "Feb 7 2011 6:00PM" 23 0.06528190 = "Feb 7 2011 6:00PM" 24 0.01186944 <= "Feb 14 2011 11:00AM" 25 0.00000000 < "Feb 14 2011 12:00PM" 26 0.06231454 = "Feb 14 2011 12:00PM" 27 0.00000000 < "Feb 14 2011 1:00PM" 28 0.05044510 = "Feb 14 2011 1:00PM"

All Stats, for both ColumnGroups and Columns, are reported in the Software Gems Help Statistic report, horizontally, and in full context of each other. That report is expanded in section [10].

2.3.1 Cell

The Histogram stores detailed information re the actual data content (values) of a Column, and its distribution. That means DataType and length are an issue when storing Histograms, and the number of Cells (identified by the USING Cell VALUES parameter) is critical. The smallest number of functional, relevant cells should be determined and maintained: too many cells results in slow UPDATE STATS performance, with no gain in query performance; too few results in inadequate Stats, with a loss in performance.

There are three types of Cells. They may not be easy to differentiate in the optdiag report: • Range Cells are the most common, they are depicted

with a "<=" operator• Frequency Cells are depicted with "<" and refer the

the next Value (which is not stored, but optdiag replicates it in the report)

• The first Cell is always the Null Cell, a Frequency Cell for Null (representing the Null Values in the Column; it is always present, regardless of whether the Column allows Nulls).

• Both Range Cells and Frequency Cells are required for important Columns in large tables

Unfortunately, Cell is also known as Step; however this document will use the correct term.

This Sybase utility reports Stats on Columns, ColumnGroups and Histograms.

Table/ColumnGroup/Column Grp Index/Notice Un M Upd Updated Req Cell Hist Delta DensityRange DensityTotal Uniquens ValueUnique ValueRange ValueTotal

SM_ComponentValue [2] [2] 441 16 5737

DateTime 4 (UC_PK)+(1) 3 3 11/08/11 11:09 20 28 2 0.00 0.0118694362 0.0898044361 0.00261 15 0.0666666667

Table Column Step Weight Op Value

SM_ComponentValue DateTime 1 0.0 <

2 0.1246290803 = 10/12/21 05:00

3 0.0 <

4 0.1246290803 = 10/12/21 12:00

5 0.0 <

6 0.0652818978 = 10/12/21 18:00

7 0.0 <

8 0.0652818978 = 10/12/21 22:00

9 0.0 <

10 0.1246290803 = 11/02/07 03:00

11 0.0 <

12 0.0474777445 = 11/02/07 04:00

13 0.0118694361 <= 11/02/07 05:00

14 0.0 <

15 0.0623145401 = 11/02/07 09:00

16 0.0 <

17 0.1246290803 = 11/02/07 11:00

18 0.0 <

19 0.0178041551 = 11/02/07 15:00

20 0.0 <

21 0.0415430255 = 11/02/07 16:00

22 0.0 <

23 0.0652818978 = 11/02/07 18:00

24 0.0118694361 <= 11/02/14 11:00

25 0.0 <

26 0.0623145401 = 11/02/14 12:00

27 0.0 <

28 0.0504451022 = 11/02/14 13:00

Continued ...




Derek Asirvadem • V2.1 • DRAFT • 20 Nov 11Sybase Statistics Demystified/3 Catalogue/4 of 16

3 Catalogue


Describing this internal table as "denormalised" is incorrect: the data is beautifully Normalised. However, many types of data, or rows, are stored in the one table. It is therefore packed or folded.• That is undesirable in a Relational database, but in an internal catalogue of a server, particularly one that

contains a large amount of varied data (note DataType, actual data values, and volume of Cells), it is quite desirable.

• The Normalised version would be five tables, plus one for each DataType. • However, implementing colidarray as a variable length column, and doing so in an index, is a crime

against Nature which cannot be forgiven.There are seven types of data, or tables in the Normal sense, folded into one table.Notice there is no relation to sysindexes, despite the presence of the indid column. It is a relic from the early days, when Stats were directly related to indices, it is always zero. Likewise, statid is not used• Ensure your code is qualified to guard against breakage due to possible changes in the future:SELECTS ... WHERE indid = 0 AND statid = 0.

c79

...

moddate

c0

usedcount

formatid

partitionid

sequence

coldarray

id

Performing anything more than perfunctory administration requires some understanding of the Sybase Catalogue. It is invaluable in understanding the server, and appreciating why tasks must be performed in certain ways. The Statistics catalogue is no exception. Demystifying Statistics is equivalent to demystifying the Statistics catalogue.

3.1 The Table 3.2 Content/EntityThe easiest, and the best method of understanding data and how it relates to other data, is to model it as a Relational Database. The seven types of data, or distinct tables, stored in sysstatistics, and their relations to each other, are:

• Columns must exist for a table, they are the central focus of Stats• ColumnGroups, however, may not exist, since there are people who do not implement Stats for

Indices or Columns that are queried together. If they do exist, they are made up of Columns.• DataChange Counters per Column are just that, not really Stats, in a convenient storage location

(they will not be expanded further)• Histogram Cell Weight & Value are stored per Column. • The new Missing Stats per Column or ColumnGroup constitute logging of events, stored in the form

of Stats.• Like ColumnGroups, Indices may not exist, but if they do, they are made up of Columns, and a

Column may exist in more than one Index. If Stats are captured for them, it will be in the form of a ColumnGroup. That is the fluid relation between Column Stats and Indices, via ColumnGroups.

sysstatistics

c79...

statid

moddate

c0usedcount

formatid

ststatus

partitionid

sequence

colidarray

indidid

sysobjects

syscolumns

syspartitions

Is Enumerated As

ColumnStats

HistogramCell Weight

HistogramCell Value

ColumnGroupStats

Table

DataChangeCounter

Index

Familiarity with the Sybase catalogue will be helpful. Familiarity wih our Sybase 15 Data Model (which contains important explanations and additional facilities) will assist even further.

Continued ...

sysindexes




Derek Asirvadem • V2.1 • DRAFT • 20 Nov 11Sybase Statistics Demystified/3 Catalogue /5 of 16

3 Catalogue


Row (PK)

Column100

ColumnGroup100

3.3 Content/RowAs we know, every row in a Relational Database must be unique, and uniquely identified by its Primary Key. The Primary Key Columns identify the row content:

It may be easier to understand in a relational diagram. Let's look at a Column first: colidarray consists of a single entry or an array of [1], a single colid, the PK of its parent, syscolumns.

Now let's tackle the ColumnGroup. Ensure that you understand 2.4 and 2.5 ColumnGroup. A Column may be a member of zero, one, or more ColumnGroups:

Just as the colid identifies a Column (within a table), the colidarray identifies an array of colids (within one table). The original Engineers have (brilliantly) used the Relational concept of Identifiers: the Identifier of a ColumnGroup is the list of Identifiers of its component Columns. They reference each other without additional Identifiers, and eliminate the need for an Associative table (or, it exists, and it is folded into sysstatistics, without repetition). (Again, that is acceptable in a server catalogue, but not in a database.)

This also, is easier to understand in a relational diagram. We can immediately identify:• Column rows (colidarray of 1 entry)• a ColumnGroup row (colidarray > 1 entry)• the members of ColumnGroup (contained in the ColumnGroup entry itself)

3.3.1 Column

3.3.2 ColumnGroup

100 Col_A

104 Col_A

102 Col_A

Table_T

Column Stats: Density & basic Distibution

Histogram Cell Value

Histogram Cell Weight

100 Col_A,Col_B

100 Col_A 100 Col_B

104 Col_A

102 Col_A

104 Col_A

102 Col_A

Table_T

ColumnGroup Stats: Density & basic distribution

• Together the Cell Weight and Cell Value comprise the Histogram, containing full data distribution• The depth of the Histogram (no of Cells) is as per default (20) or histogram tuning factor, the settable default orwhatever was previously set for the table via USING Cell VALUES

Warning• I have seen at least four different methods of 'interpreting' colidarray: every one of them uses

procedural processing (very slow for a set-processing engine) quite unnecessarily. Additionally, they are all wrong, in that every one of them additionally takes a laborious approach to unpacking colidarray: the task is simple, if one understands that it is simply an array of words (two bytes), each word being a single colid. Manually handling endian issues is stupefying, unnecessary in SQL.

• All our utilities use pure set-processing, and execute at server speeds, not at developer speeds.

The seven types of rows in sysstatistics, identified by formatid and colidarray, are as follows:

Where there are more then 80 Cells, sequence identifies rows in the series. Note: Weight & Value are stored in separate vectors: sequence and usedcount of Cell Weights do not match that of Cell Values

sequence

Type of Stats row (together with colidarray), as per the table below.formatid

>1 entry (ColumnGroup) One Foreign Key per entry (Column), referring to syscolumns

1 entry (Column) Foreign Key referring to syscolumnscolidarray

Partition level Stats; Foreign Key referring to syspartitionsnon-zero

zero Table level Statspartitionid

Foreign Key referring to sysobjectsid

ContentColumn

1 entry Histogram Cell Weight1 entry Histogram Cell Value

Column Stats1 entry> 1 entry ColumnGroup Stats

104

102

100

100

1 entry DataChange Counter108

ColumnGroup Missing

110

> 1 entryColumn Missing

110

1 entry

colidarray c0...c79formatid

Continued ...

Type




Derek Asirvadem • V2.1 • DRAFT • 20 Nov 11Sybase Statistics Demystified/4 Example/6 of 16

4 Example


Table & IndicesSM_ComponentValue

ValueMetricComponentDateTimeServer

4.1 Table

The rows in the Sybase catalogue for this table which are relevant to UPDATE STATISTICS operations, are as follows. The relations between Columns and ColumnGroups, and to indices are also shown:

The different UPDATE STATISTICS operations, and how they affect the different catalogue rows, are illustrated on the following pages.

A complete example may be useful to assist understanding. We will use a simple, Data Warehouse (Sixth Normal Form) table, with two indices. The table is APL, therefore the Clustered Index is real, and Range Queries are supported.

4.2 Catalogue

UC_PK ( Server, DateTime, Component, Metric )


sysstatistics

Column Histogram Cell Value & Weight

Column Stats Density & basic Distibution

ColumnGroup Stats Density & basic distribution

Grp=5(UC_PK)+(1)

Grp=4(UC_PK)+(1)

Grp=4(UC_PK)+(1)

Grp=1Grp=6UC_PK+[1]

100 Server,Component

100 Server,Component,Metric

100 Server,Component,Metric,DateTime

100 Value,Metric

100 Server,DateTime

100 Server,DateTime,Component

100 Server,DateTime,Component,Metric

100 Server 100 DateTime 100 Metric100 Component 100 Value

V102 Server

104 Server W102 Value V104 Value W

102 Component V104 Component W

102 Metric V104 Metric W

102 DateTime V104 DateTime W

ColumnGroups Indices

Continued ...

sysindexes

U_ComponentMetric Server,Component,Metric,DateTime

UC_PK Server,DateTime,Component,Metric

systabstats UC_PK

U_ComponentMetric

Index Metrics Height; RowSize; Row, Page, Extent & Cluster Ratio counts

Index Definition Density & basic distribution




Derek Asirvadem • V2.1 • DRAFT • 20 Nov 11Sybase Statistics Demystified/5 Column Stats/7 of 16

5 Column Stats


100 Value,Metric

100 Metric

102 Metric V104 Metric W


100 Server,DateTime



100 Server 100 DateTime 100 Component

V102 Server

104 Server W102 Component V104 Component W

102 DateTime V104 DateTime W



UPDATE STATISTICS Table (Column)

5

UPDATE STATISTICS SM_ComponentValue (Value)

100 Value

102 Value V104 Value W

Full Stats (Densities and a Histogram) can be created for any Column.

While this document provides information that is absent from the manuals, it is not intended to replace the manuals; for full syntax and options, refer to the manual.

Continued ...

SM_ComponentValue

1 Creates/updates Column Stats for the Column2 Creates/updates Histogram for the Column

If an Index containing the column exists, it is used, otherwise the Clustered Index or Heap is read

IsolationUPDATE STATISTICS uses:• ISOLATION LEVEL 1 for APL• ISOLATION LEVEL 0 for DPL/DRL.




Derek Asirvadem • V2.1 • DRAFT • 20 Nov 11Sybase Statistics Demystified/6 ColumnGroup Stats/8 of 16

6 ColumnGroup Stats


100 Metric

102 Metric C104 Metric C

C102 Server




100 Server,DateTime



100 Server 100 DateTime 100 Component

104 Server C102 DateTime C 102 Component C

104 Component C104 DateTime C

6

UPDATE STATISTICS SM_ComponentValue (Value, Metric)

UPDATE STATISTICS Table (Col_1, Col_2, ...)

100 Value,Metric

100 Value

102 Value C104 Value C

Any list of columns identified together and given to UPDATE STATISTICS in one command, form a ColumnGroup.

Continued ...

1 Creates/updates ColumnGroup Stats for the second and subequent Step in the group

2 Creates/updates Column Stats for the leading Column

3 Creates/updates Histogram for the leading Column

Record Filing Systems• These tend to have single-column indices, and many more of them• All except the Primary Key are non-unique• The value of ColumnGroups, and Steps, is lost: they must be added manually

SM_ComponentValue

If an Index containing the column list exists, it is used, otherwise the Clustered Index or Heap is read

Resource UseThe leading column in an Index (if one can be used) is already sorted. Non-leading columns need to be sorted: that requires a worktable in tempdb, and sort buffers in the procedure cache.




Derek Asirvadem • V2.1 • DRAFT • 20 Nov 11Sybase Statistics Demystified/7 Index Stats/9 of 16

7 Index Stats





100 Value,Metric

104 Value W102 Value V

100 Value

7

UPDATE INDEX STATISTICS SM_ComponentValue UC_PK

Second IndexNote that, in this example, at this point, there is no value in updating Stats for the second index, issuing: UPDATE INDEX STATISTICS SM_ComponentValue U_ComponentMetricbecause the Stats for all the Columns in that index have just been freshly updated. Instead, use: UPDATE STATISTICS SM_ComponentValue (Server, Component, Metric, Datetime)Where performance and the maintenance window are a consideration, particularly when the number of Cells is large, a redundant duplication of effort with no gain, is to be avoided.

UPDATE INDEX STATISTICS Table [Index]

100 Server,DateTime



100 Server 100 DateTime 100 Metric100 Component

V102 Server

104 Server W102 DateTime V 102 Metric V102 Component V

104 Component W 104 Metric W104 DateTime W

• Takes all Columns in the Index as a ColumnGroup , providing full Stats for each Column• If the Index is omitted, all indices for the table are processed, without doubling up at the Column level

Consideration• The leading column in an Index is already sorted. A Sort is required for the second and subsequent Columns in the group: that requires a worktable in tempdb, and sort buffers in the procedure cache.• WITH SAMPLING = Percent

• applies to non-leading and unindexed Column• does not update Column & ColumnGroup Stats (updates Histogram only)• accesses the base table (leaf-level of CI for APL; Heap for DPL/DRL)


Continued ...


2 Creates/updates Column Stats for each Column3 Creates/updates Histogram for each Column

SM_ComponentValue.UC_PK

systabstats

SM_ComponentValue.UC_PKIndex MetricsCluster Ratio Counts




Derek Asirvadem • V2.1 • DRAFT • 20 Nov 11Sybase Statistics Demystified/8 Index Stats Deprecated/10 of 16

8 Index Stats Deprecated





100 Value,Metric

100 Value100 DateTime 100 Metric100 Component

104 Value W102 Value V102 DateTime V 102 Metric V102 Component V

104 Component W 104 Metric W104 DateTime W

8

UPDATE STATISTICS SM_ComponentValue UC_PK

UPDATE STATISTICS Table [Index]

100 Server,DateTime



100 Server

V102 Server

104 Server W

The older, deprecated form of the command, If Index is omitted, all indices for the table are processed

Automatically executes the older form, if the table is populated, so ensure that UPDATE INDEX STATISTICS is executed afterward.CREATE ... INDEX Table ( Col_1 [, Col_2...] )

Continued ...


2 Creates/updates Column Stats for the leading Column

3 Creates/updates Histogram for the leading Column

SM_ComponentValue.UC_PK

systabstats

SM_ComponentValue.UC_PKIndex MetricsCluster Ratio Counts




Derek Asirvadem • V2.1 • DRAFT • 20 Nov 11Sybase Statistics Demystified/9 Table Stats/11 of 16

9 Table Stats


100 Server,DateTime



100 Server 100 DateTime 100 Metric100 Component

V102 Server

104 Server W102 DateTime V 102 Metric V102 Component V

104 Component W 104 Metric W104 DateTime W 104 Value W102 Value V




100 Value,Metric

100 Value

9

UPDATE TABLE STATISTICS SM_ComponentValue

UPDATE TABLE STATISTICS TableEXEC sp_flushstats TableThe Checkpoint task updates systabstats periodically, from memory structures, based on recovery interval. Additionally, either of the above commands may be used.

Continued ...

Memory Structure

systabstats

All IndicesIndex MetricsNot Cluster Ratio Counts

ChekpointTask

sp_flushstats




Derek Asirvadem • V2.1 • DRAFT • 20 Nov 11Sybase Statistics Demystified/10 Goals/12 of 16

10 Goals





Derek Asirvadem • V2.1 • DRAFT • 20 Nov 11Sybase Statistics Demystified/11 Help Statistic/13 of 16

11 Help Statistic


Indicates that ASE reported that Stats for the identified Column or Column Group are Missing (if such is being captured). This treatment presents it in the context of the rest of the Stats, and thus allows considered evaluation, that would otherwise be isolated. nUpd is the no of occasions the Stats were found to be missing (low occurrences can be safely ignored). Missing count is particularly useless in the absence of used count for Stats, as there is nothing to compare against, but it does provide a flag for sites that do not administer Stats.

The number of Requested & Actual Cells in the Histogram

The number of times this Statistic has been updated, and datetime of the last update

Very important single indicator, even for non-unique columns, once it is understood

Column......

ColumnGroup......

Column

Histogram

ColumnGroup

Table

The Density (Statistics) for either a Column or a ColumnGroup. ValueRange/Total are shown only if they differ from the Density

Data Change in the Column since Stats were last updated (constrained to 999 for sanity)

ColumnGroup:Index the first Index in which the ColumnGroup is indexed (matches the leading columns). If there is more than one, the remainder are identified thus:

+[n] the ColumnGroup is indexed an additional n times (identifying possible duplicate indices)Column:

Index the first Index in which the Column is indexed (matches the leading column). If there is more than one, the remainder are identified thus:+[n] the Column is indexed an additional n times (identifying possible duplicate indices)

(Index) the first Index in which the Column appears as a non-leading column. If there in more than one, the remainder are identified thus: +(n) the Column appears in n additional indices

The number of ColumnGroups this Column appears in. ColumnGroups are readily identifiable from the few lines immediately above

The number of rows used to store the table level Histogram for the Column: must be at least 2

Distinct values for the Column or ColumnGroup determined by Stats

The main identifier in the report, each row represents either a Table, or a ColumnGroup Stat, or a Column Stat, that has been stored. For ease of reference, the report is ordered by: Table ColumnGroups Columns

The named Index is Unique

This page describes the content of the Software Gems Help Statistic report. In its default operation, Columns and ColumnGroups Stats are reported at the table level, partition level Stats are not shown. The report shows all Column and ColumnGroup Stats for the tables requested; allows Columns to be easily related to their ColumnGroups and Indices. The Histograms can be viewed via Help Histogram. For detailed information in other areas, refer to the Help Table, Help Index or Help Partition reports (derived stats and index metrics belong with the index).

Table entries are useful in organising the Help Statistic report. They are shown here with the information relevant to the analysis (these columns are overloaded):

Grp [number of indices]Un [number of unique indices]Req [number of partitions]Cell number of Cells in the table level Histogram for all ColumnsHist number of rows used to store the table level Histogram for all ColumnsValueUnique number of rows in the table

Table/ColumnGroup/Column Grp Index/Notice Un M Upd Updated Req Cell Hist Delta DensityRange DensityTotal Uniquens ValueUnique ValueRange ValueTotal

C_AccessSequence [1] -NoGrpStats- [1] 4 4 1

SubjectAreaCode 0 UC_Base Un 3 10/01/06 23:10 20 2 2 0.00 1.0 1.0 1.0 1

Sequence 0 (UC_Base) Un 3 10/01/06 23:10 20 2 2 0.00 1.0 1.0 1.0 1

SM_ComponentMetric [1] -NoStats- [1] 122

SM_ComponentValue [2] [2] 441 16 5737

Server,Component U_ComponentMetric Un 3 11/08/26 02:03 0.0 0.1025204531 0.00209 12 0.0833333333

Server,Component,Metric U_ComponentMetric Un 3 11/08/26 02:03 0.0 0.0092919949 0.01987 114 0.0087719298

Server,Component,Metric,DateTime U_ComponentMetric Un 3 11/08/26 02:03 0.0 0.0001743071 1.0 5737

Server,DateTime UC_PK Un 3 11/08/26 02:01 0.0 0.0149656510 0.01673 96 0.0104166667

Server,DateTime,Component UC_PK Un 3 11/08/26 02:01 0.0 0.0020003847 0.11713 672 0.0014880952

Server,DateTime,Component,Metric UC_PK Un 3 11/08/26 02:01 0.0 0.0001743071 1.0 5737

Component,Metric 3 11/08/26 02:09 0.0 0.0092919949 0.01987 114 0.0087719298

Server 6 UC_PK+[1] 3 11/08/26 02:03 20 2 2 0.00 0.0 1.0 0.00017 1

DateTime 4 (UC_PK)+(1) 3 11/08/26 01:50 100 192 6 0.00 0.0 0.0149656510 0.01673 96 0.0104166667

Component 6 (UC_PK)+(1) 3 11/08/26 02:09 300 24 2 0.00 0.0 0.1025204531 0.00209 12 0.0833333333

Metric 4 (UC_PK)+(1) 3 11/08/26 01:59 300 223 6 0.00 0.0001743071 0.0095720044 0.01952 112 0.0089285714

SM_DeviceMetric [1] -NoStats- [1] 45

SM_DeviceValue [1] [1] 103 10 5787

Server,DateTime UC_PK Un 3 11/08/17 20:02 0.0 0.0209913480 0.00829 48 0.0208333333

Server,DateTime,DeviceType UC_PK Un 3 11/08/17 20:02 0.0 0.0086793215 0.02488 144 0.0069444444

Server,DateTime,DeviceType,Device UC_PK Un 3 11/08/17 20:02 0.0 0.0016105111 0.19734 1142 0.0008756567

Server,DateTime,DeviceType,Device,Metric UC_PK Un 3 11/08/17 20:02 0.0001728011 0.0001728011 1.0 5787

Server 4 UC_PK Un 3 11/08/17 20:02 20 2 2 603.99 0.0 1.0 0.00017 1

DateTime 4 (UC_PK) Un 3 11/08/17 20:02 20 26 2 603.99 0.0 0.0800381547 0.00225 13 0.0769230769

DeviceType 3 (UC_PK) Un 3 11/08/17 20:02 20 4 2 603.99 0.0 0.5461325009 0.00052 3 0.3333333333

Device 2 (UC_PK) Un 3 11/08/17 20:02 20 37 2 603.99 0.0130824373 0.1049543300 0.00397 23 0.0115207373 0.0434782609

Metric 1 (UC_PK) Un 3 11/08/17 20:02 20 34 2 603.99 0.0191463682 0.0483437392 0.00588 34 0.0131720430 0.0294117647

These four pages should be moved to the Reference doc.

Continued ...




Derek Asirvadem • V2.1 • DRAFT • 20 Nov 11Sybase Statistics Demystified/12 Help Statistic/Partition/14 of 16

12 Help Statistic/Partition


The number of Cells in the table level Histogram for the Column

For partitioned tables, the number of Cells in the partition level Histogram for the Column

Column

Partition

ColumnGroup

Partition

Each row represents either a Table, or a ColumnGroup Stat, or Column Stat that has been stored. For ease of reference, the report is ordered by: Table ColumnGroup (table level) ColumnGroup.Partition (partition level)

Column (table level) Column.Partition (partition level)

The row count for the table (table entry) or Partition (partition entry). This should be evaluated against ValueUnique determined by Stats

The number of rows used to store the table level Histogram for the Column

As per the previous page, in its default execution, the Help Statistic report provides information on Stats at the table level. For partitioned tables, Sybase automatically stores ColumnGroup and Column Stats (including the Histograms) for each partition. For administrators who manage their Stats at the partition partition level, such a report is provided, the additional rows and columns are described here.

The concept is, the Column or ColumnGroup exists in each partition; therefore partition is below Column or ColumnGroup. This is consistent with the optdiag report.

For partitioned tables, the number of rows used to store the partition level Histograms for the Column

ColumnGroup Stats: 1 x Table p x Partition

Histograms:: 1 x Table p x Partition

Column Stats: 1 x Table p x Partition

Column

Histogram

ColumnGroup

Table

Table/ColumnGroup/Column.Parttn Grp Index/Notice Un M Upd Updated Req CellT CellP HistT HistP Delta DensityRange DensityTotal Uniquens Row ValueUnique ValueRange ValueTotal

TestBase_APL [2] [2] [4] 2253 8000 63 224 2000010

SecurityId,Name 3 11/08/22 00:00 0.0000005000 0.0000005000 1.0 2000010

SecurityId,Name.1536158 3 11/08/21 23:59 0.0000040256 0.0000040256 0.5 248410

SecurityId,Name.1169540319 3 11/08/21 23:59 0.0000040258 0.0000040258 0.50061 248400

SecurityId,Name.1185540376 3 11/08/22 00:00 0.0000040258 0.0000040258 0.50071 248400

SecurityId,Name.1201540433 3 11/08/22 00:00 0.0000039169 0.0000039169 0.49971 255303

Name,SecurityId 3 11/08/21 23:58 0.0000005453 0.0000005453 0.91685 1833702

Name,SecurityId.1536158 3 11/08/21 23:55 0.0000020128 0.0000020128 1.0 496821

Name,SecurityId.1169540319 3 11/08/21 23:56 0.0000020153 0.0000020153 1.0 496195

Name,SecurityId.1185540376 3 11/08/21 23:57 0.0000020158 0.0000020158 1.0 496091

Name,SecurityId.1201540433 3 11/08/21 23:58 0.0000019573 0.0000019573 1.0 510903

SecurityId 2 UC_SecurityId Un 3 11/08/22 00:00 1000 1090 4000 28 104 0.00 0.0000005000 0.0000005000 1.0 2000010 2000010

SecurityId.1536158 3 11/08/21 23:59 1000 1000 26 0.00 0.0000020128 0.0000020128 1.0 496821 496821

SecurityId.1169540319 3 11/08/21 23:59 1000 1000 26 0.00 0.0000020153 0.0000020153 1.0 496195 496195

SecurityId.1185540376 3 11/08/22 00:00 1000 1000 26 0.00 0.0000020158 0.0000020158 1.0 496091 496091

SecurityId.1201540433 3 11/08/22 00:00 1000 1000 26 0.00 0.0000019573 0.0000019573 1.0 510903 510903

Name 2 U_Name Un 3 11/08/21 23:58 1000 1163 4000 35 120 0.00 0.0000005453 0.0000005453 0.91685 1833702

Name.1536158 3 11/08/21 23:55 1000 1000 30 0.00 0.0000020128 0.0000020128 1.0 496821

Name.1169540319 3 11/08/21 23:56 1000 1000 30 0.00 0.0000020153 0.0000020153 1.0 496195

Name.1185540376 3 11/08/21 23:57 1000 1000 30 0.00 0.0000020158 0.0000020158 1.0 496091

Name.1201540433 3 11/08/21 23:58 1000 1000 30 0.00 0.0000019573 0.0000019573 1.0 510903

Table entries are useful in organising the Help Statistic report. They are shown here with the information relevant to the analysis (these columns are overloaded):

Grp [number of indices]Un [number of unique indices]Req [number of partitions]Cell number of Cells in the table level Histogram for all ColumnsHist number of rows used to store the table level Histogram for all ColumnsValueUnique number of rows in the table

For partitioned tables:CellT number of Cells in the table level Histogram for all ColumnCellP number of Cells in the partition level Histogram for all ColumnHistT number of rows used to store the table level Histogram for all ColumnsHistP number of rows used to store the partition level Histogram for all Columns

Continued ...




Derek Asirvadem • V2.1 • DRAFT • 20 Nov 11Sybase Statistics Demystified/13 Help Histogram/15 of 16

13 Help Histogram



SM_ComponentValue Metric 1 0.0 = APF IO Wait \x1f

2 0.0356083103 <= APF Req/Found Sp

3 0.0326409489 <= Active Page

4 0.0385756679 <= Buffer Washed

5 0.0356083103 <= Byte Received

6 0.0356083103 <= CPU Idle

7 0.0356083103 <= Cache Miss

8 0.0326409489 <= Context Switched

9 0.0356083103 <= Device Interrupt

10 0.0356083103 <= File/Page In

11 0.0267062318 <= Flush/Commit

12 0.0 <

13 0.0148367956 = Flush/DOL Conflict

14 0.0415430255 <= Flush/Single LR

15 0.0385756679 <= Garbage Collected

16 0.0445103869 <= IO Checked

17 0.0326409489 <= IO Completed

18 0.0356083103 <= Ins/APL Heap

19 0.0326409489 <= LIO Req/Denied Pool

20 0.0356083103 <= Log Alloc

21 0.0356083103 <= Log Write

22 0.0326409489 <= Network Other

23 0.0296735913 <= PIO Queue

24 0.0326409489 <= Priority Changed

25 0.0326409489 <= Replacement/Scanned

26 0.0356083103 <= Row/Updated

27 0.0326409489 <= Scope Changed

28 0.0326409489 <= Swap/Page In

29 0.0474777445 <= ULC Granted

30 0.0504451022 <= Voluntary

31 0.0178041551 <= Xact Committed

SM_ComponentValue Value 1 0.0 <

2 0.0741839781 = 0

3 0.0415430255 <= 3

4 0.0296735913 <= 13

5 0.0296735913 <= 37

6 0.0356083103 <= 62

7 0.0296735913 <= 143

8 0.0296735913 <= 258

9 0.0296735913 <= 508

10 0.0296735913 <= 883

11 0.0296735913 <= 2470

12 0.0296735913 <= 4104

13 0.0296735913 <= 5755

14 0.0296735913 <= 16600

15 0.0296735913 <= 22140

16 0.0296735913 <= 32560

17 0.0296735913 <= 44781

18 0.0296735913 <= 65214

19 0.0296735913 <= 83090

20 0.0296735913 <= 114152

21 0.0296735913 <= 150749

22 0.0296735913 <= 209891

23 0.0296735913 <= 288532

24 0.0296735913 <= 466617

25 0.0296735913 <= 611840

26 0.0296735913 <= 729180

27 0.0296735913 <= 1302682

28 0.0296735913 <= 1793188

29 0.0296735913 <= 2341070

30 0.0534124635 <= 24862772

31 0.0534124635 <= 1370920503

The Help Histogram report provides full details of the Histograms for the requested tables. It is an adjunct to the Help Statistic report, which should satisfy most needs, however there are times when the full extent of Statistics detail needs to be inspected. In order to allow verification of the data, it matches the optdiag report exactly; but is provided in:

(a) a much more readbale and understandable format, (b) with the confusing bits removed(c) as with all our software, it uses the set-processing power of the engine, and thus

executes in set-processing speeds, not at developer speeds. This, and its absence of failure, differentiate it from the free "software" available on the net.

Advantage over optdiagIt is provided as a stored procedure, which makes it substantially easier to access and use, and the report is therefore a result set, which means the data is also easier to access, use, transfer to a spreadsheet, etc. The improved readability is afforded by:• notable values of 0.0 and 1.0 in the Weight being shortened (or, zero-fill removed):

• Null Values are shown as an empty cell, and therefore more obvious• Frequency Cells are therefore more obvious• the Null Cell, is more obvious, and • differentiated from the minimum Value cell

• repeated values (incorrectly called "duplicated values" in the manuals) are not repeated: • this makes Frequency stand out from Range cells

• the empty space makes the ordinary Weight and Values stand out; the report otherwise is a fat rectangular block of unreadable numbers

This allows:• quick problem diagnosis• accurate problem diagnosis (eliminate errors and reversals)• the minimum Stats required to be easily identified and executed

(therefore minimum effect on production)


SM_ComponentValue Server 1 0.0 <

2 1.0 = SEQUOIA

SM_ComponentValue DateTime 1 0.0 <

2 0.1246290803 = 10/12/21 05:00

3 0.0 <

4 0.1246290803 = 10/12/21 12:00

5 0.0 <

6 0.0652818978 = 10/12/21 18:00

7 0.0 <

8 0.0652818978 = 10/12/21 22:00

9 0.0 <

10 0.1246290803 = 11/02/07 03:00

11 0.0 <

12 0.0474777445 = 11/02/07 04:00

13 0.0118694361 <= 11/02/07 05:00

14 0.0 <

15 0.0623145401 = 11/02/07 09:00

16 0.0 <

17 0.1246290803 = 11/02/07 11:00

18 0.0 <

19 0.0178041551 = 11/02/07 15:00

20 0.0 <

21 0.0415430255 = 11/02/07 16:00

22 0.0 <

23 0.0652818978 = 11/02/07 18:00

24 0.0118694361 <= 11/02/14 11:00

25 0.0 <

26 0.0623145401 = 11/02/14 12:00

27 0.0 <

28 0.0504451022 = 11/02/14 13:00

SM_ComponentValue Component 1 0.0 <

2 0.1513353139 = Cac

3 0.0 <

4 0.0890207738 = Ctx

5 0.0 <

6 0.0415430255 = Dsk

7 0.0 <

8 0.1008902043 = Eng

9 0.0 <

10 0.0801186934 = Hkp

11 0.0 <

12 0.1394658685 = Log

13 0.0 <

14 0.0504451022 = Net

15 0.0 <

16 0.0623145401 = Upr

17 0.0 <

18 0.0267062318 = Usy

19 0.0 <

20 0.0356083103 = Uth

21 0.0 <

22 0.0919881314 = Uvm

23 0.0 <

24 0.1305637956 = Xct

Column

Histogram

ColumnGroup

Table

Continued ...




Derek Asirvadem • V2.1 • DRAFT • 20 Nov 11Sybase Statistics Demystified/14 Help Histogram/Formatted/16 of 16

14 Help Histogram/Formatted


Histogram

Column Stats

The formatted option of the Help Histogram report combines the essentials of the Statistics [10] and the full Histogram [12] into one report, intended for landscape printing. The Histogram remains vertical, but it is displayed in three columns, immediately below the Column Statistics. It is particulary useful when dealing with a small number of tables.

Db/DataType_ __ Table/Column.Parttn_____________ Pttn Row/Delta___ __ DensityRange/TotalDensity_______ Cell Uniq/Uniqns __ ValueRange/ValueTotal___________

Col sg_Master SM_DeviceValue 5787 0.0 1

CHAR(12) Server 603.99 1.0 2 0.00017

Cell Weight Op Value Cell Weight Op Value Cell Weight Op Value

1 0 < 2 1 = SEQUOIA



SMALLDTM DateTime 603.99 0.0800381547 26 0.00225 0.0769230769


1 0.0 < 10 0.0833333358 = 10/12/21 18:00 19 0.0 <

2 0.0833333358 = 10/12/21 2:00 11 0.0 < 20 0.0833333358 = 11/02/07 10:00

3 0.0 < 12 0.0833333358 = 10/12/21 21:00 21 0.0 <

4 0.0833333358 = 10/12/21 6:00 13 0.0 < 22 0.0833333358 = 11/02/07 14:00

5 0.0 < 14 0.0322580636 = 11/02/07 1:00 23 0.0 <

6 0.0833333358 = 10/12/21 10:00 15 0.0 < 24 0.0833333358 = 11/02/07 18:00

7 0.0 < 16 0.0510752685 = 11/02/07 2:00 25 0.0 <

8 0.0833333358 = 10/12/21 14:00 17 0.0 < 26 0.0833333358 = 11/02/07 22:00

9 0.0 < 18 0.0833333358 = 11/02/07 6:00



CHAR(1) DeviceType 603.99 0.5461325009 4 0.00052 0.3333333333


1 0 < 3 0.2043010741 <= D

2 0.7043010592 = C 4 0.0913978517 <= E


Col sg_Master SM_DeviceValue 5787 0.0130824373 23 0.0115207373

CHAR(20) Device 603.99 0.1049543300 37 0.00397 0.0434782609


1 0.0 < 14 0.1129032224 = meta_cache 27 0.0 <

2 0.0564516112 = Engine 0 15 0.0 < 28 0.0215053763 = surdev8

3 0.0 < 16 0.0188172050 = surdev1 29 0.0107526882 <= sysprocsdev

4 0.0349462368 = Engine 1 17 0.0134408604 <= surdev2 30 0.0 <

5 0.0268817209 <= dbccscan 18 0.0 < 31 0.1344086081 = tempdb_cache

6 0.0 < 19 0.0134408604 = surdev3 32 0.0 <

7 0.1236559153 = default_dc 20 0.0134408604 <= surdev4 33 0.0080645159 = tempdbdev

8 0.0 < 21 0.0 < 34 0.0 <

9 0.1747311801 = index_cache 22 0.0188172050 = surdev5 35 0.0053763441 = tmpdbdev2

10 0.0 < 23 0.0 < 36 0.0 <

11 0.1424731165 = log_cache 24 0.0215053763 = surdev6 37 0.0080645159 = tmpdblog1

12 0.0161290318 <= master 25 0.0 <

13 0.0 < 26 0.0241935477 = surdev7


Col sg_Master SM_DeviceValue 5787 0.0191463682 34 0.0131720430

CHAR(20) Metric 603.99 0.0483437392 34 0.00588 0.0294117647


1 0.0 < 13 0.0215053763 = Dirty Page Read 25 0.0403225794 = Read APF

2 0.0537634417 = 2k BufferGrab 14 0.0 < 26 0.0 <

3 0.0295698922 <= 4k Page Used 15 0.0591397844 = Hit 27 0.0645161271 = Read Non-APF

4 0.0241935477 < 16 0.0295698922 <= LIO Req/Denied Pool 28 0.0 <

Data values in the Histogram are truncated at 32 bytes

Relevant Column Stats are displayed in two rows

Column

Histogram

ColumnGroup

Table

