Vital Signs: Performance Monitoring Windows Server Module 2: Disk Microsoft Confidential.

Vital Signs: Performance Monitoring Windows

Server

Module 2: Disk

Microsoft Confidential

Conditions and Terms of Use

This training package is proprietary and confidential, and is intended only for uses described in the training materials. Content and software is provided to you under a Non-Disclosure Agreement and cannot be distributed. Copying or disclosing all or any portion of the content and/or software included in such packages is strictly prohibited.

The contents of this package are for informational and training purposes only and are provided "as is" without warranty of any kind, whether express or implied, including but not limited to the implied warranties of merchantability, fitness for a particular purpose, and non-infringement.

Training package content, including URLs and other Internet Web site references, is subject to change without notice. Because Microsoft must respond to changing market conditions, the content should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information presented after the date of publication. Unless otherwise noted, the companies, organizations, products, domain names, e-mail addresses, logos, people, places, and events depicted herein are fictitious, and no association with any real company, organization, product, domain name, e-mail address, logo, person, place, or event is intended or should be inferred.

Microsoft may have patents, patent applications, trademarks, copyrights, or other intellectual property rights covering subject matter in this document. Except as expressly provided in written license agreement from Microsoft, the furnishing of this document does not give you any license to these patents, trademarks, copyrights, or other intellectual property.

Complying with all applicable copyright laws is the responsibility of the user. Without limiting the rights under copyright, no part of this document may be reproduced, stored in or introduced into a retrieval system, or transmitted in any form or by any means (electronic, mechanical, photocopying, recording, or otherwise), or for any purpose, without the express written permission of Microsoft Corporation.

For more information, see Use of Microsoft Copyrighted Content athttp://www.microsoft.com/about/legal/permissions/

Microsoft®, Internet Explorer®, and Windows® are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. Other Microsoft products mentioned herein may be either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. All other trademarks are property of their respective owners.

Copyright and Trademarks © 2011 Microsoft Corporation. All rights reserved.


http://www.microsoft.com/about/legal/permissions/

http://www.microsoft.com/about/legal/permissions/

Students: How to View this Presentation

Switch to Notes Page view Click View on the ribbon and select Notes Page Use page up or page down to navigate Zoom in or out as needed

Most slides will have supporting text that you can view now or after the delivery

Add notes to your copy of the presentation if you want to.

You take the presentation files home with you.


Module 2: Disk

Section 1: Disk hardwareSection 2: ConceptsSection 3: Disk countersSection 4: Disk analysis tools


Section 1: Disk hardware

Hard disk magnetic mediaSolid state drive (SSD)Direct attached storage (DAS)Just a Bunch of Disks (JBOD)Network attached storage (NAS)Internet Small Computer System Interface (iSCSI)Storage area network (SAN)I/O operations per second (IOPS)Dedicated spindlesShared spindles


Hard disk magnetic media

A magnetic disk (spindle) rotates at high speed so data can be read or written This basic design has not changed much in many years, although capacity and speed have increased greatly


Solid state drive (SSD)

A data storage device that uses solid-state memory to persistently store data, in order to provide access in the same manner as a traditional block I/O hard disk driveUses NAND-based flash memory, with no moving parts

Generally uses less power and has significantly faster access times than spinning magnetic media


I/O operations per second (IOPS)

An industry measurement of the hardware throughput of a disk or a collection of disksFor example, a single 7200 RPM disk drive has an uncached, full stroke access time of 13 ms, which calculates to at least 77 IOPSAverage response time longer than 15 ms is generally considered slow

The following chart shows worst-case IOPS and access times:

Storage IOPS* Access time (ms)*

3.5“ floppy disk USB drive 1.54 6005400 RPM hard disk 59 177200 RPM hard disk 71 1310K RPM hard disk 122 815K RPM hard disk 143 7Solid state drive (SSD) 5000 0.2

* General approximations of full stroke (worst case) without cache. Does not reflect actual products.

15 ms is a generally accepted threshold


I/O operations per second (IOPS) continued…

Disk speeds are determined by moving parts.Rotational Latency is the rate at which the platter rotates.• [speed] / 60 for rotations / sec and [speed] / 60,000 for rotations / ms• A 15k disk rotates 250 / sec, which means one full rotation takes 4 ms• A 10k disk rotates 167 / sec, which means one full rotation takes 6 ms

Average Seek Time is the time it takes the head assembly on the actuator arm to travel to the track of the disk where the data will be read or written.• Average Seek Time can vary greatly; not all disks are created equal.• If the Average Seek Time is not listed on the HDD label, you may need to

contact the vendor to attain this value.• A 15k disk may have an Average Seek Time of 3 ms or more.• A 10k disk may have an Average Seek Time of at least 5.5 ms• A 7200RPM disk Average Seek Time may be closer to 9 ms


I/O operations per second (IOPS) continued…

To determine the IOPS for a disk use the following formula:• (1/[Rotational Latency (half rotation) + Average Seek Time])*1000• A 15k disk may have an IO time from 5 to 6 ms and be capable of 167 to 200

IOPS• A 10k disk may have an IO time of around 8.15 ms and be capable of 122

IOPS• A 7200 RPM disk may have an IO time of 14.1 ms and be capable of only 71

IOPS

Consult your HDD manufacturer for the Average Seek Time values.

These numbers do not account for any type of cache and are not for Solid State Disks.

Why is an access time of 15 ms or more generally considered slow?

• Because 15ms is slower than a 7200 RPM disk.


Challenge Question:

1. If your application requires 2000 IOPS, how many disks are needed (not accounting for RAID overhead)?


Answer: It depends on the disk speed. You might need 10 15k disks or you may need 17 10k disks or you may only need 1 solid state disk.

Required IOPS / IOPS per disk = Amount of disks needed

2000 (IOPS needed) / 200 (1 15k disk) = 102000 (IOPS needed) / 122 (1 10k disk) = 17 2000 (IOPS needed) / 5000 (1 SSD) = 1

Direct attached storage (DAS)

Presented as a Logical Unit Number (LUN)A hard drive that is “directly” attached to a server or workstation, without a storage network in betweenCalled a “local” diskGenerally implemented as IDE, SCSI, SATA, etc.

ID 0 Disk controller

IDE, SATA, SCSI, etc.

LUN 0

ID 1 LUN 1

LUN 0 & 1


Just a Bunch of Disks (JBOD)

JBOD is direct attached storage (DAS) that is configured, typically, in a Redundant Array of Independent Disks (RAID) setDisks might be independently addressed or concatenated by hardware

LUN 1

Server

J B O D

J B O D


Network attached storage (NAS)

File-level computer data storage that is connected to a TCP/IP Ethernet networkUses Server Message Block (SMB), Network File System (NFS), or other network protocolsAvailable via UNC network paths (\\svr\share) or in some cases as direct attached storage (DAS)

NICNAS Controllers

TCP/IP

\\svr\share1

\\svr\share2

NIC

Other NAS


Internet Small Computer System Interface (iSCSI)

A gigabit, IP-based storage networking standard for linking data storage facilitiesA protocol suite that makes it possible for SCSI commands to be encapsulated and transmitted over the TCP/IP stackTo access the iSCSI network, computers use an Ethernet network interface card (NIC)

NICSAN Controllers

TCP/IP

LUN 1 & 2

LUN 1 & 2

NIC

Other SANs

iSCSI


Storage area network (SAN) and Fibre Channel (FC)

A SAN is a network of storage resourcesA SAN consolidates and shares storage media, using hardware efficientlyComputers connect to one or more SANs by using a host bus adapter (HBA) to connect to a Fibre Channel networkFibre Channel (FC) is a gigabit-speed network technology over fiber-optic cables that is primarily used for network storage

HBASAN Controllers

Fibre Channelswitch

Fibre Channel orSAN fabric

LUN 1 & 2

LUN 1 & 2

HBA

Other SANs

SAN


Dedicated spindles

Physical spindles that are not used by or shared with any other computer• Generally reliable and consistent IOPS and response times• Each server has its own disk usage• Some servers might have idle spindles• Poor performance affects only one server

In this case, the Exchange server needs more spindles. The SQL server and the file server have idle spindles

Exchange server SQL server File server

over utilization moderate utilization under utilization

0 idle spindles 6 idle spindles 9 idle spindles


Shared spindles

Physical spindles that are used by or shared with other computers• Higher potential for unreliable or inconsistent IOPS and response times than

dedicated spindles• Spindles (and potentially controllers) are pooled together and share the load

o Less likely to have idle spindles compared to a dedicated spindle modelo Some servers might have better or worse disk performance

• Good or poor performance can affect multiple serversIn this example, disk performance of the Exchange server is better, at the cost of using the spindles for the SQL server and the file server

? idle spindlesSAN Controllers

Exchange server

SQL server

File server

variable?

variable?

variable?


Section 1 review

1. What is the difference between NAS and a SAN?2. Why is an access time of 15 ms or more generally

considered slow?


Section 1 review (answers)

1. A SAN provides direct attached storage (C:, D:, etc.), and a NAS provides network shares (\\svr\share).

2. Because 15 ms is the uncached access time median between 5400 and 7200 RPM hard disks, response time averages above it are generally considered slow.


Section 2: Concepts

Physical disks and logical disksSoftware RAIDHardware RAIDI/O request packets (IRPs)


Physical disks and logical disks

Physical disk• A LUN that is presented to the operating system• Unknown hardware behind the LUN—unknown spindles• Direct attached storage (DAS)

Logical disk• Disk volume with a drive letter• Multiple logical disks can exist on a physical disk, and vice versa

Physical disks

(LUNs)

Logical disks (drive letters and mount

points)

Disk Manager


Software RAID

Software RAID combines Windows and Windows Server physical disks into a logical RAID setWindows Server 2003 and later support software RAID 0, 1, and 5In this example, drive G is a logical software RAID 0 set, or a “spanned volume”

Drive G is a software RAID 0 set

Disk Manager


Hardware RAID

Presented to Windows and Windows Server as a single physical disk (LUN)Physical architecture is unknown to Windows and Windows Server

Disk 2 might have one or more physical spindles behind it


I/O request packets (IRPs)

An I/O request packet (IRP) is a data structure in Windows and Windows Server that controls how the I/O operation is processed at each stageLike a network packet, the structure is fully self-contained and has all routing information inside


Section 2 review

1. What is the difference between a physical disk and a logical disk from the perspective of Windows and Windows Server?

2. What is the difference between software and hardware RAID?

3. What is a data structure in Windows and Windows Server that controls how the I/O operation is processed at each stage?



1. A physical disk is a direct attached storage device that is presented to Windows or Windows Server. It is also called a LUN. A logical disk is a disk volume that has a drive letter or that is a mount point.

2. Hardware RAID consolidates multiple hard drives into a single LUN that is presented to Windows or Windows Server. Software RAID is where Windows or Windows Server consolidates multiple LUNs.

3. An I/O request packet (IRP) is a data structure in Windows and Windows Server that controls how the I/O operation is processed at each stage.


Section 3: Disk counters

PhysicalDisk and LogicalDisk counter objectsDisk countersTroubleshootingProcess I/O countersDisk free space counters


PhysicalDisk and LogicalDisk counter objects

PhysicalDisk counters• Measure LUN performance• Hardware RAID, disk virtualization, and I/O distribution can

make measurements difficult to assess• Examples: 07 (D:, E:)

LogicalDisk counters• Measure performance of lettered drives• Logical disk counters are usually more relevant to applications

and services• Example: drive D:, E:, or D:\Disk1\BackupDrive7


Disk Counters

Primary counters• Average Disk sec / Read• Average Disk sec / Write

o These counters show disk latency. How long did the data take to access?

Secondary counters• Disk Transfers / sec• Disk Bytes / sec• Average Disk Queue Length• Current Disk Queue Length

o These counters measure the amount of disk activity. How much data is being accessed?


Avg. Disk Sec/Read|Write

Avg. Disk sec/Read and Avg. Disk sec/Write are the average times, in seconds, of read or write operations respectivelyAlso called disk response timesAn initial indicator of disk performancePart of the LogicalDisk and PhysicalDisk counter objectsImplemented at the disk driver class for high accuracy Physical media

Disk queue

Driver

Hardware controller

Bus|Network|Cable

How long did it take?

Res

pons

e tim

e

(continued)Microsoft Confidential

Avg. Disk Sec/Read|Write (continued)

Software and hardware disk cache can greatly affect disk latencyDue to disk virtualization and I/O distribution, disk latency is a reliable measure of disk performanceCompare to the access time in the hardware specification of the bus/network/cable and the physical mediaExample: A 7200 RPM disk drive should respond faster than 13 ms Physical media

Disk queue

Driver

Hardware controller

Bus|Network|Cable

Measures average

response times of IRPs

Res

pons

e tim

e


Thresholds for Avg. Disk sec/Read|Write

Threshold Effect

15 ms or less

Informational Normal usage

More than 15 ms

WarningPerformance might be affected

More than 25 ms

CriticalPerformance very likely affected

Common next steps:

Check hardware for failures or faultsCheck I/O sizes more than 64 KBIf the hardware is shared, consider using dedicated hardwareIf you are using spinning media, consider defragmentingAdjust the hardware cacheConsider using faster mediaReduce unnecessary I/O

Response times longer than 15 ms (based on average I/O sizes of 64 KB or smaller) generally impact performanceQuick spikes in response times are common and normal


Example of Avg. Disk sec/Write

In this case, drive E consistently has response times of more than 25 ms, which greatly affects the application and system performance

Drive E consistently

has response times of more

than 25 ms


Disk queue counter

Disk queueAvg. Disk Queue Length counterExample of Avg. Disk Queue Length


Disk queue

Each disk has a disk queue The disk queue length is the number of “in-flight” I/O request packets (IRPs)One or more IRPs can be serviced at the same time, depending on the physical mediaIRPs might be reorganized by a driver to optimize the physical mediaExample: The HBA queue depth determines how many I/Os a SAN can service simultaneously Physical media

Disk queue

Driver

Hardware controller

Bus|Network|Cable

The number of “in-flight” IRPs waiting for service


Avg. Disk Queue Length counter

The Avg. Disk Queue Length counter is the average number of both read and write requests that are queued or “in-flight” for the selected disk during the sample intervalPart of LogicalDisk and PhysicalDisk counter objectsBecause of disk virtualization and I/O distribution, no threshold can be assignedLook for increasing trends

Physical media

Disk queue

Driver

Hardware controller

Bus|Network|Cable

The number of “in-flight” IRPs waiting for service


Example of Avg. Disk Queue Length

In this example, the average length of the disk queue of drive F is increasingMight be because of an increased load or an overwhelmed disk

Queue length of drive F

(orange line) is increasing

over time


Disk throughput counter

Disk Transfers/sec is the rate of read and write operations on the diskUse this counter to establish a performance baselineShows the number of serviced IRPs per secondPart of the LogicalDisk and PhysicalDisk counter objects

Physical media

Disk queue

Driver

Hardware controller

Bus|Network|Cable

How many per second?

Dis

k tr

ansf

ers


Disk Transfers/sec and I/Os

Disk Transfers/sec is not the same as I/O operations per second (IOPS)A read or write operation might result in more than one I/O on the hardwareConversions of read and write operations to RAID I/O:• RAID 0: (reads) + (writes)• RAID 1: (reads) + (2 x writes)• RAID 0+1: (reads) + (2 x

writes)• RAID 5: (reads) + (4 x writes)• RAID 10: (reads) + (2 x writes)

Disk queue

Windows write operation

RAID 1 (mirror)

This one write operation

results in two I/Os


50%

writes

50%

reads

Disk read/write ratio counters

The ratio of read and write operations can help you in adjusting the disk cacheCompare the Reads/sec and Writes/sec counters to determine the read/write ratioPart of the LogicalDisk and PhysicalDisk counter objectsDisk cache is generally significantly faster media in front of slower media storageAdjusting the disk cache can greatly affect performance

Physical media

Disk queue

Hardware disk cache

R R W W R W


I/O size counters

The size of the read and write operations (IRPs) can affect the response timesUse Avg. Disk Bytes/Read and Avg. Disk Bytes/Write counters to determine the I/O sizesThe threshold of a 15 ms response time is based on I/O sizes of 64 KB or smallerPart of LogicalDisk and PhysicalDisk counter objects

Physical media

Disk queue

1 MB64K 4K

30 ms10 ms

4 ms

Larger I/Os might take

longer

Dis

k tr

ansf

ers


Troubleshooting flow

To determine if you have a disk issue, use the following set of questions:1. Do I have disk latency? (Is Average Disk sec / Read or

Average Disk sec / Write greater than 0.025?)• If there is no latency there is no need to go further.

2. Are all or most of the disks showing latency?• If only one disk has latency then there might be a bottleneck in the

JBOD. • If all disks show latency then it might indicate a bottleneck in the

fabric


Troubleshooting flow continued…

3. Is there substantial queuing? (Is Average Disk Queue Length greater than 32?)• Is my HBA unable to keep up? Queuing my indicate a fabric bottleneck

4. Compare the disk activity during the time of latency with the latency during the time of maximum disk activity• If the Disk Transfers / sec are much higher at another time when latency is

very low it would point to a fabric bottleneck.


Troubleshooting flow example

Apply the list of questions from the previous slide to the following information:1. Users complain and enter trouble tickets for the helpdesk.2. Disk latency peaks at 1100 MS.3. Disk transfers / sec equal 8194. Disk bytes / sec equal 9 MB5. Avg disk queue length equals 504

Challenge questions:

At this point, Do you suspect this is a JBOD or fabric issue?How many spindles do I need to support this load?


Troubleshooting flow example continued…


Troubleshooting flow example

Now compare the activity during the time of latency with other times to help confirm your theory.

10 minutes later we have the following data:1. Disk latency is only 29 MS. *this is still high but not as unacceptable

2. Disk transfers / sec equal 2300 3. Disk bytes / sec equal 54 MB

This would indicate that the bottleneck is in the fabric because the disk is capable of doing more at this time. It can support up to 12 disks worth of I/O. Earlier it couldn’t do 5 disks worth.


Troubleshooting flow example continued…


Process I/O counters

Process I/O operationsAvg. Disk Sec/Read|Write

Thresholds for Avg. Disk sec/Read|Write

Example of Avg. Disk sec/Write


Process I/O operations

\Process(*)\IO Read Operations/sec and \Process(*)\IO Write Operations/sec measure the rate at which the process is issuing read or write (respectively) I/O operationsThis includes file, network, and device I/OsUse as supplemental data of which process might be issuing disk I/O

Physical media

Disk queue

Sqlservr.exe

\Process(sqlservr)\IO Write Operations/sec


Disk free space counters

% Free Space measures the percentage of total usable space on a disk driveFree Megabytes displays the amount of unallocated space on the diskIf the free disk space is low, defragmentation failures or failed writes can resultThese counters are only part of the LogicalDisk counter object and do not measure free space for the LUN.If a disk is close to full, the Seek Time will be greater because the read/write heads generally travel more. Again, this counter does not measure the actual free space on each disk. It measures by partition.Data stored on the inner portion of a platter is generally slower to access. This only applies to a single spindle (workstations or very small servers)


Thresholds for % Free Space

Threshold Effect

10% or more


Less than 10%

WarningDisk latency might occur

Less than 5%

CriticalDisk latency is very likely. The system might not be able to write to the disk

Common next steps:

Delete or move any unneeded filesIncrease disk capacity if you can


% Idle Time counter

The % Idle Time counter reports the percentage of time that the disk queue is emptyPart of LogicalDisk and PhysicalDisk counter objectsA sustained average of less than 10 might warrant investigation, but no threshold can be assigned because of disk virtualization and I/O distribution

Physical media

Disk queue

Driver

Hardware controller

Bus|Network|Cable

How often is the queue

empty?


Thresholds for % Idle Time

Threshold Effect

10% or more


Less than 10%

WarningDisk latency might occur

Common next steps:

Correlate with disk response time counters such as Avg. Disk sec/Read and Avg. Disk sec/Write

% Idle Time is a good indicator of how idle the disk queue is.This counter could be used an initial indicator that warrants more investigation.Correlate this counter with other disk related counters.


Example of % Idle Time

In this example, drive E goes down to 0% idle time.This indicates that there were one or more IRPs constantly in the disk queue

% Idle Time of drive E

(orange line) shows

constant I/O in the disk queue


Section 3 review

1. True or false: IRPs in the disk queue are always served first in first out (FIFO).

2. What is the general warning threshold for disk response times?

3. The response time thresholds are based on what I/O size?4. How many I/O operations per second (IOPS) occur on a

RAID 5 disk array with 100 write operations per second?5. What kind of I/O do the \Process(*)\IO Read

Operations/sec and \Process(*)\IO Write Operations/sec counters measure?



1. False: IRPs in the disk queue can be reorganized by the disk driver for speed and efficiency.

2. The general warning threshold for disk response times is 15 ms.

3. The response time thresholds are based on an I/O size of 64 KB or smaller.

4. 400 I/O operations per second (IOPS) occur on a RAID 5 disk array with 100 write operations per second. RAID 5: (read) + (4 x write).

5. The \Process(*)\IO Read Operations/sec and \Process(*)\IO Write Operations/sec counters measure disk, network, and device I/O.


Section 4: Disk analysis tools

Resource MonitorProcess MonitorMicrosoft xperf


Resource Monitor

Provides live disk usage informationResource Monitor is built in to Windows Vista, Windows Server 2008, and laterMore detailed information in Windows Server 2008 R2 and Windows 7

Launch Resource Monitor


Process Monitor

Process Monitor (Microsoft SysInternals) can show details of disk I/O, such as process and files, number of I/Os, and aggregate call stacks of the I/OsCan show whether drivers, such as anti-virus, are slowing I/O performancehttp://live.sysinternals.com/procmon.exe


http://live.sysinternals.com/procmon.exe

Microsoft xperf

Microsoft xperf can show disk I/O detailsPart of the Windows Server 2008 Performance Tool KitFor more information, go to the “Windows Performance Analysis Developer Center” at http://msdn.microsoft.com/en-us/performance/default.aspx


http://msdn.microsoft.com/en-us/performance/default.aspx

http://msdn.microsoft.com/en-us/performance/default.aspx

Section 4 review

1. Which disk analysis tool is built in to Windows 7 and Windows Server 2008 R2?

2. Which disk analysis tool visually shows detailed I/O patterns?

3. Which disk analysis tool can be run directly from http://live.sysinternals.com?


http://live.sysinternals.com/


1. Resource Monitor is built in to Windows 7 and Windows Server 2008 R2.

2. Microsoft xperf visually shows detailed I/O patterns.3. Process Monitor can be run directly from

http://live.sysinternals.com.


http://live.sysinternals.com/

Lab 2a: The tale of two SQLs

GoalsGoals

ScenarioScenario

Use the LogicalDisk and PhysicalDisk counter objects.Investigate the disk performance of the database servers.Use the Process counters to identify which processes are consuming the most I/O

Use the LogicalDisk and PhysicalDisk counter objects.Investigate the disk performance of the database servers.Use the Process counters to identify which processes are consuming the most I/O

Between 10:40 A.M. and 10:50 A.M., users submitted trouble tickets regarding the performance of the database server.

Between 10:40 A.M. and 10:50 A.M., users submitted trouble tickets regarding the performance of the database server.


Questions?


References

Windows Internals, 5th edition, by Mark Russinovich and David A. Solomonhttp://technet.microsoft.com/en-us/sysinternals/bb963901.aspx

Taking Your Server’s Pulsehttp://technet.microsoft.com/en-us/magazine/cc718984.aspx

Windows Sysinternals Administator’s Referencehttp://technet.microsoft.com/sysinternals/hh290819 “Key OS Performance Metrics_v7.xps” • In the Supporting Documents folder with your classroom material.


http://technet.microsoft.com/en-us/sysinternals/bb963901.aspx

http://technet.microsoft.com/en-us/sysinternals/bb963901.aspx

http://technet.microsoft.com/en-us/magazine/cc718984.aspx

http://technet.microsoft.com/en-us/magazine/cc718984.aspx

http://technet.microsoft.com/sysinternals/hh290819

http://technet.microsoft.com/sysinternals/hh290819

Vital Signs: Performance Monitoring Windows Server Module 2: Disk Microsoft Confidential.

Documents

microsoft products

registered trademarks

training package content

notes page view

notes page use page

content andor software

intellectual property

training purposes