This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Methods for Detecting Demoted Direct I/O Performance Problems Barry J. Saad, ATS pSeries Advanced Technical Support Harold R. Lee, ATS pSeries Advanced Technical Support
Direct I/O (DIO) is an alternative method for accessing data stored in files on a file
system that is functionally equivalent to raw I/O for devices. The primary benefit of DIO
is reduced CPU utilization; however, if read and write requests are not properly constructed
the applications DIO request will be “demoted” resulting in significant negative impact to
application and system performance.
This paper provides a brief introduction to DIO, and introduces the techniques and tools
required to detect problems associated with demoted DIO read and write requests. To
accomplish this goal, the following topics will be discussed:
• DIO Primer - Understand the basics concepts of DIO
• Performance Impact of DIO – Understand the performance impact associated with
DIO, including the impact of DIO demotions.
• Using DIO – How to access files using DIO
• Test Scenarios – Sample program to exercise DIO
• Tools – AIX tools and techniques to monitor and troubleshoot DIO problems
DIO Primer
The default option when mounting a file system and accessing files is to allow the virtual
memory manager (VMM) to cache file-pages in main memory. In this situation, when the
application makes a read request, the VMM will determine if the file-page is in memory, if
so you have a cache hit and the data is copied from the main memory to the application’s
buffer. If the file-page isn’t in memory an I/O request will be issued to the physical disk
drive – the data will then be loaded into cache and copied to the application’s buffer. When
an application writes data, it is written from the application’s buffer to filesystem cache,
and then control returns to the application (i.e., the application doesn’t have to wait for the
physical write to occur, which improves performance). The actual writes to disk are done
later under the control of the system (i.e. the random/sequential write behind mechanism or
Methods for Detecting Demoted Direct I/O Performance Problems
physical disk, copy the data from the application’s cache into the system cache, write the
data to physical disk synchronously, and then discarding the file cache. The additional
overhead associated with demoted DIO read and write requests require additional CPU
usage and increase application response time.
Using DIO There are three different methods by which DIO can be used. The specific method that is
best for your application depends on the scope of the files requiring DIO access and your
ability to modify the application. The three options, described below, are:
• Method 1: Mount the file system with the –o DIO option
• Method 2: Mount a portion of the file system with the –v namefs option
• Method 3: Modify the application to use DIO
Method 1: Mount the file system with the –o DIO option Use this method when all of the files in the file system need DIO access and the application
program can not be modified.
For example, mount –o dio /dev/lv05 /app1 In this case all the files in the /app1 directory will be accessed using DIO. Method 2: Mount a portion of the file system with the –v namefs option Use this method when only a portion of the files in a file system need DIO access and the
application program can not be modified.
For example, mount –v namefs –o dio /app1/read_once_files /rof In this case, you would move files that require DIO access into a common directory (e.g.,
/app1/read_once_files), and then mount that sub-directory using –v namefs option of the
mount command. Now, applications reading or writing files from /rof will use DIO.
Methods for Detecting Demoted Direct I/O Performance Problems
Method to Detect Performance Problems The AIX trace command is the primary method for detected demoted DIO requests. This
section will provide examples of how to use the trace command to detect DIO demotion
problems, and in conjunction with the filemon and tprof commands, illustrate the impact
associated with demoted DIO requests.
Test Case 1: Reading a JFS file with an alignment size of 4096 # trace –aj3B1,3B2,3B3 # ./directio –b 4096 –f /home/bjsaad/proj/test.file # trcstop # trcrpt –o trcrpt.tc1 Explanation of the commands:
1. The trace –aj3B1,3B2,3B3 command starts trace in the background, and limits
the to trace hooks 3B1, 3B2 and 3B3. In this case we’re limiting the trace hooks
for two reasons (1) we’re only interesting is seeing the DIO related hooks, and (2)
we’re limiting the size of the trace file (located in /var/adm/ras/trcfile). To know
what trace hooks to use you need to review the trace hook id header file
(/usr/include/sys/trchkid.h), which is shown below in Figure 3.
2. The ./directio –b 4096 –f /home/bjsaad/proj/test.file command reads
the specified file using a 4096 block size.
3. The trcstop command stops the background trace.
4. The trcrpt –o trcrpt.tc1 command formats the raw trace output in a readable
format.
Now, determining whether your DIO requests have been demoted is a straight forward
exercise. Review the trace file, shown in Figure 4, and every time you see 3B3 the DIO
request has been demoted to an inefficient cached I/O request is being executed. As the
trace file snippet shows, you have no 3B3 events, which means that DIO is functioning as
Methods for Detecting Demoted Direct I/O Performance Problems
Test Case 2: Reading a JFS file with an alignment size of 2048 # trace –a # ./directio –b 2048 –f /home/bjsaad/proj/test.file # trcstop # trcrpt –d3B1,3B2,3B3 –Oexec=on,pid=on –o trcrpt.tc2 Explanation of the commands:
1. The trace –a command starts trace in the background. Unlike the first test case,
we’re collecting all the trace hooks.
2. The ./directio –b 2048 –f /home/bjsaad/proj/test.file command reads
the specified file using a 2048 block size.
3. The trcstop command stops the background trace,
4. The trcrpt –Opid=on –o trcrpt.tc2 command formats the raw trace output
into a readable format, and limits the output to hooks 3B1, 3B2, and 3B3. Limiting
hooks on output make the report easier to analyze, and allows the option ‘-Opid=on’
to provide the correct data, which places a new column in the trace output file that
contains the process ID number, which is beneficial in identifying the specific
process that has demoted DIO requests.
In this case the DIO read is being made using a block size of 2048. As the trace file shows,
Figure 5, 3B3 events have occurred, which indicate that the DIO requests have been
demoted. When a request is demoted the following events occur:
• The first read request for 2048 bytes cause the DIO to be demoted
• <Extra Work> A page is allocated in system cache
• Read request initiated for 4096 bytes
• <Extra Work> Data is copied into the system cache
• <Extra Work> Data is copied from the system cache to the application buffer
• <Extra Work> The system cache data is discarded
• The next read request for 2048 bytes causes the DIO to be demoted
• <Extra Work> A page is allocated in the system cache
Methods for Detecting Demoted Direct I/O Performance Problems
# of 512 blocks read from the physical device = equals the size of the file.
# of 512 blocks read from the physical device = 2 x the size of the file. Note not exactly 2x because the file isn’t a even multiple of 4k. Increase # of physical reads is due the DIO demotion!
Methods for Detecting Demoted Direct I/O Performance Problems
Test Case 3: Reading a JFS file with an alignment size of 4096 and an offset of 1024 # trace –aj3B1,3B2,3B3 # ./directio –s 1024 –b 4096 –f /home/bjsaad/proj/test.file # trcstop # trcrpt –o trcrpt.tc3
In this example, the application reads on 4096 block size; however, the offset alignment is
incorrect. Therefore, all the I/O requests will be demoted. The snippet, shown in Figure 7,
shows 3B3 events, which indicate demoted DIO requests.
Methods for Detecting Demoted Direct I/O Performance Problems
Test Case 4: Reading a JFS2 file with an alignment size of 4096 # trace –aj59B,59C # ./directio –b 4096 –f /home/hrlee/proj/test.file # trcstop # trcrpt –o trcrpt.tc4 The same technique is used for JFS2; however, the trace hooks ids have changed. To
determine the correct hook id we check the file /usr/include/sys/trchkid.h, as shown in
Figure 9. The trace hook header file indicates that we need to look for hook 59B sub-hook
3 events to determine whether a DIO request has been demoted. Search for this sub-hook
has been made easier because the appropriate entries have been made to the trace format
file (/etc/trcfmt), shown in Figure 10. So, to determine whether a DIO request has been
demoted review the trace file output looking of the phase “DIO DEMOTED”.
The trace report, shown in Figure 8, shows no DIO DEMOTED events.
Methods for Detecting Demoted Direct I/O Performance Problems
For JFS2 trace using hooks 59B and 59C. Sub-Hook 3 indicates that DIO output has been demoted. The JFS2 hook have been added to the /etc/trcfmt file; therefore, you search for DIO DEMOTED in the trace file.
Figure 9: Track Hooks Ids
For Hook 59B sub-hook 3 the trace format file specified what to print. Search the file for DIO DEMOTED to determine whether DIO request has been demoted.
Methods for Detecting Demoted Direct I/O Performance Problems
Test Case 5: Reading a JFS2 file with an alignment size of 2048 # trace –aj59B,59C # ./directio –b 2048 –f /home/hrlee/proj/test.file # trcstop # trcrpt –o trcrpt.tc4
This example, reads the file using a block size of 2048. As shown in Figure 11, DIO
DEMOTED events have occurred, which indicate that the DIO requests has been demoted,
Methods for Detecting Demoted Direct I/O Performance Problems
Test Case 6: Writing a 1 MB JFS2 file with an alignment size of 2048 # trace –aj59B,59C # directio_w –w 1048576 –b 2048 –f /home/hrlee/test_file # trcstop # trcrpt –o trcrpt.tc7 In this example, we’re writing 1MB to a file using 2048 block size. As shown in Figure 12,
the DIO DEMOTED event has occurred, which means that the DIO write request is using
inefficient cache I/O. In addition, as shown in the filemon output , Figure 13, the system is
not only force to write more data than required, but it also has to read data before each
write. All the operations are synchronous, which result in a significant impact to the
Methods for Detecting Demoted Direct I/O Performance Problems
References
• General Programming Concepts: Writing and Debugging Programs (SC23-4128-08) • Improve Database Performance on File System Containers in IBM DB2 Universal
Database Stinger using Concurrent I/O on AIX – dated August 2004
Methods for Detecting Demoted Direct I/O Performance Problems
printf("Error: Unable to to seek to location %d\n", offset); } /* * write the file */ r = 0 ; do { r = r + write( fd, buffer, blk_size ) ; } while( r < write_size ); exit(0); }