MPI Tuning with the Intel® Trace Analyzer and Collector Cluster Software and Technologies Software & Solutions Group.

MPI Tuning with the Intel® Trace Analyzer

and Collector

Cluster Software and Technologies

Software & Solutions Group

2

Copyright © 2008, Intel Corporation. All rights reserved.

MPI Tuning with Intel® Trace Analyzer & Intel® Trace Collector

Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.

Agenda

Introduction

Basic Usage

Trace Collector in depth (lab exercises)

Trace Analyzer in depth (lab exercises)

3




Introduction – What is Tracing?

• Record program execution• Program events such as function enter/exit, communication

• 1:1 protocol of the actual program execution• Sampling gathers statistical information

• Accurate data

• Easily get loads of data

4




Trace Universe

Intel® TraceCollector

Application

TracefileIntel® Trace

Analyzer

5




Components and Interaction

TracesSTF

Intel® Trace Collector Lib

API

Intel® Trace Collector Lib

itcpin instrument

Executable

Application

Compiler

Linker

Instrumented

Executable

pre-load

6




Supported Programming Models

MPI: Message Passing

Languages: C/C++, Java*, Fortran

Multithreading: explicit threads - implicit threads (OpenMP)

Hybrid Models: MPI + OpenMP

7




Supported Environments

Intel® Xeon® Processor

Intel 64 Bit Xeon Processor

Intel Itanium® Processor

Intel MPI, MPICH, SGI* MPT

Linux (RHEL, RHAS, SLES)

SGI* Altix*

Microsoft* Windows*

8




Introduction:Intel® Trace Collector

Key features

Usage

9




Trace Universe – Intel® Trace Collector


Application


Analyzer

10




Intel® Trace Collector - Overview

Event based approach

Low impact on application performance

Provides API to instrument user code

Trace optimized program runs

Analyzes communication layer (default)

11




Event based approach

Event = time stamp + thread ID + description

Function entry/exit

Messages

Collective operations

Counter samples

12




Strengths of Event-based Tracing

Predict detailed program behavior

Record exact sequence of program states – keep timing consistent

Collect information about exchange of messages: at what times and in which order

An event-based approach is able to detect temporal dependencies!

13




Key Features

Catch all MPI events

Strong configuration mechanism• Filters, settings, features

Automatic source-code references

Instrumentation• Rich API• Binary instrumentation (itcpin)• Compiler based (-tcollect)

Fail-safe version

Comparison Feature

MPI Correctness Checking

14




How to use Intel® Trace Collector

Step 1: Generate instrumented binary re–link your application:• Using Intel MPI:mpicc –trace tst.o –o tst.inst

• Using any other supported MPI:mpicc –shlib tst.o $(LFLAGS) -lVT –lmpi –ldwarf –lelf –lnsl –lm –

o tst.inst

Step 2: Produce tracefile

run the instrumented binary for a representative amount of time (reduce initialization influences) on representative data (no corner cases)

Alternative 1: Instrument binary

itcpin –-run –- tst

Alternative 2: Preload the Trace Collector library

mpiexec –trace –n 2 ./tst

15




Activity 1 - MPI Tracing with Trace Collector

Build two simple “hello world” parallel applications

Create and view the program traces

16




Activity 2 – Trace File Generation

Objective: How to generate a tracefile.

This directory contains cpi.c which is a simple example program.

Re-link (or compile and link) with Intel® Trace Collector library.

Run again – check for console message on 'writing tracefile'.

Open tracefile with Intel® Trace Analyzer.

[traceanalyzer cpi_itc.stf]

17




Introduction:Intel® Trace Analyzer in depth

Key features

Usage

18




Trace Universe - Intel® Trace Analyzer


Application


Analyzer

19




Intel® Trace Analyzer - Overview

Enables the user to quickly focus at the appropriate level of detail Offline trace analysis (postmortem)

Use of hierarchical techniques to address scalability in function, time and processor–space

High–performance graphics, excellent zooming and filtering

20




Views and Charts - I

What’s a Chart?

What Charts are provided?

What’s a View?

What’s the purpose of a View?

21




Chart

A Chart is a numerical or graphical diagram

Chart

22




Timelines: Event Timeline

Get impression of program structure

Display functions, messages and collective operations for each process/thread along time-axis

Retrieval of detailed event information

23




Timelines: Qualitative Timeline

Find patterns and irregularities

Display attributes of functions, messages or collective operations as they occur for any process/thread

Retrieval of detailed event information

24




Timelines: Quantitative Timeline

Get impression on parallelism and load balance

Show for every function how many threads/processes are currently executing it

25




Profiles: Flat Function Profile

Statistics about functions

26




Profiles: Call-Tree and Call-Graph

Function statistics including calling hierarchy

• Tree: call-stack

• Graph: calling dependencies

27




Communication Profiles

Statistics about point-to-point or collective communication

Generic matrix supports grouping by several attributes in each dimensionSender, Receiver, Data volume per msg, Tag, Communicator, Type

Available attributesCount, Bytes transferred, Time, Transfer rate

28




View

Helps navigating through the trace data and keep orientation

Every View can contain several Charts

A View on a file is defined by a triplet of• time-span• set of threads• set of functions

All Charts follow changes to View (e.g. zooming)

Timelines are correctly aligned along time

29




View - zooming

30




Flexibility of Views

Several Views can be opened (on the same or on different files)

Location, orientation and size of charts can easily be changed

Entire Views can and individual charts can be cloned and closed

Individual charts can be cloned in own View

31




Activity 7 – Getting Started with Trace Analyzer

In this activity, you will use the Trace Analyzer tool to recreate a particular view.

32




Activity 8 – Getting Started with Trace Analyzer

In this activity, you will use the Trace Analyzer tool to recreate a “tiled” view.

33




Activity 9 – More Charts with Trace Analyzer

In this activity, you will use the Trace Analyzer to recreate more charts. This will familiarize you with many of the charts and features of the Trace Analyzer.

34




Charting and Viewing

Start with

traces/ztomo/out_altixtest_ori.2.stf

Use Chart menu to open

• EventTimeline

• Quantitative Timeline

• Function Profile (Flat Profile)

• Function Profile (CallGraph)

35




Lab 4 - Results

36




Charting and Viewing

Start with

traces/ztomo/out_altixtest_ori.2.stf

Use Chart menu to open• EventTimeline (zoom to very end (left))• Collective Operation Profile

Use Layout menu to get time line to left

Use Context menu in CollOpChart• Colums to show Communicator• Rows to show Processes

37




Lab 4b - Results

38




Grouping and Aggregation

Allow analysis on different levels of detail by aggregating data upon group-definitions

Functions and threads can be grouped hierarchically• Function Groups and Thread Groups

Arbitrary nesting is supported• Functions/threads on the same level as groups• User can define his/her own groups

Aggregation is part of View-definition• All charts in a View adapt to requested grouping

• All charts support aggregation

39




Aggregation Example

40




Lab 10 – Trace Analyzer Function Aggregation

In this activity, you will use the Trace Analyzer tool to aggregate functions to determine how much effect the library zlib has on overall performance.

41




Tagging & Filtering

Help concentrating on relevant parts

Avoid getting lost in huge amounts of trace data

Define a set of interesting data • E.g. all occurrences of function x

• E.g. all messages with tag y on communicator z

Combine several filters:Intersection, Union, Complement

Apply it • Tagging: Highlight messages

• Filtering: Suppress all non-matching events

42




Tagging Example

43




Filtering Example

44




Lab 11 – Trace Analyzer Tagging

In this activity, you will use the tagging feature of Trace Analyzer and highlight a function to determine which is the most expensive caller to the function. You will also use tagging to look for an instrumented function “f” from an earlier Trace Collector exercise.

45




Lab 12 – Trace Analyzer Filtering

In this activity, you will use the Trace Analyzer tool to aggregate functions to determine how much effect the library zlib has on overall performance.

46




Comparison Feature

Provides side-by-side comparison of any two trace files and/or regions

Ability to couple zoom and navigation keys

Addition of several comparison-specific charts, such as

• Function Profile

• Message Profile

• Collective Operations Profile

47




Comparison Feature Optimized/non-optimized Poisson code

MPI reduced to 43% by optimization

48




Comparison Feature cont’d

In comparison view, all charts like

• Function ProfileIn function profile, several settings are possible, e.g. the ‚B/A‘ setting

which shows factors of the second (B) as compared to the first (A) trace file

• Message Profile

• Collective Operations Profile

compare the two comparands.

49




Understanding your code

Parallel Poisson Solver

Example of intuitive parallelization with disadvantageous communication pattern

50




Partial Trace of Poisson Solver

51




Understanding the problem

Blocking border exchange

Pn has blocks until communication between Pn+1 and Pn+2 was completed

Solution: Non blocking communication

52




Non-Blocking Version

53




Detecting Load Imbalance

Mandelbrot set

Example of intuitive parallelization with huge load imbalance

54




Mandelbrot Example Trace

55




Understanding Load Imbalance

Static partitioning assigns contiguous blocks of rows• Workload changes gradually with row-index!

What is the/a solution?• Use modulo operator• Use dynamic master/slave approach

56




Understanding Load Imbalance

57




Lab 13 – Trace Analyzer Load Balance

In this activity, you will use the Function Profile Load balance feature of Trace Analyzer to see how well an application's calculations are spread across processors.

58




Supplemental material

59




Lab 14 – Using Trace Collector to Generate Text-Based Statistics Files

In this activity, you will relink with the Trace Collector Library and set up a configuration file that causes the trace files produced to be readable by text based tools such as vi and awk, while keeping the size of trace files produced to a minimum.

Objective: How to generate a tracefile with only statistic information

Generate file my_vt_config

Set environment variable and build example

Run example and look at stats-file cpi_itc.prot

60




Lab 15 – Using STFTOOL to Consolidate stf Files for Analysis on Other Systems

In this activity, you will use the stftool to consolidate stf files for analysis on other systems. You will use this tool to create a single file containing all trace information. You will then use the tool to expand the single file back into the suite of related stf in another directory.

Objective: Learn about stftool, use single trace file to simplify transfer of storage

Note: Can degrade performance of ITA

Note: ITC config allows creation of single-STF

61




Backup

62




Instrumenting Functions

Generic API to instrument functions

Hierarchical grouping (classes)

Insert probes to

• declare function

• enter/leave the function

63




ITC API: Instrument a function

Allocate a handle for a class name (optionally)int VT_classdef( const char * classname,

int * classhandle )

classname: name of the class

returns handle (int) referencing this class

Allocate a handle for a stateint VT_funcdef( const char * symname,

int classhandle, int * statehandle )

symname: name of the symbol

classhandle: handle of the class (see above)

returns handle (int) referencing this function

64




Lab 3 – Instrument a Function: Trace Collector API

Add lines to the cpi.c source file to instrument a particular function in cpi.c.

Edit cpi.c to instrument function 'f' as of group 'Application'• Use VT_classdef() and VT_funcdef() to declare class/function• Use VT_enter() and VT_leave(), to record function begin and end

Objective: Use of Intel® Trace Collector API to instrument a function

Once instrumented and when the “Application Group” is ungrouped, the function can be seen in various charts in Trace Analyzer

Compile-link-run and view tracefile in Intel® Trace Analyzer• Find events 'f' in the timeline, in the function profile and in the call tree

display

65




Solution – Use of API

#include <VT.h>[...]int application_class; double f( double a ){ static int f_state = 0; if( f_state == 0 ) { VT_funcdef( "f", application_class, &f_state ); } VT_enter( f_state, VT_NOSCL ); [...] VT_leave( VT_NOSCL ); return (result);}int main( int argc, char *argv[]) [...] MPI_Init(&argc,&argv); VT_classdef( "Application", &application_class );

66




Lab 4 – Instrument a Function Trace Collector API, including Source & Line Numbers

Add additional lines to the cpi.c source file to instrument a particular function in cpi.c so.

Once the code is instrumented, the resulting trace enaables us to drill down to the “f” function source and see line number information

67




Lab 5 – Instrument a Function Trace Collector API for for Selective Tracing

Add lines to the cpi.c source file to selectively turn tracing on and off, portions where tracing is performed in an application

68




Lab 6 – Using Trace Collector API to Count Events

In this activity, you will add more lines of code to the cpi.c source file, re-compile and relink with the Trace Collector Library. The changes made will allow us to count events as we define them in the application.

In the example used, we define a counter, f_result, to count updates to the variable named 'result'

69




MPI Tuning with the Intel® Trace Analyzer and Collector Cluster Software and Technologies Software & Solutions Group.

Documents

intel logo

mpi tuning

united states

respective owners

collector cluster software

openmp slide

load slide

programming models mpi