2015 02 24 lmtv baselining

© 2015, The Technology Firm www.thetechfirm.com

Baselining

Getting Started

From The: Throughput and Latency Class

Tony Fortunato, Sr Network Performance Specialist

© 2015, The Technology Firm WWW.THETECHFIRM.COM

Why Baseline?? I’m surprised how many times I see people instructed to ‘create a baseline’ with no

real purpose or goal in mind.

This makes your job difficult and may turn into one of those never ending requests.

Here’s some examples of why you may want to create a baseline:• Educational exercise;

• to learn more about your application or protocols• Understand how your application behaves for future reference or tuning

• Measure application performance to properly predict network requirements• Document impact of network or application changes• To create an effective troubleshooting methodology• Determine application behavior to properly configure network equipment

(security)

Do not proceed without a without a clear goal or target!

Same applies to general tool use!!

2

© 2015, The Technology Firm WWW.THETECHFIRM.COM 3

Bandwidth Exercise

Here’s the scenario: Copy a file to and from your server and record how long it takes.

• 12 MB file on a 10 Mb Ethernet• 120 MB file on a 100 Mb Ethernet• 1,200 MB file on a 1 Gb Ethernet• 67.5 MB on 54 Mb WiFi

• Theoretically, all these scenarios should take approximately 10 seconds. How did you do? What if it took you 11 vs 55 seconds?

• Basic formula is bits/8 x 10 seconds which is strictly theoritical

How would you measure the network load you created during this test?o Using SNMP from your switch port?o Span or mirror your port to a Protocol Analyzer or other types of testers or probes?o Use Microsoft’s Perfmon or other OS utilities (if available)?

o Does it matter if the server is local, remote?o What about protocols?o LAN vs WiFio Disk vs RAM


Common Baselines

• Baselining is often a forgotten art and most people don’t baseline for the following reasons:o When should I start?o What should a baseline look like?o What if someone actually reads it and wants more information?o Why bother? Everything changes so fast around here.o I don’t know how to baseline.

• Baselines should be:o Clearly defined. o For example PC bootup baseline, Login baseline, Application login Baseline, throughput

or Network Upgrade Baseline.

• As long as your goal is clear and the methodology is documented your baseline is correct.

• If you have performed a baseline correctly, you will typically find problems to fix along the way or settings to tune.


Baselining Options

The first step to a successful baseline is to determine what you want to baseline.

Examples:Network

• Record the load, protocol distribution and other network characteristics

Network Equipment• Log equipment specific details such as temperature, port errors

Workstations• Document protocols, application dependencies

Measurement point and methodology must be documented so you can properly duplicate or retest


Synthetic Transaction Performance

• Several products on the market use “synthetic” transactions.

• Synthetic transactions are modeled after popular applications. i.e. ftp, Telnet, HTTP, etc… These synthetic transactions are transmitted across a network to estimate transaction response time.

• Results only as accurate as the synthetic transaction and test PC’s.

• In many cases multiple workstations can be simulated.

• The accuracy of the synthetic transaction versus performing the same transaction [I.e. ftp] comes into question since the test nodes may not properly represent the production nodes specifications or load.

• Ensure that these tests are not executed solely from RAM to include realistic disk latency


Protocol Analyzer - Performance Measurement

• Data is gathered using existing traffic.

• Results are what is really happening on the wire.

• Does not require the introduction of additional traffic onto the wire.

• Takes into account production server/client loads, network configuration and any undocumented variables.

• Analysis points must be selected carefully to provide accurate results.

• This approach is the most labor intensive and requires a high level of protocol analysis understanding.

• Try to produce reports from your trace files to better explain and understand your findings.


Network Configuration Example

Client Switch

Switch

Switch

Switch

Server

Where would you go for your capture packets?

B

D

CA X


Review

• There is no single correct means of measuring network performance.

• Not only must the proper tool be selected for the job, but the proper methodology must be used along with the tool.

• In some cases you may want to try to document “in what direction is the main source of latency?”

• Document your methodology as well as your findings to promote consistency.

• Always re-measure after any changes.

• In many cases, you should take more than one measurement.• i.e. I try to take 5, drop the high, low and average the three.

• Automating or documenting tasks will ensure each test is performed identically.


Where Do You Start?

• When you baseline or troubleshoot where do you start?

• It is quite common for a help desk person to ping a server with the application that a client may be complaining about.

• Is a ping the same as running the application?

• Do you dismiss the issue as a local issue since no one else has complained … YET!!!

• Do you start troubleshooting near the server, on the Gig backbone, or near the client?

• I typically start investigating problems and baselines from the clients’ perspective.


Application Analysis Steps

• Baseline the application.

• Review the trace files.

• If required, predict the application performance.

• Measure and compare the application performance in production.

• Troubleshoot the application if necessary.


Baseline the Application

• The purpose of this step is to collect all of the traffic that crosses the network from the start of the application to the end of the application. This traffic will be divided up into individual tasks for analysis.

• It is imperative that every step and the EXACT configuration and location of all equipment is documented properly.

• This step directly impacts the remaining process, so be thorough.

• Sample of items to document:o Segment Name.o Switch Identifier, Slot and Card.o Note if mirror ports or hubs are used.o Time/Date.o Source Station IP, MAC Address, Protocols loaded, Operating System Version, Application

Version.o Script of what the operator will do in detail. Screen shots are optimal.


Data Collection And Intercept options

Capturing the data can be done many different ways, but again, the methodology and tool must be documented.

You can use the following methods to create a baseline:Netflow, sflow, etc..SNMPServer or pc specific statsPacket capture application installed

• On server• On client• On device on network path

Packet intercept options• Span, or similar commands• Hub – 10/100 HD connections• TAP


• Set up Analyzer to capture all traffic to and from the test workstation.• The Hub configuration is only applicable for Half-Duplex configurations and when you

want to observe application behavior or protocols• Full Duplex configurations require a full-duplex tap or mirror/span port.

14

HUB

SwitchTest Workstation

Protocol Analyzer

Capturing the Application Traffic – HUB Example

SERVER


• Setup a filter to capture all traffic to and from the MAC address of the workstation.

• Why the MAC?

Filter On Specific Relevant Data


• Create a worksheet that outlines a script of tasks to be performed.

• This will ensure consistency.

• Provide columns for starting and ending frame numbers.

16

Create a Script

Description StartFrame

EndFrame

Launch Application via Icon [app.exe].Enter Login Name [Joe Smith]Enter Login Password [don’t document]Select Account Query from Main ScreenQuery Account 1234

Wireshark


Record Frame Numbers

• Before each task is performed, note the frame number on the analyzer.

• After the task is complete note the frame number.

• Creating a packet bookmark – like a ping between tasks is always helpful

• Repeat this process for all items on your script.

• Make sure the person running the application waits for your signal before they move on to the next task.

• After the testing is complete, go through the capture file with the worksheet.

• Try to resist the impulse to start analyzing data since it will distract you.

Description Start Frame End FrameLaunch Application via Icon [app.exe]. 1 111Enter Login Name [Joe Smith] 112 121Enter Login Password [don’t document] 122 144Select Account Query from Main Screen 145 222Query Account 1234 223 332


Reporting Your Results

• Now that you have gathered and calculated the traffic per task, prepare a chart with your findings.

• The only other variable to include is how many clients will be using this application.

• This is important because you may be asked to predict load based on a number of clients that will be accessing this application.

Description Start Frame End Frame BytesLaunch Application via Icon [app.exe]. 1 111 120,000Enter Login Name [Joe Smith] 112 121 10,000Enter Login Password [don’t document] 122 144 10,120Select Account Query from Main Screen 145 222 60,000Query Account 1234 223 332 132,022


Trace File Review

• Analyze each task associated with the application to determine the processing time consumed by the client and each of its dependencies.

• Note if any additional servers, protocols or nodes are required to complete any tasks.

• You may want to graphically display these results.

Application X

0

1000020000

3000040000

5000060000

70000

3:3

2:1

7

3:3

3:2

6

3:3

3:3

7

3:3

3:4

8

3:3

3:5

9

3:3

4:1

0

3:3

4:2

1

3:3

4:3

2

3:3

4:4

3

3:3

5:4

7

3:3

6:0

0

3:3

6:1

1

3:3

6:2

5

3:3

6:3

7

3:3

6:4

8

3:3

7:0

1

Time

Bit

s/S

ec

Login QueryExcel Chart


Reporting Your Results

• Sample chart comparing various protocols and applications copying the same file.

File Copy Comparison

-

1,000,000

2,000,000

3,000,000

4,000,000

5,000,000

6,000,000

7,000,000

8,000,000

9,000,000

10,000,000

1 2 3 4 5 6 7 8 9 10 11 12

Seconds

bit

s

smb ip copy smb ipx copy smb nb copy http copy ftp copy


Bootup Configuration Baseline

Goal:• To record the behavior of a PC 'on the wire' when it is powered on. The client does

not to be present since no login is required.

• Any dependencies, protocols and load will be documented.

Methodology: • Place your protocol analyzer either on the same hub as the workstation or configure

some form of port monitoring.

• Be careful if you set up a MAC/DLC filter based on the target workstation's hardware address. You may miss some packets, like some DHCP replies.. If DHCP is used and you wish to use a MAC filter, include the bootp/DHCP protocol.

• After powering the workstation on ensure that all background services are loaded by simply observing the hard drive light. When the hard drive light 'settles down', and the frame counters stop, you're done.


Simple Baseline Example Simply go to a folder that is located on your network drive and search for something

common. This was a baseline that I did for a client who complained about performance when they searched for documents.

In my example, I started a capture and navigated to my articles archive and searched for the word Cisco.

The goal of this exercise was to record which protocols are used and how much traffic is generated.

This type of test allows us to understand how remote users may experience this same task which they perform regularly.

22


Test Configuration

23

Search Articles on O: drive folder for the word Cisco

Capture filter with server MAC address

10.44.10.94

10.44.10.171


Sample Results

24


Protocol Breakdown

25

Note TCP port 445 was used for this test


Document Reported Issues

26

Nothing wrong with an opinion as long as you remember that’s all it is


Packet Slicing Since we are not examining the payload, we can packet slice, or only save the first

128 Bytes

Slicing the trace file makes the file size much smaller which allows Wireshark, or other tools to process it much quicker.

This will provide all the addressing and enough of the protocol for this exercise

I don’t know of a way to slice a trace file from within Wireshark after I’ve captured it and simply use the editcap utility that Wireshark provides

"\Program Files\Wireshark\editcap.exe" -s 128 "search for cisco for mapped drive.pcapng" "search for cisco for mapped drive sliced.pcapng"

In this example, the trace file went from approximately 600 MB to 98 MB

27


Conclusions, Questions, or Recommendations

Can a different protocol be used instead of SMB

Do the results change if you chose a subfolder and perform a similar search

What effect does IPV6 have

Since this department does this operation often, can they create an application to make this more efficient

Would remote clients require internet access with significant symmetrical bandwidth

28

© 2015, The Technology Firm www.thetechfirm.com

Baselining

Getting Started

Thanks for watching

Tony Fortunato, Sr Network Performance Specialist

2015 02 24 lmtv baselining

Technology

network upgrade baseline

application login baseline

successful baseline

o span

o i dont

people dont baseline

technology firm

network load