Benchmarking

cs4414 Fall 2013University of Virginia

David Evans

Class 15:Benchmarking

University of Virginia cs4414 2

Plan for Today

• Finish up Internet (from last class)• Midterm Discussion• Benchmarking

17 October 2013

University of Virginia cs4414 3

Measuring NetworksLatency

Time from sending a bit until it arrives seconds (or seconds per geographic distance)

Bandwidth Rate at which can you transmitbits per second

17 October 2013

University of Virginia cs4414 4

$ traceroute -q 1 -w 30 www.busselton.wa.gov.autraceroute to busselton.wa.gov.au (203.41.180.233), 64 hops max, 52 byte packets 1 dd-wrt (192.168.1.1) 11.156 ms 2 c-24-127-51-1.hsd1.va.comcast.net (24.127.51.1) 32.497 ms 3 te-2-3-ur01.charlville.va.richmond.comcast.net (68.85.226.149) 13.971 ms 4 xe-11-2-0-0-sur01.charlville.va.richmond.comcast.net (69.139.165.221) 12.312 ms 5 xe-4-1-2-0-ar02.charlvilleco.va.richmond.comcast.net (69.139.165.65) 12.395 ms 6 pos-1-2-0-0-cr01.ashburn.va.ibone.comcast.net (68.86.91.53) 25.624 ms 7 pos-3-10-0-0-cr01.56marietta.ga.ibone.comcast.net (68.86.86.221) 31.483 ms 8 pos-1-9-0-0-cr01.dallas.tx.ibone.comcast.net (68.86.87.233) 52.515 ms 9 he-0-12-0-0-cr01.losangeles.ca.ibone.comcast.net (68.86.86.117) 83.242 ms10 as4637-cr01.losangeles.ca.ibone.comcast.net (75.149.228.222) 78.134 ms11 i-0-2-0-11.tlot-core01.bi.telstraglobal.net (202.40.149.185) 86.131 ms12 i-0-0-0-0.sydo-core01.bx.telstraglobal.net (202.84.140.5) 287.302 ms13 tengige0-1-0-14.oxf-gw2.sydney.telstra.net (203.50.13.133) 300.060 ms14 bundle-ether2.oxf-gw1.sydney.telstra.net (203.50.6.85) 274.270 ms15 bundle-ether1.ken-core4.sydney.telstra.net (203.50.6.5) 270.694 ms16 bundle-ether10.win-core1.melbourne.telstra.net (203.50.11.13) 275.252 ms17 bundle-ether6.fli-core1.adelaide.telstra.net (203.50.11.90) 405.600 ms18 bundle-ether5.wel-core3.perth.telstra.net (203.50.11.19) 411.510 ms19 gigabitethernet0-1.wel13.perth.telstra.net (203.50.115.151) 406.044 ms20 *

17 October 2013

Does traceroute tell us anything about the bandwidth between here an Busselton, Austrailia?

University of Virginia cs4414 5

What Impacts Bandwidth?

17 October 2013

LAN: Ethernet (97.6% efficient for 12Kb packets)WAN: PPP (99.9% efficient – only 1-2 bytes overhead)

IPv4: up to 99.96% efficient

University of Virginia cs4414 6

Below all this: Physical Layer

17 October 2013

University of Virginia cs4414 7

BandwidthHow much data can you transfer in a given amount of time?

17 October 2013

University of Virginia cs4414 8

Improving Bandwidth

• Faster transmission– Train signalers to move semaphore flags faster– Use something less physically demanding to transmit

• Bigger pipes– Have multiple signalers transmit every other letter at the

same time

• Better encoding– Figure out how to code more than 98 symbols with

semaphore signal– Morse code (1840s)

17 October 2013

University of Virginia cs4414 9

Morse CodeRepresent letters with series ofshort and long electrical pulses

17 October 2013

University of Virginia cs4414 10

Circuit Switching

• Reserve a whole path through the network for the whole message transmission

Paris Toulon

Nantes

LyonBourges

Once you start a transmission,know you will have use of the network until it is finished. But,wastes network resources.

17 October 2013

University of Virginia cs4414 11

Packet Switching

• Use one link at a time

Paris Toulon

Nantes

LyonBourges

Interleave messages – send whenever the next link is free.

17 October 2013

University of Virginia cs4414 12

internetwork

A collection of multiple networks connected together, so messages can be transmitted between nodes on different networks.

17 October 2013

University of Virginia cs4414 13

The First (international) internet

17 October 2013

1800: Edelcrantz links Sweden and Denmark telegraph networks to coordinate defense

University of Virginia cs4414 14

End of First (international) internet

17 October 2013

1801: British attach Copenhagen; Sweden doesn’t help Denmark; network disconnected

University of Virginia cs4414 15

The (capital-I) Internet

Packet Switching: Leonard Kleinrock (UCLA) thinks he did, Donald Davies and Paul Baran, Edelcrantz’s signalling network (1809)Internet Protocol: Vint Cerf, Bob KahnVision, Funding (DARPA): J.C.R. Licklider, Bob Taylor Government: Al Gore

First politician to promote Internet, 1986; act to connect government networks to form “Interagency Network”

17 October 2013

Vint Cerf (in the Rotunda, 2010)

University of Virginia cs4414 16

First Use of the Internet

October 1969: First packets on the ARPANet from UCLA to Stanford. Starts to send "LOGIN", but it crashes on the G.

How impressive is this compared to communications event 3 months earlier?

17 October 2013

University of Virginia cs4414 17

20 July 1969:Live b/w video from the moon, transmitted

live to millions of televisions worldwide

17 October 2013

University of Virginia cs4414 18

Today’s Internet: Bandwidth

Ethernet: up to 100 Mbits/sec

My office: ~50 M bits / sec

UVa Wireless: 57 M bits / sec

Cable modem at home:~ 30 M bits / sec

17 October 2013

University of Virginia cs4414 1917 October 2013

projections

1 Petabyte = 1015 bytes1 Million petabytes = 1 zettabyte(NSA’s Utah facility is storing “Zbytes”?)

University of Virginia cs4414 20

Midterm Discussion

17 October 2013

University of Virginia cs4414 21

Easiest Way to Ace Midterm

17 October 2013

4 people selected #9, but none were bold enough to leave all the other answers blank. (You should only be “especially proud of” your answer to #9 if you are confident enough to leave all other answers blank.)

University of Virginia cs4414 22

How many processes should a web browser create?

17 October 2013

1 process

several process

one per “tab”

several per “tab”

University of Virginia cs4414 23

How many threads should a web browser create?

17 October 2013

1 process

several process

one per “tab”

several per “tab”

University of Virginia cs4414 24

How many processes should a web browser developed in Rust

create?

17 October 2013

1 process

several process

one per “tab”

several per “tab”

University of Virginia cs4414 25

How many tasks should a web browser developed in Rust

create?

17 October 2013

1 process

several process

one per “tab”

several per “tab”

University of Virginia cs4414 26

Your technically unsophisticated (but smart, rich, and curious) uncle complains that his laptop is too noisy since the fan is running all the time. He expects that as someone taking upper-level computing courses at an elite, public university, you should be able to help him. What do you do?

17 October 2013

University of Virginia cs4414 2717 October 2013

Hardware problem

or software problem?

University of Virginia cs4414 2817 October 2013

Hardware problem or

software problem?

University of Virginia cs4414 2917 October 2013

SOSP 2009

University of Virginia cs4414 30

Uncle’s Fan Problems

Could it be caused by deadlock?

17 October 2013

University of Virginia cs4414 31

Fan Removes Heat

• CPU generates heat when it is executing lots of instructions

• At least with my MBP, one thread is not enough: 100% CPU usage – no fan200% CPU usage – light fan800% CPU usage – max fan

17 October 2013

University of Virginia cs4414 32

PerformanceBenchmarking

17 October 2013

University of Virginia cs4414 33

Why Benchmark?

17 October 2013

University of Virginia cs4414 34

Developer Benchmarks

• Find bottlenecks: know what to spend time optimizing

• Measure impact of changes to optimize performance

• Predict what resources you will need to scale service (“healthcare.gov”)

17 October 2013

Goal is a benchmark that represents well the (anticipated) actual usage

University of Virginia cs4414 35

Micro vs. Macro Benchmarks

• Micro Benchmarks– Test one specific operation

17 October 2013

University of Virginia cs4414 36

“Customer” Benchmarks

• Allow fair and accurate comparisons between different solutions

• Standard accepted by all/many vendors

17 October 2013

Goal is a benchmark that represents well the (anticipated) actual usage for “typical” users

University of Virginia cs4414 37

“Cheating” on Benchmarks

17 October 2013

How could you “cheat” on a benchmark?

University of Virginia cs4414 3817 October 2013

1. On the Exynos 5410, Samsung was detecting the presence of certain benchmarks and raising thermal limits (and thus max GPU frequency) in order to gain an edge on those benchmarks, and

2. On both Snapdragon 600 and Exynos 5410 SGS4 platforms, Samsung was detecting the presence of certain benchmarks and automatically driving CPU voltage/frequency to their highest state right away. Also on Snapdragon platforms, all cores are plugged in immediately upon benchmark detect.

University of Virginia cs4414 3917 October 2013

Everyone (except Google and NVidia) is cheating!

Samsung is just better at it than anyone else.

University of Virginia cs4414 40

Is It Possible to Prevent Benchmark “Cheating”?

17 October 2013

University of Virginia cs4414 41

Benchmarking Zhtta

17 October 2013

University of Virginia cs4414 4217 October 2013

http://www.comlore.com/redist_files/httperf-www2007-slides.pdf

University of Virginia cs4414 43

Example

17 October 2013

httperf --hog --server=localhost --port=4414 \ --num-conns=1000 --rate=10

10 requests per second

Should you run httperf on the machine running your server?

University of Virginia cs4414 4417 October 2013

Maximum connect burst length: 1

Total: connections 1000 requests 0 replies 0 test-duration 99.901 s

Connection rate: 10.0 conn/s (99.9 ms/conn, <=1 concurrent connections)Connection time [ms]: min 0.0 avg 0.0 max 0.0 median 0.0 stddev 0.0Connection time [ms]: connect 0.1Connection length [replies/conn]: 0.000

Request rate: 0.0 req/s (0.0 ms/req)Request size [B]: 0.0

Reply rate [replies/s]: min 0.0 avg 0.0 max 0.0 stddev 0.0 (19 samples)Reply time [ms]: response 0.0 transfer 0.0Reply size [B]: header 0.0 content 0.0 footer 0.0 (total 0.0)Reply status: 1xx=0 2xx=0 3xx=0 4xx=0 5xx=0

CPU time [s]: user 27.52 system 72.40 (user 27.6% system 72.5% total 100.0%)Net I/O: 0.0 KB/s (0.0*10^6 bps)

Errors: total 1000 client-timo 0 socket-timo 0 connrefused 1000 connreset 0Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0

University of Virginia cs4414 45

Macrobenchmarks

Need a workload that represents typical load

17 October 2013

University of Virginia cs4414 4617 October 2013

http://ita.ee.lbl.gov/html/traces.html

University of Virginia cs4414 4717 October 2013

http://www.hpl.hp.com/techreports/1999/HPL-1999-35R1.pdf

University of Virginia cs4414 4817 October 2013

University of Virginia cs4414 4917 October 2013

1998Image from http://www.hpl.hp.com/techreports/1999/HPL-1999-35R1.pdf

University of Virginia cs4414 50

Macrobenchmarks

Need a workload that represents typical load

17 October 2013

Synthetic Workload Replayed Workload

University of Virginia cs4414 51

Weilin’s Test Workload

17 October 2013

> dd if=/dev/urandom of=5K.bin bs=5K count=1> dd if=/dev/urandom of=5M.bin bs=5M count=1> dd if=/dev/urandom of=10M.bin bs=10M count=1> dd if=/dev/urandom of=20M.bin bs=20M count=1> dd if=/dev/urandom of=40M.bin bs=40M count=1> dd if=/dev/urandom of=80M.bin bs=80M count=1> dd if=/dev/urandom of=512M.bin bs=512M count=1

http://www.cs.virginia.edu/~wx4ed/cs4414/ps3/zhtta-test-urls.txt

/5K.bin/index.htm/5K.bin/5K.bin/80M.bin/5K.bin/5K.bin/5K.bin/5K.bin/index.htm/5K.bin…

University of Virginia cs4414 52

Testing Zhtta

17 October 2013

> httperf --server localhost --port 4414 --rate 60 \ --num-conns 60 --wlog=y,./zhtta-test-urls.httperfTotal: connections 60 requests 60 replies 60 test-duration 32.588 s

Connection rate: 1.8 conn/s (543.1 ms/conn, <=46 concurrent connections)Connection time [ms]: min 0.4 avg 1237.8 max 31954.9 median 563.5 stddev 4132.5Connection time [ms]: connect 0.0Connection length [replies/conn]: 1.000

Request rate: 1.8 req/s (543.1 ms/req)Request size [B]: 67.0

Reply rate [replies/s]: min 0.0 avg 2.0 max 11.6 stddev 4.7 (6 samples)Reply time [ms]: response 1215.7 transfer 22.1Reply size [B]: header 71.0 content 16640936.0 footer 0.0 (total 16641007.0)Reply status: 1xx=0 2xx=60 3xx=0 4xx=0 5xx=0

CPU time [s]: user 7.40 system 25.12 (user 22.7% system 77.1% total 99.8%)Net I/O: 29920.6 KB/s (245.1*10^6 bps)

Errors: total 0 client-timo 0 socket-timo 0 connrefused 0 connreset 0Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0

University of Virginia cs4414 5317 October 2013

Single-task server

Zhttpto

Zhtta with FIFO

Zhtta with SRPT

Zhtta with SRPT and caching

0 5 10 15 20 25 30 35 40

Test Total Duration (seconds)

Is our memory caching helping?

University of Virginia cs4414 5417 October 2013

Single-task server

Zhttpto

Zhtta with FIFO

Zhtta with SRPT

Zhtta with SRPT and caching

0 5000 10000 15000

Average Response Time (ms)

Single-task server

Zhttpto

Zhtta with FIFO

Zhtta with SRPT

Zhtta with SRPT and caching

0 5 10 15 20 25 30 35 40

Test Duration (seconds)

How good is this?

University of Virginia cs4414 55

Possible to do Much Better!

17 October 2013

Single-task server

Zhttpto

Zhtta with FIFO

Zhtta with SRPT

Zhtta with SRPT and caching

Apache2

0 5 10 15 20 25 30 35 40

Test Duration (seconds)

University of Virginia cs4414 5617 October 2013

Single-task server

Zhttpto

Zhtta with FIFO

Zhtta with SRPT

Zhtta with SRPT and caching

Apache2

0 2000 4000 6000 8000 10000 12000 14000 16000

Average Response Time (ms)

0.2 ms

httperf is measuring “Response Time” as latency to receive first byte!

University of Virginia cs4414 57

Charge

• You should definitely be able to show improvements using Weilin’s benchmark (posted with today’s notes)

• To prevent overly-targeted optimizing, we will also test your servers with a different workload

• Demo will separately check correctness– e.g., if a served file changes, should serve new file

not cached old one (within 10 seconds)– To prevent “cheating”, we may look at your code

17 October 2013

PS3 is due less than one week from today! Still, no one has claimed OpenPiazza!