Tracking Performance - Velocity NYC 2013

Tracking PerformancePatrick Meenan@patmeenan

Slides will be available

http://www.slideshare.net/patrickmeenan

Introducing shop-de-nile.com

System Architecture

Product DB

Load Balancer

Load Balancer

Web Server (App)

Web Server (App)

Web Server (App)

Web Server (App)

Web Server (Static)

Web Server (Static)

SOLR FT Search

ProductImages

Reviews DB

Us Them

http://www.memegen.com/meme/dbuj21

New System Architecture

Product DB

Load Balancer

Load Balancer

Web Server (App)

Web Server (App)

Web Server (App)

Web Server (App)

Web Server (Static)

Web Server (Static)

SOLR FT Search

ProductImages

Reviews DB

Varnish

Varnish

MemcacheMemcache

Memcache

Testing Plan● 14 Locations● 3 “Representative” Pages

○ Landing Page○ Search Results Page○ Product Listing Page

● 1 Browser● First View Only● Every 5 Minutes (12 samples per hour)

12,096 Tests Per Day

http://www.memegen.com/meme/ek1v79

CDN test plan

● Test 3 CDN’s● Over 2 Weeks● Compare against:

○ Each Other○ Local Performance○ No CDN

36,288 Tests Per Day

Re-Run Competitive Comparison

● Globally● 20 Page Loads Per Day● 3 Representative Pages● 1,680 Page Loads

http://www.memegen.com/meme/duqjfs

Real User Monitoring (RUM)

5.28 1.38

http://www.memegen.com/meme/9vyywz

Instrument back-endhttp://www.slideshare.net/mikebrittain/metrics-driven-engineering-at-etsy

http://codeascraft.com/

Access Logs -> logster -> statsd -> graphite

Application Performance Measurement (APM)

Cache Hits

Not no much

Testing Plan● 14 Locations● 3 “Representative” Pages

○ Landing Page○ Search Results Page○ Product Listing Page

● 1 Browser● First View Only● Every 5 Minutes (12 samples per hour)

http://www.youtube.com/watch?v=H7PJ1oeEyGg

Go buy an SSD

@crucially

New New System Architecture

Product DB (SSD)

Load Balancer

Load Balancer

Web Server (App)

Web Server (App)

Web Server (App)

Web Server (App)

Web Server (Static)

Web Server (Static)

SOLR FT Search (SSD)

ProductImages (SSD)

Reviews DB (SSD)

Varnish(SSD)

Varnish(SSD)

MemcacheMemcache

Memcache

4.58

Not matching synthetic testing

● Backbone vs last-mile● Long-tail images

http://www.belshe.com/2010/05/24/more-bandwidth-doesnt-matter-much/

http://www.belshe.com/2010/05/24/more-bandwidth-doesnt-matter-much/

Our Testing

http://www.belshe.com/2010/05/24/more-bandwidth-doesnt-matter-much/

Our Users

Testing Plan● 14 Locations● 3 “Representative” Pages

○ Landing Page○ Search Results Page○ Product Listing Page

● 1 Browser● First View Only● Every 5 Minutes (12 samples per hour)

Images always in edge cache

New Experience Testing Plan● 14 Last-Mile Locations● 3 “Representative” Pages

○ Landing Page○ Random Search Results Page○ Random Product Listing Page

● 2 Browsers○ Desktop○ Mobile (actual device)

● First View Only● 20 Samples per day

Measured Load Times still too slow

● Social Sharing Widgets● Ads● Retargeting pixels● Analytics

All already Async and using best practices

Ummm….

● Large drop in revenue● Loss of tracking● Doesn’t “feel” significantly faster

Speed Index FTW

● Measures visual experience● Synthetic Only (WebPagetest)

Speed Index 3rd Party Evaluation

Speed Index FTW

● Measures visual experience● Synthetic Only (WebPagetest)

We got this...it’s only codeonload=”performance.mark(‘aft.Image Loaded’)”...

http://blog.patrickmeenan.com/2013/07/measuring-performance-of-user-experience.html

window.onload:● performance.getEntriesByType(“mark”)● Report latest aft.* as custom time

Roll-your-own

● Send http beacon to beacon server○ All timings as query params○ 204 or transparent 1px gif○ Log requests to access log

● IP, User Agent and Timings all in each record

● access log -> logster -> statsd -> graphite● Profit!

Google Analytics_gaq.push(['_trackTiming', 'UserTimings', 'aft', measuredTime);

Boomerang/Soasta mPulseBOOMR.plugins.RT.setTimer(‘custom0’, measuredTime);

https://gist.github.com/pmeenan/5902672

AFT vs onload

http://www.soasta.com/products/mpulse/

AFT

Median 0.932

95th Percentile 4.141

98th Percentile 7.262

onLoad

Median 2.235

95th Percentile 11.787

98th Percentile 26.72

Custom timings in WebPagetest

Where does that leave us?● Competitive testing with synthetic last-mile

○ After builds pushed to staging/prod

● Performance tracking with RUM custom measurements○ Data can be noisy○ No competitive capability (opportunity)

Where does that leave us?● Competitive testing with synthetic last-mile

○ After builds pushed to staging/prod

● Performance tracking with RUM custom measurements○ Data can be noisy○ No competitive capability (opportunity)

Synthetic testing in CI● PhantomJS + CI● WebPagetest Private Instance + CI

○ https://github.com/marcelduran/webpagetest-api● Graphing/Trending artifacts

○ httparchive○ graphite○ Show Slow○ tsviewdb

tsviewdb

https://github.com/google/tsviewdb

● Trending● Regression Detection

People will optimize for what you measure….

So measure what you want optimized.