Quantifying The Mobile Difference

Akamai Confidential

Guy Podjarny @guypodhttp://www.guypo.com/

Quantifying the Mobile Difference

http://twitter.com/guypod

http://blaze.io/blog/

©2012 AkamaiFaster ForwardTM

Mobile Isn’t Desktop

• It’s different in many ways…• This presentation focuses on sizing the gap

•How big is the gap between Mobile & Non-Mobile?•How much does it impact performance?

• We can’t cover everything, so we’ll focus on three topics:•Mobile CPU != Desktop/Laptop CPU•Cellular Networks != Broadband Networks•Mobile Browsers != Desktop Browsers


Akamai Confidential

Mobile Computing Power!=

Desktop/Laptop Computing Power


Apple Laptop vs. Mobile, CPU/RAM


Fast Mobile Browsing is all the rage


Better CPU matters in Benchmarks

http://www.anandtech.com/show/5810/samsung-galaxy-s-iii-performance-preview

Quad Core, ~1.5 Ghz

Dual Core, ~1 Ghz


Perf of iOS5 across different hardware

25% Better

37% Better

10x Better


Test Time!


Test Time!

• Question:Does mobile computing power impact page load time?

• Tested Sites: Alexa US Top 100• Test Tool: Mobitest• Details:

•Test iPhone 4, iPhone 4S and iOS Simulator on 2-yr old MacBook Pro•Note: iOS Simulator uses the actual hardware•Tested overnight•High speed Cable connection, WiFi


Results: Hardware Load Time Impact

MeasureiPhone 4

vs. iPhone 4S

iPhone 4S vs.

iOS Simulator

Sunspider 37% 90%

Load Time 15% 48%


CPU Difference - Conclusions

• Mobile CPU is still underpowered compared to desktop• It indeed impacts web performance, even for regular sites• Impact on load time is lighter than on benchmarks

•Roughly 25-50% of impact on benchmarks

• Computing power is more than just CPU•At least GPU and RAM also have an impact•Stick around for Ariya’s Presentation!

• Recommendation: Test on old devices too

Akamai Confidential

Cellular Network!=

Broadband Network


Claim: Cellular Networks are SSSLLLOOOOWWWW

Low Bandwidth

High Latency

High Packet Loss

Volatility


Bandwidth

http://www.flickr.com/photos/pasukaru76/4228438307/


Cellular Network Speed – WikipediaProtocol Max Downlink Max Uplink

GPRS 56-115 Kbps

CDMA200 (1X) 153.6 Kbps

Edge 236.8 Kbps

3G Bar 2 Mbps stationary client384 Kbps moving client

EV-DO 2.5-4.9 Mbps 0.15-1.8 Mbps

EDGE Evolution 1.6 Mbps 0.5 Mbps

HSDPA 2 Mbps 200 Kbps

HSPA+ 14.4 Mbps

4G Bar 1 Gbps stationary client100 Mbps moving client

HSPA+ Release 11 21-672 Mbps 5.8-168 Mbps

LTE 100-300 Mbps 50-75 Mbps

WiMax 37-365 Mbps 17-376 Mbps


You only get those speeds it if you’re this guy…

http://www.flickr.com/photos/23081877@N02/2798378080/


PCWorld/Novarum

• 3G•Download: 0.59 – 3.84 Mbps•Upload: 0.56 – 1.44 Mbps

• 4G•Download: 2.81 – 9.12 Mbps•Upload: 0.97 – 5.86 Mbps

• Date: Apr, 2012

Source (PCWorld): http://bit.ly/JX7MQw

http://bit.ly/JX7MQw



Mobile Network Speed, Akamai SOTI, Q4 2011

http://www.akamai.com/stateoftheinternet/


Average Broadband Speeds


http://www.netindex.com/download/






Cellular Network Speed – Conclusions

• Theoretical limits are NOT the reality• Cellular bandwidth definitely less than broadband• There is no single number for cellular network speed

• But we need SOME numbers for testing…• So here’s a SWAG at numbers to use for testing

•Merging many data points, and rounding numbers for convenience

Test Case Download (Kbps) Upload (Kbps)

Poor Cell Network 400 250

Average Cell Network 1200 600

©2012 AkamaiFaster ForwardTMLatency


Broadband Latency (FCC)

ConnectionType

RTT

Fiber 17ms

Cable 28ms

DSL 44ms

http://www.fcc.gov/measuring-broadband-america

“Fiber-to-the-home services provided 17 ms round-trip latency on average, while cable-based services averaged 28 ms, and DSL-based services averaged 44 ms.”


Ofcom study, UK Mobile Networks, May ‘11

http://stakeholders.ofcom.org.uk/binaries/research/telecoms-research/bbspeeds2010/Mobile_BB_performance.pdf

Static Tests:

User Panel:

“Latency… was an average of 117 milliseconds. Nearly two thirds of tests took more than 100 milliseconds”

“Average latency was 192 milliseconds and around a third of tests delivered latency of higher than 200 milliseconds”


Cisco Cloud Index, 2010-2015

“Global average fixed latency is 125 and average mobile latency is 290… Global mobile consumer average latency is 307.3 ms”


Yep, Mobile Latency is different

• Chart showing latency on Mobile vs Broadband, Ofcom numbers


Cellular Network Latency - Conclusions

• Not a pretty sight…• MUCH higher than broadband latency• Measured averages around 100-300 ms

• Updated table for suggested testing numbers:

Test Case Download

(Kbps)

Upload (Kbps)

Latency (ms)

Poor Cell Network 500 250 300

Average Cell Network 1200 600 200


Test Time!


Test Time!

• Question:How much impact does the difference in Bandwidth & Latency have on web performance?

• Tested Sites: Alexa US Top 100• Test Tool: WebPageTest• Details:

•Increase Latency, Download & Upload Bandwidth separately•Set no limits on other two variables•Run each test 3 times, use median result


Download BW Impact on Load Time

DSLCable FIBER

Poor Cell Conn

Avg Cell Conn


Upload BW Impact on Load Time

DSL

Cable FIBER

Poor Cell Conn Avg Cell Conn


Latency Impact on Load Time

DSL

CableFIBER

Poor Cell Conn

Avg Cell Conn


Bandwidth/Latency Difference - Conclusions

• Upload difference doesn’t matter much• Download difference DOES matter• Latency difference matters the most

• Tip #1: Test using Cell Network speeds•Know what you’re getting

• Tip #2: Reduce number of requests•Especially blocking requests, e.g scripts, CSS

• Tip #3: Reduce number of bytes


Packet Loss


Packet Loss – Broadband vs Mobile

Broadband

Mobile

Average Packet Loss

0.017%

0.6%


Packet Loss, Akamai/MIT Research

• Tested 149 Million connections in 51 countries•Compared PL for mobile networks from non-mobile networks

• Most connections have no packet loss (Median PL is 0%).•75th percentile usually zero too

• Average connection packet loss: 2.5%•Fixed network average is 2.2%

• Implies connections are usually fine, but sometimes terrible•The average PL for the non-zero half is 5%

• Suggests “small” connections have higher Packet Loss•HTTP Connections are often small

http://pam2012.ftw.at/papers/PAM2012paper6.pdf




Packet Loss - Conclusions

• Packet Loss is significantly higher in Mobile, But…•It’s small (even in Mobile)•It usually doesn’t happen (even in Mobile)

• Smaller connections lose more packets

• Recommendations:•Don’t worry about Packet Loss too much…•Include some Packet Loss in your tests•Ideal: Lots of caching, good user-feedback on error and no SPOFs

SWAG Test Case Download

(Kbps)

Upload (Kbps)

Latency (ms)

Packet

Loss

Poor Cell Network 500 250 300 1%

Average Cell Network

1200 600 200 0%


Apple Network Link Conditioner

850/420 Kbps190ms RTT 780/330 Kbps

200ms RTT1% Packet Loss

780/330 Kbps200ms RTT


Network Link Conditioner in iOS 6? (via @jedisct1)


Volatility


Different protocols lead to different speeds

http://www.ericsson.com/res/docs/2012/traffic_and_market_report_june_2012.pdf


Signal Strength affects download speed

http://stakeholders.ofcom.org.uk/binaries/research/telecoms-research/bbspeeds2010/Mobile_BB_performance.pdf


Different Carriers have different average speeds

Source: Ofcomhttp://stakeholders.ofcom.org.uk/binaries/research/telecoms-research/bbspeeds2010/Mobile_BB_performance.pdf

Source (PCWorld): http://bit.ly/JX7MQw




Volatility - Conclusions

• Stop Guessing! •You can’t predict your user’s speeds, today or in the future

• Little value in doing something “only on cellular”

• Strategy #1: Aim at the lowest common denominator•Design for a usable experience in a slow network•Enhance the page when possible

• Strategy #2: Adapt to Network Conditions•E.g. reduce image quality when conditions are bad•Not easy to do, but doable•Check your options with your Server/Host/CDN


Cellular Networks - Summary

• Cellular Networks are kinda crappy•Can’t live with ‘em, can’t live without ‘em…

• You can assume latency will be high•And download won’t be too awesome

• Don’t worry about Packet Loss & Upload rate•You have bigger things to worry about

• Make Peace with it – you can’t predict cellular network speed•Best bet is to aim low, and adapt to network conditions

SWAG Test Case Download

(Kbps)

Upload (Kbps)

Latency (ms)

Packet

Loss

Poor Cell Network 500 250 300 1%

Average Cell Network

1200 600 200 0%

Akamai Confidential

Mobile Browser!=

Desktop Browser


Actually, they’re not that different…

• Mobile & Desktop browsers are actually similar• Many share the same underlying engines (e.g. WebKit)• Some share the browser itself (e.g. Safari, Chrome, Firefox)

• How are they different, then?•HTTP Pipelining•Connection Models•Cache Sizes •Fragmentation


HTTP Pipelining


Mobile Software: Pipelining

• HTTP Pipelining is around since HTTP 1.1•Send multiple requests on connection before receiving response•Most useful in high latency environment

• Big in Mobile•All big mobile browsers

•iOS 5+, Android, Opera•Hardly used on Desktop

•Opera Only (Default)•Chrome, Firefox (Opt-in)

49


Network Capture

• Samsung Galaxy S•Max Conn: 12•Conn Per Host: 12•Max Piped Reqs: 6•Max “In-Flight” Reqs: 72 (!!!)

• Full Details: http://www.guypo.com/technical/http-pipelining-request-distribution-algorithms/

50

http://www.guypo.com/technical/http-pipelining-request-distribution-algorithms/






Test Time!


Test Time!

• Question:Does HTTP Pipelining help make websites faster?

• Tested Sites: Alexa US Top 500• Test Tool: WebPageTest• Details:

•Use Firefox agent, which supports pipelining•Turn pipelining on and off•Set bandwidth to DSL, Cable & FIOS•Increase Latency to see if impact changes


Test Results – Pipelining has little effect…


Why doesn’t pipelining have an effect?

• Are you sure pipelining was used?•Yes… confirmed via tcpdump and various other metrics

• I don’t know for sure, but I can guess…•18 domains referenced on a page (average)•9 domains used only for one request (average)•3-4 domains used only for two requests (average)•Browsers prefer new connection to piping a request•Other bottlenecks…

• Saw similar results for SPDY in recent test•Details: http://bit.ly/MptaBV

• Conclusion: Not much you can/should do…

http://bit.ly/MptaBV

http://bit.ly/MptaBV

Akamai Confidential

Summary


Summary

• Those were A LOT of numbers…• Mobile is truly different, and it matters for web performance

•Who saw that coming?!

• Main differences for Web Performance:•CPU delta, but its effect is not linear•Network delta in Latency & Download BW, not upload/Packet-Loss

• Main takeaways for web developers:•Test performance on old devices too•Test with at least average and poor cell network performance•Provide a decent experience in the worst of conditions

• Remember - we only looked at a handful of aspects…

Akamai Confidential

Guy Podjarny @guypodhttp://www.guypo.com/

The Mobile Difference – In Numbers

Thank You!Questions?

http://twitter.com/guypod

http://blaze.io/blog/

Quantifying The Mobile Difference

Technology

mobile latency

latency faster forwardtm

akamai soti

cpuram faster forwardtm

lyjx7mqw faster forwardtm

mobile network speed

mobile difference

mobile vs broadband