SA 092711 Network Benchmark Study - Report Preview

7/31/2019 SA 092711 Network Benchmark Study - Report Preview

1/24

October 3, 2011, Vol. 7 No. 1112 PREVIEW

PREVIEW EDITION


2/24

InIIal fdbak rIvd In hfIrs fw hors afr rlasInghIs rpor Inldd hfollowIng ommns:

This is a real differentiator well worth it.

Marketing Director, inrastructure supplier

Very detailed and above and beyond my expectations.CTO, tier two operator

Great report as alwaysCTO, leading mobile operator

High impact stuff Nobody even comes close to delivering this kind ofdata. Nice, nice move

Marketing Director, inrastructure supplier

YOU kick $@tt This is great stuffManaging Director, fnancial institution

Great report again.Senior Technical Fellow, inrastructure supplier

Great stuffwell thought out and most of all fair.

Network Services, leading mobile operator


3/24

3 October 3, 2011 | Signals Ahead, Vol. 7, Number 1112 PREVIEW

1.0 xecutie suy

Tis document contains a highly redacted executive summary a complete

table o contents, and our complete test methodology (Chapter 8 rom the

main report) or a Signals Ahead research product that we published on

September 28th. Additionally, it provides a summary o past topics that we

have covered in Signals Ahead and a list o likely topics that we plan to

tackle in the coming months. Te 155-page report contains 125 gures and

tables, with many o the gures and tables consisting o multiple parts. Tis

report can be purchased separately or $1,995 or it is included with any paid

corporate subscription to Signals Ahead.

For reasons that escape us, and which call into question our sanity, we took it upon ourselves

to drive test al l o the next-generation broadband wireless technologies in order to determine the

relative perormance dierences o each technology in a side-by-side comparison. o make matters

worse, but nonetheless more interest ing, we decided to include as many vendor + technology

permutations as possible. We have visited numerous markets to conduct these tests, including

Phoenix, Dallas, Kansas City, San Francisco, San Jose, Oakland, Myrtle Beach, SC, Charleston

SC, and Houston. As we wind down the data collection phase o this study we have transerred

well more than 500GB o data on the various networks that we have tested and driven literally

hundreds o miles.Operators in this study include A& (HSPA+ and LE), Clearwire (LE 2x20MHz and

Mobile WiMAX), -Mobile USA (DC-HSDPA and HSPA+), and Verizon Wireless (EV-DO

Rev A and LE). Inrastructure suppliers in this study include Alcatel Lucent (EV-DO Rev A

HSPA+ and LE), Ericsson (DC-HSDPA, EV-DO, HSPA+ and LE), Huawei (LE and Mobile

WiMAX), Nokia Siemens Networks (DC-HSDPA, HSPA+ and Mobile WiMAX), and Samsung

(Mobile WiMAX). With the exception o mobile operators who provided us with ree and unlim

ited access to their networks, this study was entirely unded by Signals Research Group without any

involvement rom the aorementioned companies.

In the rst volume o a special three-part series we oer high-level results about how each tech

nology perorms. In the second volume we will analyze results rom separately collected user experi

ence test scenarios and in the third volume we will take a deep dive into the results in order toanalyze important technology enablers, such as 64QAM and MIMO, and to identiy inrastructure

supplier dierentiators that currently exist.

Our ability to collect and analyze the network perormance data would not have been possible

without the support o Accuver, who allowed us to use its suite o network drive test tools, including

its recently released XCAL-MO network benchmarking tool and XCAL-M drive test solution, as

well as its XCAP post-processing sotware to analyze the results. We have used the Accuver tools

several times in the past or various Signals Ahead reports and we have grown quite ond o their

capabilities and their ease o use. In particular, or the most recent round o network benchmark

we e tee ee t 500gb t

te iu et tte e tete ie

itey ue ie.

we eee accueuite et ietet t it t-cei te t

yze te eut.


4/24


testing, the companys XCAL-MO tool proved to be an invaluable asset and without it, we would

have ound it next to impossible to accomplish our objectives.

For the rst time, we are now able to oer their tools with our services or commissioned-based

projects on behal o operators, government regulators, vendors, trade associations or other inter-

ested parties on a global basis. We look orward to discussing such opportunities with anyone that

is interested.

Although the actual report provides very comprehensive and in-depth analysis o each network/

technology conguration in a ourteen page executive summary, we can oer some very high-level

takeaways in this report preview.

key hi-lee ey ( y) icue te i:

1) By and large, all technologies perormed as expected and as advertised by the operators. However,

none o them qualiy as IM- Advanced technologies.

2) Operators control their own destiny and the decisions that they make can greatly alter the peror-

mance capabilities o their chosen network technology. We identied several key shortcomings

in certain operator deployments which had a meaningul impact on perormance relative to the

capabilities o the technology.

3) Latency is an area where all technologies/networks could show meaningul improvement amere 40ms o latency separates the best perorming technology rom the worst perorming tech-

nology. Interestingly, we observed the lowest latency (27ms) on -Mobiles DC-HSDPA network

during early morning rush hour trac.

4) Te perormance o the network depends on the capabilities o the chipset/device, which in some

cases we ound to be lacking.

5) Once normalized or channel bandwidth and MIMO, not to mention taking into consideration

network loading, the perormance dierences across all technologies were relatively modest in

the downlink. In a ew cases, DC-HSDPA outperormed LE, even without making these

adjustments and EV-DO Rev A outperormed HSPA+. Uplink throughput is an area where LE

generally has a big advantage over its peers.

6) A&s LE network, as exemplied by its perormance in Houston, was markedly better than

the Verizon Wireless network. HOWEVER, much o this advantage can be readily explained

and it will not as evident in at least some other markets, while in some markets certain KPIs will

avor Verizon Wireless. FURHER, we disagree with the notion that network loading is an issue

at this time on the Verizon Wireless network at least when it came to our test methodology and

results. In act, we argue that the overall perormance on both LE networks could be markedly

better in the uture than it is today, even i LE network trac increases as expected. Much has

to do with the hoped-or maturity o the LE ecosystem and operator initiatives to improve the

perormance o their networks.

e e ece t l et cu

e ey ette i teutue t it i ty,

ee i l et tcicee execte.


5/24


Te ul l report contains graphs and graphica l maps which show network perormance and cel

site density, among other metrics. We have included a couple o gures in this report preview

Without going into too much detai l, Figure 1 helps justi y our conclusion that tremendous

perormance dierences exist between dierent combinations o devices + chipsets, especially

with respect to LE.

Figure 2 on the ollowing page shows the drive route that we used on a single day o testing

in the Bay Area. Although our objective wasnt too map out the network perormance coverage

maps or each operator/technology, we wanted to collect enough data in order to reach denitive

and largely indisputable conclusions on how the various technologies perorm in a commerciallydeployed network.

Source: The Mother of all Network Benchmark Tests Figure 16

Verizon Wireless LTE Device 1 Average APP Layer Throughput = 19.1MbpsVerizon Wireless LTE Device 2 Average APP Layer Throughput = 9.9Mbps

Combined LTE Device 1 and Device 2 Median APP Layer Throughput = 29.0Mbps

Time

Mbps

0

5

10

15

20

25

30

35

40

45

0 50 100 150 200 250

Verizon Wireless Device 2 APP LayerDownlink Throughput (Mbps)

Combined APP Layer Downlink Throughput (Mbps)

Verizon WirelessDevice 1 APP Layer

Downlink Throughput (Mbps)

fiue 1. l l app lye di uut eu ie (et scei 161)


6/24


Source: The Mother of all Network Benchmark Tests Figure 22

fiue 2. die rute by ae (o, s Je, s fcic, it i etee)

50


7/24


lIns oss(pre-publishing and post-publishing prices)Volume 1 AVAILABLE NOW!Network and Technology Performance ($1,995)Volume 2 Quantifying the User Experience ($1,295, $1,495)Volume 3 Detailed Performance Analysis ($1,295, $1,495)fu ret 3 ue ($3,300, $3,995)

ona InformaIonYou may call us at +1 (510) 273-2439 or email us at [email protected] and we will contact you for your billing informationor respond to any further inquiries that you may have. Subscriptioninformation for our Signals Ahead research product, which includesthese reports, can be found on the last page of this report. You canalso visit our website at www.signalsresearch.com or write us at

Signals Research Group, LLC10 Ormindale Court

Oakland, CA 94611

pr-ordr or rpor lIns now (icue t Signals Aheaduciti)

Coming Soon!


8/24


Our three-part series o reports is included with a subscription to Signals Ahead or it can be

purchased on an individual basis the ormer option is ar more economical since it includes at least

14 additional Signals Ahead reports. A summary o each report, including the report associated with

this report preview, ollows in the subsequent paragraphs.

vue 1 (net ecy pece)Next-generation network technologies are not created equal. On paper, there exist meaningul

perormance dierences, some o which are due to channel bandwidth considerations, but also to

the underlying technology itsel. Further, operator deployment philosophies and the maturity o the

solutions can have an over-arching impact on the results.

As operators around the globe struggle to make crucial strategic decisions regarding their network

technology evolution, it is imperative that they ully understand and appreciate the potential o

these technologies as well as their limitations. While this report is intended to address the needs o

operators worldwide by ocusing on the perormance o the technologies, as a secondary eature it

also provides va luable insight into the perormance o each major network deployment in the United

States. Everyone claims that they have the best network, but only one operator can be right.

Specic topics addressed in Volume 1 include, but are not limited to, the ollowing:

Application and/or Physical Layer Troughput

Mean, median and CDF plots

Geo plots o throughput or all test scenarios using Google Earth

echnology comparisons, including

DC-HSDPA versus LE (2x10MHz) with 2x2 MIMO

LE (2x20MHz) versus LE (2x10MHz)

Mobile WiMA X versus HSPA+, LE and DC-HSDPA

DC-HSDPA versus HSPA+

EV-DO versus LE, HSPA+, etc

Single User Spectral Eciency Results

Troughput normalized or channel bandwidth and duplex scheme

Does LE with MIMO really outperorm narrow bandwidth solutions

LE network perormance with multiple devices

DC-HSDPA and HSPA+ devices in the same 10MHz channel allocations

Side-by-Side operator network coverage maps or drive routes used in each market

Downlink throughput Uplink throughput

Network Latency

Variance based on time o day and network loading

LE network deployment philosophies (LE cell site density relative to the legacy network) and

their implications or coverage and capacity

A& HSPA+ versus A& LE

vue 1 i citic et u te

e e cuetyi tteic ecii

ei tei ettecy euti.


9/24


Verizon Wireless EV-DO versus Verizon Wireless LE

A& LE versus Verizon Wireless LE

Mobile WiMAX (2500MHz) versus LE (700MHz)

vue 2 (Qutiyi te e xeiece)Although mobile operators, industry pundits and most well-inormed consumers understand the

notion that a higher megabit-per-second throughput is preerable, the typical consumer is generally

clueless when it comes to understanding what these obscure marketing messages really mean or the

mobile Internet experience. Most operators recognize that they need to move away rom the my

pipe is bigger than your pipe marketing mentality, but it is easier said than done.

Further, it is readily apparent that the capabilities o these next-generation networks requently

exceed the requirements o the application and/or the capabilities o the Internet itsel. Very ew

applications and/or web site servers support high double-digit megabit-per-second throughput

Instead, it may actually be the combination o relatively high throughput and low network latency

oset by transport latency that really denes the user experience. But to what degree do these

relationships provide the most benet to the user?

Mobile video, be it Youube or Netix, is driving mobile data growth and the capabilities o next

generation networks will only serve as an impetus to even higher data usage. Te perceived quality

o the video playback also matters, not only or consumers, but also or mobile operators, content

owners, and video hosting services. As higher resolution video ormats with higher encoding rates

become more mainstream, this issue becomes even more important, especially i the perormance o

next-generation networks ails to keep up with the requirements o the video content.

Tis report is cr itical or operators trying to understand how to market their broadband wireless

service oering as well as how they should prioritize their network optimization activities in order

to achieve the best possible user experience or their subscribers. In addition to mobile operators

this report provides invaluable insight to application developers and content providers who require a

greater appreciation or how network perormance characteristics impact the user experience.

Specic topics addressed in Volume 2 may include, but are not limited to, the ollowing:

Quantiying the user experience based on HP web page download times

Popular websites, including Yahoo, CNN, iunes, Amazon, Youube, etc.

Results down to the millisecond, based on device/chipset signaling messages

Network/technology comparisons

Comparisons based on network loading same location over a 12-15 hour period o time

Determining i perceived dierences in network/technology perormance have more to do with

network loading than the actual capabilities o the network/technology itsel

Determining how the combination o throughput and latency impact the HP web page

download time results

3 axis plot, showing maximum achievable throughput and network latency versus webpage load

time and required throughput

Which matters most latency or throughput

Does DC-HSDPA really oer a quantiable benet over HSPA+

Determining the crossover point when higher throughput become irrelevant

vue 2 i citic et tyi

t uet tet tei iee eice ei

e tey uiitize tei et

tiizti ctiitie.


10/24


Quantiying the user experience based on downloading Google email attachments

Quantiying the user experience based on downloading video and audio content rom iunes

Determining the crossover point when higher throughput become irrelevant

Netix video streaming requirements

Determining the chokepoints in the network (rom end user to the original source o the content)

how they vary as a unction o loading, and their impact on the user experience

vue 3 (detie pece ayi)In our third and nal installment we delve much deeper into the KPIs that we captured with the

Accuver suite o drive test tools. As we have witnessed in the past there are discernible dierences

in how each vendor implements a technology. Frankly, some vendors do a much better job than

their peers.

Given that we collected network perormance data in a number o key markets and that we know

ingly included virtually every single vendor + technology combination that exists in North America

this report provides invaluable competitive intelligence or vendors, subsystem suppliers and mobile

operators.

Further, by peeling back the layers o the proverbial technology onion it becomes possible to gain

a greater appreciation or how each technology delivers its results.

Specic topics addressed in Volume 3 may include, but are not limited to, the ollowing:

Modulation Utilization (QPSK, 16QAM, and 64QAM) by primary and/or secondary carriers

as appropriate

MIMO RI 1 and RI 2 how MIMO perorms at 700MHz

CQI (average and median) by primary and secondary carriers

HS-PDSCH Codes (average, % > 10, distribution) by primary and secondary carriers

HS-SCCH Scheduling Success Rate by primary and secondary carriers

Average PHY Layer Served Rate by primary and secondary carriers

Maximum PHY Layer Scheduled/Served Rate DC-HSDPA only

UL ransmit Power (average and median)

We will also leverage the capabilities o the XCAP-M tool to analyze these KPIs by severa

dierent means, potentially including, but not limited to the ollowing:

MAC-HS Troughput versus RSCP scatter plot

MAC-HS Troughput versus Reported CQI Values scatter plot

Reported CQI Values versus 64QAM Availability scatter plot

MAC-HS Troughput versus Cell ID (real time)

MAC-HS Troughput versus # o Assigned HS-PDSCH Codes (real time)

MAC-HS Troughput (primary, secondary, and combined)

HSPA+ MAC-HS Troughput versus DC-HSDPA MAC-HS Troughput (primary, secondary

and combined)

vue 3 ieiue cetitieiteiece ie

i ee t ti ete eciti

ec tecyeie it eut.


11/24


CINR versus RSSI (scatter plot and real-time)

Troughput versus CINR (scatter plot and real-time)

Troughput versus Cell ID (e.g., handover perormance)

CINR versus RSSI (scatter plot and real-time)

CINR versus Modulation Scheme and/or MCS

UL ransmit Power versus Cell ID

UL Troughput versus ransmit Power

Modulation Scheme (antenna 1 and antenna 2)

pteti tic te ci ye icue:

Te Mother o all Network Drive ests (LE, DC-HSDPA, HSPA+, Rev A and Mobile

WiMAX)

Te challenges o delivering video in a mobile network

How network perormance (throughput and latency) impacts the user experience

Embedded modules/netbooks

D-LE network perormance benchmark results

CoMP and LE Advanced

Going Green nancial implications and challenges

Smartphone signaling implications and LE

LE chipset perormance benchmark test results

Te impact o ype 3i receivers on UE perormance (includes chipset benchmark tests o leadingsolutions)

Whatever happened to IMS?

LE Americas

4G World and GSMA MAC

HSPA+ (MIMO) network perormance benchmark results

Te impact o latency

Public Saety Options with 700MHz

EV-DO Rev B network perormance benchmark results

LE chipset landscape


12/24


7/6/11 Mobile Platforms the center of mobilenetworks In this report we discuss the recent trendsimpacting the various mobile platorms that exist and what hastranspired since our piece rom three years ago on Web 2.0.

We address the state o the mobile platorms that exist, provide

our thoughts on the current and uture prospects and look atthe various trends that are driving the industry.

6/8/2011 United we stand, fragmented we fail Weprovide the key takeaways rom the LE World Summit, heldin Amsterdam. Spectrum ragmentation tops the list o keyLE topics, although a growing ocus on the use o 1800MHzor those operators that have access to it is encouraging.VoLE, or the lack thereo, is still on everyones minds, butin the interim CSFB isnt even working as promised. Finally,there was a lot o talk about Mobile WiMAX, but the emphasisseemed to be on how to best move away rom the technologyand adopt D-LE.

5/16/2011 HetNet: When big cells and small cellscollide In addition to covering the basics o heterogeneousnetworks (HetNet), a key LE-Advanced (R10) eature, wepresent a compelling series o analyt ical studies which demon-strate the need or macro network ofoad, starting as early as2015. We also get into the technical details o how HetNet

works, including discussions on eICIC, ABS and the impor-tance o intererence cancellation in the handset. Finally, welook at what is being done with legacy 3G emtocells to limitintererence-related problems that they introduce, both withthe macro network and between each other.

4/26/2011 Chips and Salsa XIII: Now Seasoned with

Soy Sauce In collaboration with Spirent Communicationswe provide results rom the industrys only independentperormance benchmark tests o HSPA+/HSPA chipsets.In the most recent benchmark study we tested 16 dierentdevice congurations, representing chipsets rom 9 dierentsuppliers, including new entrants, such as Samsung (HSPA+),Intel (HSPA+), Mediaek and HiSilicon. We provide theresults, based on a total o 42 HSPA+ test scenarios and 26HSPA test scenarios.

3/15/2011 Looking beyond HSPA+: keeping up withthe Joneses Based on interviews with 3GPP membercompanies and a thorough review o 3GPP submissions, we

oer an in-depth look at the uture o HSPA+ (Release 11 andbeyond). Ultimately, we conclude that many o the eaturesthat are being incorporated into LE will nd their way intoHSPA+, thus blurring the perormance dierences betweenthe two technologies. Latency and the impact o new eatureson legacy devices are two areas o prime importance whereHSPA+ could ace challenges relative to LE.

1/12/2011 DC-HSDPA: Double the Bandwidth, Doublethe Pleasure, Part II In collaboration with Accuver, whoprovided us with its XCAL-W drive test tool and XCAP-Wpost-processing sotware, we provide results and analysis roman extensive drive test o elstras DC-HSDPA network. We

compare DC-HSDPA with HSPA+ perormance in a numbero side-by-side tests.

1/12/2011 DC-HSDPA: Double the Bandwidth, Doublethe Pleasure, Part I In collaboration with Accuver, whoprovided us with its XCAL-W drive test tool and XCAP-Wpost-processing sotware, we provide results and analysis roman extensive drive test o elstras DC-HSDPA network. Wecompare DC-HSDPA with HSPA+ perormance in a numbero side-by-side tests.

12/10/2010Can you schedule me now? In collabora-tion with Sanjole we examine how some o todays commer-cial LE eNodeBs allocate network resources when servingmultiple devices. We determine that while LE may delivera compelling user experience, it is largely due to an emptynetwork and the large channel bandwidths, and that urtherimprovements are necessary i LE is going to supportmultiple users in an ecient manner.

12/3/2010 A Perspective from LTE Americas and theGSMA Mobile Asia Congress We provide and discuss

various data points which stem rom our part icipation at theLE Americas event in Dallas and the GSMA MAC eventin Hong Kong. We provide an LE market update, including

D-LE, discuss the debate about a smart or dumb pipestrategy, and the impact o smartphones and social networking

services, including the use o cloud computing, intelligentnetworks, network ofoading and data caching.

11/4/2010 A G-Wiz World We provide drive test resultsor eliaSoneras HSPA+ network in Sweden and provide datapoints rom this years 4G World event in Chicago. We alsodiscuss the growing trend o operators who are intelligentlyadding more capacity to their networks through the use ohigher perorming devices/chipsets and upgrades to theirnetwork inrastructure.

10/7/2010 2x20MHz of LTE and the HeisenbergUncertainty Principle We provide an update on

LE network perormance based on extensive testing oeliaSoneras LE networks in Stockholm and Gothenburg,Sweden. Te 60+ page report provides detailed results andanalysis based on more than 600GB o transerred data.

In Case You Missed It


13/24


e tet

Volume 1: Network and Technology Performance 1

1.0 Executive Summary 3

2.0 Introduction 14

3.0 Key Conclusions and Observations 16

4.0 Detailed Test Results and Analysis Downlink Throughput 30

4.1 Clearwire 2x20MHz LTE Phoenix demonstration network 30

4.2 d-hsdpa eu l 32

4.2.1 DC-HSDPA versus LTE (Dallas, June 16, 2011 0650) 32

4.3 d-hsdpa hspa+ reut 37

4.3.1 DC-HSDPA and HSPA+ Results (Houston, September 6, 0638) 37

4.3.2 DC-HSDPA and HSPA+ Results (Oakland, July 8, 0540 39

4.3.3 DC-HSDPA and HSPA+ (San Francisco, June 29, 1245 40

4.4 LTE and LTE Results 42

4.4.1 LTE and LTE Results (Oakland, July 8, 0652) 424.4.2 Verizon Wireless LTE and Verizon Wireless LTE Results(Houston, September 6, 0910) 43

4.4.3 Verizon Wireless LTE and Verizon Wireless LTE Results(Houston, September 6, 0957) 44

4.5 a ecie 45

4.5.1 All Technologies (Dallas, June 14, 1450) 45


4.5.3 All Technologies (San Francisco) 47

4.6 veiz wiee l eu a& l 59

4.6.1 Verizon Wireless LTE versus AT&T LTE (Houston, September 5, 1501) 61

4.6.2 Verizon Wireless LTE versus AT&T LTE (Houston, September 5, 0634) 63

4.7 hspa+ eu mie wimaX (k ity, Jue 17, 0540) 65

4.8 hspa+ eu hspa+ (k ity, Jue 17, 1300) 67

4.9 d-hsdpa eu mie wimaX (hut, setee 6, 0722) 68

4.10 LTE versus HSPA+ (Houston, September 5, 1622) 69

5.0 Detailed Test Results and Analysis Uplink Throughput 72

5.1 eie 2x20mhz l peix etti et 72

5.2 All Technologies 735.2.1 DC-HSDPA versus LTE (Dallas, June 16, 0815) 73

5.2.2 All Technologies (San Jose, July 1, 1425) 75

5.3 l fcu utie eice et ci 77

5.3.1 Verizon Wireless (San Francisco, July 17, 0805) 77

5.3.2 AT&T (Houston, September 7, 1212) 78

5.3.3 Verizon Wireless LTE versus AT&T LTE (Houston, September 5-7) 80


14/24


6.0 Detailed Test Results and Analysis Latency 81

6.1 eie veiz wiee l net (peix, Jue 10) 81

6.2 nueu ecie (d) 83

6.2.1 Numerous Technologies (Dallas, June 14, 0930) 83


6.3 nueu ecie (k ity) 85

6.3.1 Numerous Technologies (Kansas City, June 17, 1340) 85

6.3.2 Numerous Technologies (Kansas City, June 17, 1450) 86

6.3.3 LTE and EV-DO Rev A (Kansas City Airport, June 17, 1815) 87

6.4 a ecie 88

6.4.1 All Technologies (Oakland Hills, June 29, 1700) 88

6.4.2 All Technologies (San Francisco, July 29) 89

6.4.3 All Technologies (Houston) 90

7.0 Network Design Criteria 91

7.1 mutie ecie 927.1.1 Downtown San Francisco all technologies 92

7.1.2 Dallas 96

7.1.3 Peninsula 99

7.2 veiz wiee v-do eu veiz wiee l utie cei 103

7.3 a& hspa+ eu a& l utie cei 105

7.4 a& l eu veiz wiee l utie cei 109

8.0 Test Methodology 114

9.0 Conclusions 117

10.0 Appendix 1 118


15/24


Iex fiuefiue 1. Clearwire 2x20MHz LTE APP Layer Downlink Throughput Results CDF and Pie Chart Distribution 30

fiue 2. Clearwire 2x20MHz Vehicular Mode Geo Plot of Downlink APP Layer Data Rates 31

fiue 3. DC-HSDPA versus LTE APP Layer Downlink Throughput versus Time (Test Scenario 2) 32

fiue 4. DC-HSDPA versus LTE APP Layer Downlink Throughput CDF (Test Scenario 2) 33

fiue 5. DC-HSDPA Performance in Dallas during early Morning Hours Geo Plot of App Layer Throughput 34

fiue 6. DC-HSDPA APP Layer Downlink Throughput Results for all Dallas Test Scenarios - CDF and Pie Chart 35

fiue 7. LTE APP Layer Downlink Throughput Results for all Dallas Test Scenarios - CDF and Pie Chart 36

fiue 8. Drive Route (Test Scenario 157) 37

fiue 9. DC-HSDPA and HSPA+ APP Layer Downlink Throughput Results for all Dallas Test Scenarios CDF (Test Scenario 157) 38

fiue 10. DC-HSDPA and HSPA+ APP Layer Downlink Throughput versus Time (Test Scenario 114) 39

fiue 11. DC-HSDPA and HSPA+ APP Layer Downlink Throughput versus Time (Test Scenario 103) 40

fiue 12. Drive Route (Test Scenarios 102-103) 41

fiue 13. LTE and LTE APP Layer Downlink Throughput versus Time (Test Scenario 115) 42


fiue 15. Drive Route (Test Scenario 152) 43


fiue 17. All Technologies APP Layer Throughput Results CDF (Test Scenario 12) 45

fiue 18. Drive Route Test Scenarios 12-13 46

fiue 19. All Technologies APP Layer Downlink Throughput Results CDF (Test Scenario 13) 47

fiue 20. All Technologies APP Layer Downlink Throughput Results CDF and Pie Chart Distribution (Bay Area, July 1) 48

fiue 21. All Technologies APP Layer Downlink Throughput Results

CDF and Pie Chart Distribution (Oakland, July 8) 49

fiue 22. Drive Route Bay Area (Oakland, San Jose, San Francisco, and all points in between) 50

fiue 23. Drive Route Test Scenario 73 51

fiue 24. All Technologies APP Layer Downlink Throughput versus Time (Test Scenario 73) 52


fiue 26. All Technologies APP Layer Downlink Throughput Results with Bandwidth and MIMO Adjustments CDF (Test Scenario 33) 53

fiue 27. Drive Route Test Scenarios 83 and 88 53


fiue 29. All Technologies APP Layer Downlink Throughput Results CDF (Test Scenario 83) 54fiue 30. All Technologies APP Layer Downlink Throughput Results with Bandwidth and MIMO Adjustments CDF (Test Scenario 83) 55



fiue 33. All Technologies APP Layer Downlink Throughput Results with Bandwidth and MIMO Adjustments CDF (Test Scenario 88) 56


16/24




fiue 36. All Technologies APP Layer Downlink Throughput Results CDF (Test Scenario 104)58

fiue 37. AT&T LTE PHY Layer Downlink Throughput Results geo plot (Houston) 59

fiue 38. Verizon Wireless LTE versus AT&T Wireless LTE PHY Layer Downlink Throughput Results CDF (ALL COMPARATIVE RESULTS) 60

fiue 39. Drive Route Test Scenario 137 61fiue 40. Verizon Wireless LTE versus AT&T Wireless PHY Layer Downlink Throughput versus Time(Test Scenario 137) 62

fiue 41. Verizon Wireless LTE versus AT&T Wireless LTE PHY Layer Downlink Throughput Results CDF (Test Scenario 137) 62


fiue 43. Verizon Wireless LTE versus AT&T Wireless PHY Layer Downlink Throughput versus Time(Test Scenario 124) 64

fiue 44. Verizon Wireless LTE versus AT&T Wireless LTE PHY Layer Downlink Throughput Results CDF (Test Scenario 124) 64

fiue 45. Drive Route Test Scenario 3665fiue 46. HSPA+ versus Mobile WiMAX APP Layer Downlink Results CDF (Test Scenario 36)66

fiue 47. Drive Route Test Scenario 4367

fiue 48. T-Mobile HSPA+ and AT&T HSPA+ APP Layer Downlink Results CDF (Test Scenario 43) 67

fiue 49. T-Mobile DC-HSDPA and Clearwire Mobile WiMAX APP Layer Downlink Results CDF (Test Scenario 158-159) 68

fiue 50. AT&T LTE and AT&T HSPA+ PHY/MAC Layer Downlink Results CDF (Test Scenario 140) 69


fiue 52.Inter-RAT Handover 70

fiue 53. Clearwire 2x20MHz LTE Uplink Throughput Results CDF and Pie Chart Distribution (Test Scenario 57) 72


fiue 55. DC-HSDPA versus LTE APP Layer Uplink Throughput versus Time (Test Scenario 4) 74

fiue 56. DC-HSDPA versus LTE APP Layer Uplink Throughput Results CDF (Test Scenario 4) 74

fiue 57. All Technologies APP Layer Uplink Throughput versus Time (Test Scenario 97) 75

fiue 58. All Technologies APP Layer Uplink Throughput Results CDF (Test Scenario 97) 75

fiue 59. All Technologies APP Layer Uplink Throughput Results with Bandwidth Adjustments CDF (Test Scenario 97) 76

fiue 60. LTE APP Layer Uplink Throughput versus Time (Test Scenario 119) 77

fiue 61. LTE PHY Layer Uplink Throughput versus Time (Test Scenario 150)78

fiue 62. AT&T LTE PHY Layer Uplink Throughput Results geo plot (Houston)79

fiue 63. Verizon Wireless LTE and AT&T LTE APP Layer Uplink Throughput Results CDF (ALL RESULTS) 80

fiue 64. Clearwire Demo LTE and Verizon Wireless LTE Latency Results (Test Scenario 2, 4) 81

fiue 65. LTE, DC-HSDPA and HSPA+ Latency Results (Test Scenario 5) 83

fiue 66. All Technologies Latency Results (Test Scenario 13) 84

fiue 67. AT&T HSPA+, Mobile WiMAX and Verizon Wireless EV-DO Rev A Latency Results (Test Scenario 17) 85


17/24


fiue 68. AT&T HSPA+, T-Mobile HSPA+ and Verizon Wireless EV-DO Rev A Latency Results (Test Scenario 18) 86

fiue 69. Verizon Wireless LTE and EV-DO Rev A Latency Results (Test Scenario 20) 87

fiue 70. All Technologies Latency Results (Test Scenario 21) 88

fiue 71. All Technologies Latency Results 89

fiue 72. All Technologies Latency Results 90

fiue 73. Drive Route with Serving Cell ID Sector Information San Francisco (by network)Verizon Wireless EV-DO Rev A 850/1900MHz 92

fiue 73a. Drive Route with Serving Cell ID Sector Information San Francisco (by network)Verizon Wireless LTE 700MHz 93

fiue 73b. Drive Route with Serving Cell ID Sector Information San Francisco (by network)AT&T HSPA+ 1900MHz 93

fiue 73. Drive Route with Serving Cell ID Sector Information San Francisco (by network)T-Mobile DC-HSDPA 1700MHz94

fiue 74. Cell Sector Handoffs versus Time San Francisco (by network) 95

fiue 75. Drive Route with Serving Cell ID Sector Information Dallas (by network)Verizon Wireless LTE 700MHz 96

fiue 75a. Drive Route with Serving Cell ID Sector Information Dallas (by network)Clearwire Mobile WiMAX 2500MHz 96

fiue 75b. Drive Route with Serving Cell ID Sector Information Dallas (by network)T-Mobile DC-HSDPA 1700MHz97

fiue 75. Drive Route with Serving Cell ID Sector Information Dallas (by network)AT&T HSPA+ Coverage 850MHz 97

fiue 76. Cell Sector Handoffs versus Time Dallas (by network) 98

fiue 77. Drive Route with Serving Cell ID Sector Information Peninsula (by network)AT&T HSPA+ 850/1900MHz 99

fiue 77. Drive Route with Serving Cell ID Sector Information Peninsula (by network)Clearwire Mobile WiMAX 2500MHz 100

fiue 77a. Drive Route with Serving Cell ID Sector Information Peninsula (by network)T-Mobile DC-HSDPA 1700MHz 100

fiue 77b. Drive Route with Serving Cell ID Sector Information Peninsula (by network)Verizon Wireless EV-DO Rev A 850MHz 101

fiue 77. Drive Route with Serving Cell ID Sector Information Peninsula (by network)Verizon Wireless LTE 700MHz 101

fiue 78. Cell Sector Handoffs versus Time Peninsula (HSPA+ and Mobile WiMAX) 102

fiue 79. Drive Route with Serving Cell ID Sector Information San Francisco and Oakland (EV-DO Rev A and LTE) Verizon Wireless EV-DO Rev A 850/1900MHz 103

fiue 79a. Drive Route with Serving Cell ID Sector Information San Francisco and Oakland (EV-DO Rev A and LTE) Verizon Wireless LTE 700MHz 104

fiue 80. Cell Sector Handoffs versus Time Peninsula (EV-DO Rev A and LTE) 104

fiue 81. Drive Route with Serving Cell ID Sector Information Houston (HSPA+ and LTE)AT&T HSPA+ 1900MHz 105

fiue 81a. Drive Route with Serving Cell ID Sector Information Houston (HSPA+ and LTE)AT&T LTE 700MHz 106

fiue 82. Cell Sector Handoffs versus Time Houston (HSPA+ 1900MHz and LTE) 106


18/24


fiue 83. Drive Route with Serving Cell ID Sector Information Houston (HSPA+ and LTE)AT&T HSPA+ 850MHz/1900MHz 107

fiue 83a. Drive Route with Serving Cell ID Sector Information Houston (HSPA+ and LTE)AT&T LTE 700MHz 108

fiue 84. Drive Route with Serving Cell ID Sector Information Houston (LTE and LTE)AT&T LTE 700MHz 109

fiue 84a. Drive Route with Serving Cell ID Sector Information Houston (LTE and LTE)

Verizon Wireless LTE 700MHz 110fiue 85. Cell Sector Handoffs versus Time Houston (LTE and LTE) 110

fiue 86. Drive Route with Serving Cell ID Sector Information Houston (LTE and LTE)AT&T LTE 700MHz 111

fiue 86a. Drive Route with Serving Cell ID Sector Information Houston (LTE and LTE)Verizon Wireless LTE 700MHz 112

fiue 87. Cell Sector Handoffs versus Time Houston (LTE and LTE) 112

fiue 88. XCAL-M Drive Test Tool in Action DL performance 114

fiue 89. XCAL-M Drive Test Tool in Action UL performance 115

fiue 90. Verizon Wireless APP Layer Downlink Throughput Results CDF and Pie Chart Distribution (Phoenix) 118

fiue 91. Verizon Wireless APP Layer Uplink Throughput Results CDF and Pie Chart Distribution (Phoenix) 118

fiue 92. Clearwire Mobile WiMAX APP Layer Downlink Throughput Results CDF and Pie Chart Distribution (Dallas) 119

fiue 93. Clearwire Mobile WiMAX APP Layer Downlink Throughput Results geo plot (Dallas) 119

fiue 94. Clearwire Mobile WiMAX APP Layer Uplink Throughput Results CDF and Pie Chart Distribution (Dallas) 120

fiue 95. Verizon Wireless LTE APP Layer Downlink Throughput Results CDF and Pie Chart Distribution (Dallas) 120

fiue 96. Verizon Wireless LTE APP Layer Downlink Throughput Results geo plot (Dallas) 121

fiue 97. Verizon Wireless LTE APP Layer Uplink Throughput Results CDF and Pie Chart Distribution (Dallas) 121

fiue 98. T-Mobile DC-HSDPA APP Layer Downlink Throughput Results CDF and Pie Chart Distribution (Dallas) 122

fiue 99. T-Mobile DC-HSDPA APP Layer Downlink Throughput Results geo plot (Dallas) 122

fiue 100. T-Mobile DC-HSDPA APP Layer Uplink Throughput Results CDF and Pie Chart Distribution (Dallas) 123

fiue 101. AT&T HSPA+ APP Layer Downlink Throughput Results CDF and Pie Chart Distribution (Dallas) 123

fiue 102. AT&T HSPA+ APP Layer Downlink Throughput Results geo plot (Dallas) 124

fiue 103. AT&T HSPA+ APP Layer Uplink Throughput Results CDF and Pie Chart Distribution (Dallas) 124

fiue 104. AT&T HSPA+ APP Layer Downlink Throughput Results - CDF and Pie Chart Distribution (Kansas City) 125

fiue 105. AT&T HSPA+ APP Layer Downlink Throughput Results geo plot (Kansas City) 125

fiue 106. AT&T HSPA+ APP Layer Uplink Throughput Results CDF and Pie Chart Distribution (Kansas City) 126

fiue 107. T-Mobile HSPA+ APP Layer Downlink Throughput Results CDF and Pie Chart Distribution (Kansas City) 126

fiue 108. T-Mobile HSPA+ APP Layer Downlink Throughput Results geo plot (Kansas City) 127

fiue 109. T-Mobile HSPA+ APP Layer Uplink Throughput Results CDF and Pie Chart Distribution (Kansas City) 127

fiue 110. Verizon Wireless EV-DO Rev A APP Layer Downlink Throughput Results CDF and Pie Chart Distribution (Kansas City) 128

fiue 111. Verizon Wireless EV-DO Rev A APP Layer Uplink Throughput Results CDF and Pie Chart Distribution (Kansas City) 128


19/24


fiue 112. Clearwire Mobile WiMAX APP Layer Downlink Throughput Results CDF and Pie Chart Distribution (Kansas City) 129

fiue 113. Clearwire Mobile WiMAX APP Layer Downlink Throughput Results geo plot (Kansas City) 129

fiue 114. Clearwire Mobile WiMAX APP Layer Uplink Throughput Results CDF and Pie Chart Distribution (Kansas City) 130

fiue 115. Verizon Wireless LTE APP Layer Downlink Throughput Results geo plot (Bay Area) 131

fiue 116. Verizon Wireless EV-DO Rev A APP Layer Downlink Throughput Results geo plot (Bay Area) 132

fiue 117. T-Mobile DC-HSDPA APP Layer Downlink Throughput Results geo plot (Bay Area) 133

fiue 118. AT&T HSPA+ APP Layer Downlink Throughput Results geo plot (Bay Area) 134

fiue 119. Clearwire Mobile WiMAX APP Layer Downlink Throughput Results geo plot (Bay Area) 135

fiue 120. Cell Sector Handoffs versus Time Oakland (by network) 135

fiue 121. Oakland Drive Route for Cell Handover Study 136

fiue 122. Verizon Wireless LTE PHY Layer Downlink Throughput Results geo plot (Houston) 136

fiue 123. Verizon Wireless LTE PHY Layer Downlink Throughput Results geo plot (Houston) 137

Iex ee a. Summary of Downlink and Uplink Throughput Test Scenarios 138


20/24


8.0 et metyFor the drive tests that we have been conducting this summer we primarily used the Accuver

XCAL-MO network benchmarking tool along with the Accuver XCAL drive test tool to collect the

underlying perormance indicators and to conduct the user experience tests. For purposes o our tests

we limited the XCAL-MO to only our dongles one dongle or each network/technology that

we wanted to test. In theory we could have installed multiple dongles or each network/technology

We used the Accuver XCAP post-processing tool to analyze the data and to help us create the

gures which appear in this summary report. Tanks to a combination o the powerul tool andcountless hours spent on the road, we are convinced that we have witnessed network perormance

both good and bad that would have otherwise not been observed.

Figure 88 and Figure 89 illustrate a typical user display that we used when collecting the data. We

have included two gures since they also help prove that we observed downlink data rates greater

than 61Mbps (Figure 88) and uplink data rates in excess o 23Mbps (Figure 89).

we ue te accue

Xal-mo Xal-m tt cect te ueyi

ece iict te accue Xap-m

t-cei t t te yi te

t tt e cecte.

fiue 88. Xal-m die et i acti dl ece

Source: Accuver XCAL and SRG


21/24


Each operator provided us with at least two dongles, although in the case o operators, such as

Clearwire, with multiple network/technology deployments (e.g., 2x20MHz LE and 1x10MHz

Mobile WiMAX), we received multiple dongles.

In order to ensure that we ully loaded the air link channel or the networks/technologies that

we were testing, we leveraged multiple high-bandwidth servers, including servers in Phoenix

(>100Mbps), Dallas (300Mbps) and Chicago (150Mbps). Further, we established multiple sessions inorder to oset some o eects o transport latency and the CP ACK window associated with FP.

Although everyone that we spoke to agreed with our approach, it did result in unintentional

consequences. Specically, we observed that the individual FP sessions would periodically

stop sending data with the requency and the length o the stop period increasing during times

when we believed the wireless networks were experiencing higher network loading. Reducing the

number o FP sessions to a single session resolved the issue and we were generally able to use at

least two FP sessions, thus solving the problem that we were originally trying to address. We

cant explain what was causing the perormance issue but it was evident in all networks that we

tested and across all vendors, albeit to varying degrees. Weve ruled out RF-related issues as being

the source o the problem.

For purposes o these tests, we used the Windows 7 operating system, which uses a dynamicwindow allocation methodology to theoretical ly deliver the best possible throughput or the given

combination o latency and channel conditions. As we have documented in the past, we believe

Windows 7 leaves something on the table, meaning that the super-high data rates associated with

LE, in particular with a 20MHz channel, are not always achieved. We believe our approach

is still appropriate given that it is logistically impossible to purchase a notebook computer with

the Windows XP operating system. We also note that the connection manager associated with at

least one operators technology/network does not change the CP window size when it is installed

meaning that the data rates could already be limited, even with the legacy operating system.

we ue te wi 7eti yte, icue yic i

cti ety tteeticy eie te et

ie tuut teie citi tecy

ce citi.

fiue 89. Xal-m die et i acti l ece

Source: Accuver XCAL and SRG


22/24


Network latency tests were done to servers located in the vicinity o the market in which we were

doing the tests. Since we cant rule out the eect o transport latency, readers should ocus their

attention on the relative perormance dierences versus the absolute results.

esting in each market took place rom as early as 4AM local time until the early evening

hours. We also did a lot o user experience testing during the dead o the night when we suspect

the networks were wide open. Since we were using test equipment we had the ability to deter

mine whether or not network loading was impacting the results. Suce it to say that in the early

morning hours network loading was not a concern or any o the networks. Later in the day

network loading impacted the perormance o certain networks/technologies while it was not even

a consideration with other networks/technologies. We take this phenomenon into consideration

when doing our analysis.

A large percentage o our test data was collected rom a moving vehicle. Tis approach ensured that

we achieved statistically meaningul results since as we have demonstrated in past reports, moving

a ew eet or turning 90 degrees can meaningully impact the achievable throughput. Further, we

based our analysis and conclusions on literally hundreds o Gigabytes o transerred data. Tis

approach is markedly dierent rom the more commonly used method which involves using popular

web-based speed testing sites and transerring tens o Megabytes o data. From our perspective

this latter approach achieves anecdotal results which are statistically meaningless and not neces

sarily reective o the overall network perormance. Tis sampling o the network perormance

also provides no insight whatsoever into how/why the throughput was achieved since the KPIs

are limited to throughput and latency, versus KPIs, such as modulation type, number o assigned

resource blocks, MIMO availability, scheduling requency, etc.

One drawback o our approach is that it does tend to understate the perormance o the network

since the ading conditions rom a moving vehicle are more challenging than rom a stationary

position or someone walking down the street. Given the emergence o next-generation smartphones

(LE, HSPA+, Mobile WiMAX, etc.), accessing the broadband wireless network rom a moving

vehicle will be more commonplace now than in the past. Ideally, we would include stationary tests

rom hundreds o locations in a given market and all times o day, but this approach goes well

beyond something that we can reasonably do or these studies.

Although we do not include results rom our user experience tests in this summary report, we usedpopular websites or our HP web page download tests and Google Mail or the email application

We also used popular video content delivery services, such as Netix and Youube, not to mention

Skype Video and iunes where we downloaded numerous large les o video content.

Like all Signals Ahead reports, we received no sponsorship or unding rom the participating

companies, in order to maintain our independence. As such, we oot the bill or all o our travel

expenses not to mention an inordinate amount o time and eort collecting the data and writing

these series o reports.

We also could not have done this report without the support o Accuver who provided us with its

suite o drive test tools and post-processing sotware. SRG takes ull responsibility or the analysis

and conclusions that are documented in this report and in our orthcoming series o reports.

by teti ieice y teiue giyte

t, e eiee ttu ccui e

ttiticy iict,ee i te eut y

ity uette te tueciitie te et.


23/24


mice eeMichael Telander is the CEO and Founder o Signals Research Group. In his current

endeavor he leads a team o industry experts providing technica l and operator economics anal-

ysis or clients on a global basis. Mr. Telander is also responsible or the consultancys Signals

Aheadresearch product, including its widely acclaimed Chips and Salsa series o reports that

ocus on the wireless IC industr y.

Previously, Mr. Telander was an analyst with Deutsche Bank Equity Research. Prior to joiningDeutsche Bank, Mr. Telander was a consultant with KPMG (now known as BearingPoint)

and a communications ocer with the United States Army. Mr. Telander has also published

numerous articles or leading trade publications and engineering journals throughout his career.

He has been an invited speaker at industry conerences around the world and he is requently

quoted by major news sources and industry newsletters, including he Economist, Te Wall Street

Journal, Investors Business Daily, Reuters, Bloomberg News, and Te China Daily. Mr. Telander

earned a Masters o Science in Solid State Physics rom North Carolina State University and a

Masters o Business Administration rom the University o Chicago, Graduate School o Business.

4g w, Chicago, ILOct. 24-27

l aeic, Dallas, TXNov. 8-9Invited speaker

rr wiee o et, Orange County, CANov. 10

Invited Speaker

ue ectic s, Las Vegas, NVJan. 10-13

mie w e, Barcelona, SpainFeb. 20-23


24/24

please note disclaimer:Te view s expre ssed in this newslet ter reect those o Signal s Resea rch Group, LLC and are bas ed on ou r understa nding o past and current e vents shaping the w irelessindustry. Tis report is provided or inormational purposes only and on the condition that it will not orm a basis or any investment decision. Te inormation has been obtained rom sources believedto be reliable, but Signals Research Group, LLC makes no representation as to the accuracy or completeness o such inormation. Opinions, estimates, projections or orecasts in this report constitutethe current judgment o the author(s) as o the date o this report. Signals Research Group, LLC has no obligation to update, modiy or amend this report or to otherwise notiy a reader thereo in theevent that any matter stated herein, or any opinion, projection, orecast or estimate set orth herein, changes or subsequently becomes inaccurate.

I you eel our opinions, analysis or interpretations o events are inaccurate, please ell ree to contact Signals Research Group, LLC. We are always seeking a more accurate understanding o thetopics that inuence the wireless industry. Reerence in the newsletter to a company that is publicly traded is not a recommendation to buy or sell the shares o such company. Signals Research Group,LLC and/or its aliates/investors may hold securities positions in the companies discussed in this report and may requently trade in such positions. Such investment activity may be inconsistent withthe analysis provided in this report. Signals Research Group, LLC seeks to do business and may currently be doing business with companies discussed in this report. Readers should be aware that SignalsResearch Group, LLC might have a conict o interest that could aect the objectivity o this report. Additional inormation and disclosures can be ound at our website at www.signalsresearch.comTis repor t may not be reproduced , copied, d istributed or publishe d without the pr ior written aut horization o Signal s Research Group, LLC (copy right 2011, al l rights rese rved by Signals ResearchGroup, LLC).

Signals Ahead SubscriptionThe Signals Ahead newsletter is available on a subscription basis. We offer four distinct packages that have been tai-lored to address the needs of our corporate users. The Group License includes up to ve users from the same company. TheGlobal License is the most attractive package for companies that have several readers since it is offered to an unlimitednumber of employees from the same organization. Finally, the Platinum package includes the Global License, plus up to vehours of analyst time. Other packages are available.

Corporate Rates (18 issues)Group License ($3,995) Global License ($7,995) Platinum ($9,495)

Payment Terms

American Express Visa MasterCard Credit Card # Exp Date / /

Check Check Number

Purchase Order PO Number

Name: Title:

Afliation: Phone: ( )

Mailing Address:

Mailing AddressSignals Research Group, LLC ATTN: Sales10 Ormindale CourtOakland, CA 94611

Our fax number is (510) 338-1284.

Alternatively, you may contact us at (510) 273-2439or at [email protected] and we will contact you for your bill-ing information. We will not process your payment until after the trial subscription period is completed.

Terms and Conditions: Any copying, redistributing, or republishing of this material, including unauthorizedsharing of user accounts, is strictly prohibited without the written consent of SRG.

SA 092711 Network Benchmark Study - Report Preview

Documents