Test Coverage Index, Volume 2 - 2015

Mobile Test Coverage Index

THE

DEVICE SELECTOR™

OPTIMIZE

RESET

30%10-16

50%17-25

80%26-32

BU

1. DEVICE COVERAGE

2. CUSTOMER SPECIFIC REQUIREMENTS

3. DEVICE CHARACTERISTICS

4. MIX CATEGORIES

Geography Targets

OS Mix

Reference Devices Market Leaders Legacy Devices New & Emerging

Hardware Mix Form Factors (Screen Size) Launch Date

BU Requirements Tablets/SmartphonesOS Targets Legacy length of OS/DevicesCarriers

Edition 2 August 2015

FEATURING NEW EUROPEAN DATA

UK Germany

2

IntroductionThe latest Mobile Test Coverage Index, released in August 2015, is the second in a series of reports Perfecto Mobile will publish each quarter to help answer the question: On what devices and operating systems should I be testing my apps and websites? As device proliferation continues, mobile app developers, testers and product managers need an effective strategy for ensuring real world performance and consistent quality for any user on any device or platform. To help mitigate risk, Perfecto Mobile has developed the Continuous Quality Coverage Framework™ (CQCF), an answer for generating coverage requirements based on evolving market share data, industry research and Perfecto Mobile’s proprietary MobileCloud™ data. The CQCF test coverage optimizer is designed to convert best-practice methodologies into an indexing algorithm for reaching 30%, 50% or 80% coverage for specific geographic, technical and project requirements. As part of the CQCF, we have created this quarterly Mobile Test Coverage Index to serve as an industry baseline that companies can customize to their own needs. The report aggregates and analyzes a wide range of public, private and proprietary data – 4,000+ device profiles and 360,000+ Perfecto Mobile user profiles – to calculate test coverage recommendations for the U.S. and EU5 markets, which include Spain, France, Italy, Germany and the UK. What’s new in this edition of the index? We highlight device coverage and usage data for Germany and the UK—two European countries where adoption of newer iOS, Android and Windows Phone devices is high. We have also included a calendar of new and upcoming mobile devices and platforms so you can plan your coverage strategy accordingly.

3

What the Index Helps You AchieveWith a rapidly expanding device matrix, achieving full coverage of the mobile device market is neither practical nor necessary. Instead, businesses must choose devices based on priorities, relevance and strategic objectives for a given project. With that in mind, this report offers device selection for three levels of mobile market coverage —30%, 50% and 80%.

Index MethodologyOur device selections were developed based on extensive market analysis across four key sectors:

1. OS coverage: Assures performance across relevant OS versions from Android, iOS, Windows Phone and BlackBerry.

2. Screen sizes and resolutions: Analyzes how manufacturer and device variations can impact app quality.

3. Hardware coverage: Accounts for discrepancies in hardware capabilities such as CPU, memory, system on chip.

4. Device mix: Reduces risk and ensures broad app appeal by covering emerging technology, popular favorites and reference devices.

By analyzing an extensive dataset containing over 360,000 consumer usage profiles across 4,000+ unique device configurations, we identified a recommended list of smartphones and tablets to test and develop against in order to assure sufficient coverage of the U.S. and EU5 markets.

30% of the market can be covered through an indexed list 10-16 devices. In this first level, we have included the most important devices in the U.S. and Europe based on market adoption, market device leaders, reference devices and device characteristics such as screen size, hardware and more. Additionally, we have assigned leading versions of Android, iOS, Windows Phone and Blackberry to these devices.

50% of the market can be covered with 17-25 devices. In addition to those listed in the 30% coverage index, this second level accounts for an increased number of OS versions, carriers and devices from manufacturers such as Sony, LG, Samsung, Apple, HTC and more.

80% market coverage can be achieved with an index of 26-32 devices. At the third level, an organization gains a high level of confidence for coverage in the specific market(s) being targeted. This index helps organizations identify defects in devices heavily used by end-users in both regions.

30% 50% 80%

MT

3

4

Key Takeaways

• Apple continues to dominate the U.S. market, while Android dominates Europe. Apple’s steady success is backed up by industry market share data as well as Perfecto Mobile’s own cloud data that shows a huge preference for iOS testing in the U.S. This success is likely to continue with the releases of iPhone 6S and 6S Plus this fall.

• Both market share data and Perfecto Mobile cloud data reveal that the Windows Phone platform is much more popular in Europe than the U.S., so it should be included in your index if you have a presence in European markets. Windows Phone accounts for 10% market share in the EU5 region, with a breakout of 7% market share in Germany and 13% in the UK. Share in the U.S., however, is less than 3%.

• The leading smartphone screen resolutions across geographies are 1440 x 2560 and 1080 x 1920 and the leading screen sizes are between 5-5.5”, which accounts for almost 80% of the market.

• It is important to include Sony devices in the index when addressing the EU5 region as the Sony brand has 8% market share. In the U.S., however, Sony holds a share of less than 1%.

• While adoption of Google Nexus devices is slow, they are the first devices to run the latest version of Android. From a test standpoint, this early access makes Nexus an essential part of every index. In contrast, it takes up to six to eight months to get access to the latest Android version on non-Nexus devices.

• While iPhone 6 and iPhone 5S are the two most popular devices in the U.S. and the UK, Samsung Galaxy S5 tops the list in Germany.

• Apple and Samsung are neck-and-neck in the UK with similarly strong market share (28% and 30% respectively).

What to Keep an Eye on in Q3

• iOS 9 and iPhone 6S and 6S Plus will be available this fall. The iOS 9 beta is available now for developers and we recommend to start testing on it right away.

• The next major Android release (Android M) is also releasing this fall. The Android M beta is now available, and we recommend testing your mobile apps on it using relevant Nexus devices.

• Chinese device maker Huawei recently released a new device (Huawei Ascend P8) and is growing quickly. We believe Huawei devices will soon be widely adopted and recommend testing on them as soon as possible.

7%13%

3%30%

Windows Phone devices in GermanyWindows Phone devices in UK

Apple & Samsung share the same strong market share in the UK

Chinese manufacturer Huawei is starting to gain market share (3%) in Europe, mainly in Germany

UK Germany

5

30-80% Coverage Device Name Release Date Screen Size Screen Resolution System On Chip

1 iPhone 6 Sep-2014 4.7’’ 750 x 1334 Apple A8

2 iPhone 5S Sep-2013 4.0’’ 640 x 1136 Apple A7

3 Samsung Galaxy S6 Mar-2015 5.1’’ 1440 x 2560 Exynos 7420

4 Samsung Galaxy S5 Feb-2014 5.1’’ 1080 x 1920 Qualcomm Snapdragon 801

5 iPhone 6 Plus Sep-2014 5.5’’ 1080 x 1920 Apple A8

6 Samsung Galaxy S6 Edge Mar-2015 5.1’’ 1440 x 2560 Exynos 7420

7 Samsung Galaxy Note 4 Sep-2014 5.7’’ 1440 x 2560 Qualcomm Snapdragon 805

8 Apple iPad Mini 3 Oct-2014 7.9’’ 1536 x 2048 Apple A7

9 Apple iPad Air 2 Oct-2014 9.7’’ 1536 x 2048 Apple A8X

10 HTC One M8 Mar-2014 5.0’’ 1080 x 1920 Qualcomm Snapdragon 801

11 Motorola Google Nexus 6 Oct-2014 6.0’’ 1440 x 2560 Qualcomm Snapdragon 805


13 LG G4 Apr-2015 5.5’’ 1440 x 2560 Qualcomm Snapdragon 808

14 LG G3 May-2014 5.5’’ 1440 x 2560 Qualcomm Snapdragon 801

15 Samsung Galaxy Tab 4 – 8’’ Apr-2014 8.0’’ 800x 1280 Qualcomm Snapdragon 400

16 LG G Pad10.1 LTE July-2014 10.1’’ 1280 x 800 Qualcomm Snapdragon 400


18 Google (LG) Nexus 5 Oct-2013 5.0’’ 1080 x 1920 Qualcomm Snapdragon 800

19 Motorola Moto G (2nd Gen) Sep-2014 5.0’’ 720 x 1280 Qualcomm Snapdragon 400

20 Samsung Galaxy Note Pro 12.2 Jan-2014 12.2’’ 2560 x 1600 Exynos 5 Octa 5420

21 Samsung Galaxy Tab 4 -10.1 Apr-2014 10.1’’ 1280 x 1800 Qualcomm Snapdragon 400

22 Google (HTC) Nexus 9 Oct-2014 8.9’’ 1536 x 2048 Nvidia Tegra K1

23 Apple iPad Air Oct-2013 9.7’’ 1536 x 2048 Apple A7

24 Sony Xperia Z3v Oct-2014 5.2’’ 1080 x 1920 Qualcomm Snapdragon 801

25 Amazon Kindle Fire HDX 8.9 Sep-2013 8.9’’ 2560 x 1600 Qualcomm Snapdragon 800

26 Samsung Galaxy S4 Mar-2013 5.0’’ 1080x 1920 Exynus 5 Octa 5410

27 Motorola Moto X (2nd Gen) Sep-2014 5.2’’ 1080 x 1920 Qualcomm Snapdragon 801

28 LG G Pad 8’’ LTE Aug-2014 8.0’’ 800 x 1280 Qualcomm Snapdragon 400

29 iPhone 4S Oct-2011 3.5’’ 640 x 960 Apple A5

30 HTC Desire 510 Aug-2014 4.7’’ 480 x 854 Qualcomm Snapdragon 400

31 Motorola Droid MAXX July-2013 5.0’’ 720 x 1280 Qualcomm Snapdragon S4 Pro

32 HTC One X Feb-2012 4.7’’ 720 x 1280 Nvidia Tegra 3

*Rankings based on Perfecto Mobile cloud usage and market data

30%

50%

80%

Test Coverage – U.S.

6

Test Coverage – EU5

















16 Huawei Ascend P7 May-2014 5.0’’ 1080 x 1920 Hisilicon Kirin 910T

17 Huawei Ascend Y300 Mar-2013 4.0’’ 480 x 800 Qualcomm Snapdragon S4 Play



20 Huawei Ascend Mate 2 – 4G Jan-2014 6.1’’ 720 x 1280 Qualcomm Snapdragon 400

21 Sony Xperia Z3 Oct-2014 5.2’’ 1080 x 1920 Qualcomm Snapdragon 801


23 Samsung Galaxy Tab 4 8’’ Apr-2014 8.0’’ 720 x 1280 Qualcomm Snapdragon 400

24 Sony Xperia Z1 Sep-2013 5.0’’ 1080 x 1920 Qualcomm Snapdragon 800

25 Samsung Galaxy S3 May-2012 4.8’’ 720 x 1280 Exynos 4412 Quad



28 Sony Xperia Z3 Compact Sep-2014 4.6’’ 720 x 1280 Qualcomm Snapdragon 801


30 Huawei Ascend P8 Apr-2015 4.6’’ 720 x 1280 HiSilicon Kirin 930/935

31 Samsung Galaxy A7 Feb-2015 5.5’’ 1080 x 1920 Exynos 5 Octa 5430

32 Samsung Galaxy Tab S 10.5’ Jun -2014 10.5’’ 2560 x 1600 Exynos 5 Octa 5420

30%

50%

80%


7

















16 Huawei Ascend P7 May-2014 5.0’’ 1080 x 1920 Hisilicon Kirin 910T




20 Huawei Ascend Mate 2 – 4G Jan-2014 6.1’’ 720 x 1280 Qualcomm Snapdragon 400



23 Samsung Galaxy Tab 4 8’’ Apr-2014 8.0’’ 720 x 1280 Qualcomm Snapdragon 400







30 Huawei Ascend P8 Apr-2015 4.6’’ 720 x 1280 HiSilicon Kirin 930/935

31 Samsung Galaxy A7 Feb-2015 5.5’’ 1080 x 1920 Exynos 5 Octa 5430

32 Samsung Galaxy Tab S 10.5’ Jun -2014 10.5’’ 2560 x 1600 Exynos 5 Octa 5420

Test Coverage – Germany











10 Nokia Lumia 520 Apr-2013 4.0’’ 480 x 800 Qualcomm MSM8227








18 Samsung Galaxy Tab 10.5 S Jun-2014 10.5’’ 2560 x 1600 Exynos 5 Octa 5420



21 Samsung Galaxy Tab 4 -10.1 Apr-2014 10.1’’ 1280 x 1800 Qualcomm Snapdragon 400



24 Sony Xperia Z Jan-2013 5.0’’ 1080 x 1920 Qualcomm Snapdragon S4 Pro




28 HTC One M7 Feb-2013 4.7’’ 1080 x 1920 Qualcomm Snapdragon 600


30 HTC One Mini July-2013 4.3’’ 720 x 1280 Qualcomm Snapdragon 400

31 LG G Pad 8 May-2014 8.0’’ 800 x 1280 Qualcomm Snapdragon 400

32 Sony Xperia Tablet Z2 Feb-2014 10.1’’ 1200 x 1920 Qualcomm Snapdragon 801

30%

50%

80%


8

Test Coverage – UK






















21 Nokia Lumia 635 Jun-2014 4.5’’ 480 x 854 Qualcomm Snapdragon 400





26 Samsung Galaxy Tab 4 – 7 Apr-2014 7.0’’ 800 x 1280 Marvell PXA1088

27 Sony Xperia Z2 Tablet Feb-2014 10.1’’ 1200 x 1920 Qualcomm Snapdragon 801




31 Sony Xperia M2 Feb-2014 4.8’’ 540 x 960 Qualcomm Snapdragon 400

32 LG G2 Aug-2013 5.2’’ 1080 x 1920 Qualcomm Snapdragon 800

30%

50%

80%


9






















21 Nokia Lumia 635 Jun-2014 4.5’’ 480 x 854 Qualcomm Snapdragon 400





26 Samsung Galaxy Tab 4 – 7 Apr-2014 7.0’’ 800 x 1280 Marvell PXA1088

27 Sony Xperia Z2 Tablet Feb-2014 10.1’’ 1200 x 1920 Qualcomm Snapdragon 801




31 Sony Xperia M2 Feb-2014 4.8’’ 540 x 960 Qualcomm Snapdragon 400

32 LG G2 Aug-2013 5.2’’ 1080 x 1920 Qualcomm Snapdragon 800

Android and iOS – What You Need to KnowAndroid: Android versions KitKat and Jelly Bean are the most widely adopted and should be included in Android test plans. Lollipop, the latest Android version, also remains important because it’s running on exclusive Google Nexus devices. And while Gingerbread is the oldest version, it’s worth doing key regression tests on because it still has 10% market share in Europe. Note that Android M (6.0) beta is available for testing now in the Perfecto Mobile CQ Lab. We highly recommend starting preliminary testing on this platform prior to its general release, scheduled for September.

*Add the listed OS versions in the 30% column, 50% column and 80% column together to achieve the full 80% OS coverage spectrum.

iOS:The iOS family accounts for the majority of the OS market. iOS 8 makes up 84% of this family, but there are earlier iOS 8 versions that should be included in test plans for app functionality, regression and performance. The same goes for iOS 7, which still accounts for 14% of total users. For older versions like iOS 6, we recommend testing only the latest 6.1.6 version for basic sanity testing on legacy iPhone devices. Looking forward, iOS 9 beta is now available to developers in the Perfecto Mobile CQ Lab and should be included in test plan activities. It is scheduled for general release in September.

*Add the listed OS versions in the 30% column, 50% column and 80% column together to achieve the full 80% OS coverage spectrum.

Platform Family 30% 50% 80%

Lollipop 5.1 5.0.2 5.0, 5.0.1

KitKat 4.4.4 4.4.3, 4.3 4.4.2

Jelly Bean 4.2.2 4.1.2 4.1.1

Ice Cream Sandwich 4.0.4 4.0.3 NA

Gingerbread 2.3.7 2.3.6 2.3.3, 2.3.5

Platform Family 30% 50% 80%

iOS 8 8.4 8.3 8.2, 8.1.3

iOS 7 7.1.2, 7.1.1 7.0.4 7.0

iOS 6 NA NA 6.1.6

10

What’s New and On the HorizonTo keep developers aligned with changes in usage trends, Perfecto Mobile refreshes the granular data powering the CQCF analytics on a quarterly basis. Here is a preview of new and upcoming releases, which we will factor into the next report.

New & Emerging Devices

New & Emerging Operating Systems

Device Name Release Date Screen Size Screen Resolution System on a Chip

iPhone 6S/6S Plus Fall-2015 4.7”/5.5” 750x1334/1080x1920

Samsung Galaxy Tab A 9.7’’ May-2015 9.7’’ 768 x 1024 Nvidia Tegra 3

Samsung Galaxy Note 5 Aug-2015 6.0’’ 2560 x 1440 Qualcomm Snapdragon 820

Motorola Moto G (3rd Gen) 2015 Q3-2015 5.0’’ 1080 x 1920 Qualcomm Snapdragon 410

LG G Pro 3 Apr-2015 6.0’’ 2560 x 1440 Qualcomm Snapdragon 820

Sony Xperia C4 May-2015 5.0’’ 1080 x 1920 MediaTek MT6752

Sony Xperia M4 Aqua June- 2015 5.0’’ 720 x 1280 Qualcomm Snapdragon 615

Huawei Ascend P8 Apr-2015 5.2’’ 1080 x 1920 HiSilicon Kirin 930/935

OS Name Version Expected Release Date

iOS 9.0 Beta iOS 9 – September

Android 1) 5.2 2) Android M 6.0 Beta Android M – September

Windows 10 Mobile Insider Preview – Build 10166 Windows 10 Mobile – September

Mob

ile M

arke

t Cal

enda

r 201

5Be

sur

e to

pla

n ah

ead

to g

et m

axim

um te

st c

over

age.

DEV

ICES

APR

ILM

AYJU

NE

JULY

AUG

UST

SEPT

EMBE

RO

CTO

BER

PLAT

FOR

M

iOS

9 Be

ta

iOS

9W

indo

ws

10 fo

r Pho

nes

and

Andr

oid

M

You

can

do b

eta

test

ing

now

in

the

Perfe

cto

Mob

ile C

loud

New

iPho

ne 6

/6 P

lus

& iP

ads

Sam

sung

Not

e 5

Mot

orol

a M

oto

G (3

rd)

Lum

ia 9

50Lu

mia

950

XL

Sony

Xpe

ria Z

4 (Z

3+)

Sony

Xpe

ria

M4

Aqua

Sam

sung

Gal

axy

Tab

A 9.

7’’ T

able

tSo

ny X

peria

Z4 T

able

t

App

le W

atch

Rel

ease

Sam

sung

Gal

axy

S6Sa

msu

ng G

alax

y S6

Edg

e

LG G

4N

okia

Lum

ia 6

40 X

L

ZTE

Blad

e S6

Plu

sH

uaw

ei P

8Le

novo

A60

00 P

lus

Firs

t Sam

sung

dev

ices

toru

n An

droi

d Lo

llipop

off

the

shel

f

ww

w.pe

rfect

omob

ile.c

om/te

stco

vera

ge

12

��

��

39% 3% 17%28% 5%8%

37% 4% 4% 19%18% 10% 8%

��

��

39% 3% 17%28% 5%8%

37% 4% 4% 19%18% 10% 8%

OS and Device Breakdown by RegionU.S.

EU5

Device Makers

Device Makers

Other

Other

39+58+3+yScreen SizesOS

4” - 5”

5’’- 5.5’’

3.5”- 4”

5.5” - 6”58+20+13+9+y58%58%

39%

3%

20%

13%

9%

60+18+16+6+y60%18%

16%

6%18+72+10+yScreen SizesOS

4” - 5”

5’’- 5.5’’

3.5”- 4”

5.5” - 6”

72%

18%10%

*Based on acquired market share data

13

60+21+14+5+yUK

Germany

Device Makers

Device Makers

Other

Other

60+21+14+5+y60%

60%

21%

21%

14%

14%

5%

5%


4” - 5”

5’’- 5.5’’

3.5”- 4”

5.5” - 6”

59%

28%13%


4” - 5”

5’’- 5.5’’

3.5”- 4”

5.5” - 6”

76%

17%7%

��

��

30% 6% 16%28% 7%13%

42% 6% 20%17% 7%8%

��

��

30% 6% 16%28% 7%13%

42% 6% 20%17% 7%8%

*Based on acquired market share data

14

Perfecto Mobile uses a proprietary Test Coverage Optimizer to determine the exact number of devices that represent a 30/50/80% coverage rate. First, we aggregate raw data from public and private sources to understand usage profiles at a granular level. However, simply taking the X most popular devices will not ensure adequate coverage in the real world. Hardware/screen size/OS permutations are constantly changing and new promotions and device releases can impact turnover rates. The Perfecto Mobile Test Coverage Optimizer factors data from each of the four sectors to create a precise list to test and develop against for each targeted coverage rate. While the 30% coverage rate of 10-16 devices will provide basic device/OS coverage and some of the mix considerations, we recommend looking at the higher coverage goals as development progresses.

The number of devices in each test bracket may vary from quarter to quarter to reflect changes in mobile preferences. Additionally, the information in this report is based on generic requirements. Customizing test parameters for specific development projects may also cause the numbers to shift slightly. It is important to note that developers are more successful when the target device list is continuously tailored to specific test requirements. The Perfecto Mobile CQCF enters project specific device parameters for geography, screen size, OS, hardware, BU requirements, device mix and more. The CQCF analyzes current market data to output a customized, highly targeted Mobile Test Coverage Index.

30% 50% 80%

accounts for 10-16 unique device models



How the CQCF Works

Customer Speci�cCon�gurationRequirements

Customer SDLCConsiderations

Market Data & Perfecto Mobile

Cloud Data

Continuous Quality

Coverage Framework

Inputs Recommendations

1

3

2

1

2

3

4

Unique devices and operating systems for U.S. & Europe

Quality lab con�guration tuned for speci�c SDLC Practices

Total device quantity for lab sizing based on team size, locations and environment

speci�c requirements

TEST COVERAGEOPTIMIZER

*Items 2 & 3 are fully supported as part of an on-demand workshop by the Perfecto Mobile professional services team.

perfectomobile.com

© 2015 Perfecto Mobile Ltd.

About Perfecto MobilePerfecto Mobile, the world’s leader in mobile app quality, provides a hybrid cloud-based Continuous Quality Lab that enables mobile app development and testing teams to deliver better apps faster. The Continuous Quality Lab supports testing processes earlier and more often in the development cycle, giving way to faster feedback and improved time to market. Users can access an exhaustive selection of real mobile devices connected to live networks around the world and leverage them for testing and monitoring throughout the mobile application development lifecycle – from development, functional and performance testing to monitoring and support.

www.perfectomobile.com/testcoverageor call 781-205-4111

781-205-4111

http://www.perfectomobile.com/testcoverage

Test Coverage Index, Volume 2 - 2015

Documents