77523429 Mobile Testing

Mobile Handset and Mobile App Testing

Session for Test Management CoPSession for Test Management CoP

Date – 22 Sep 2011

The contents of this deck and the session conducted for TMCoP leverages

• iGATE Patni’s past experiences in this space

• Research references from Industry Research firms

• Individual opinion and research conducted on this subject over a period of time

This is not a snap-shot of iGATE Patni’s capability in Mobile Testing

For any comments and queries regarding this presentation please contact

Disclaimer

Proprietary and ConfidentialSeptember 27, 2011 - 2 -

For any comments and queries regarding this presentation please contact

[email protected]

Mobile Telephony Trends� More than 60 % of world’s population today has access to a Mobile Phone!

� Ratio of Feature phone to SmartPhone users was 80:20 ( Q4 2009 )

� Projected to become 70:30 by Q4 2011 world-wide- The Nielsen Company


Nokia, 453

Samsung, 280.2

LG, 116.7

RIM, 48.8

Apple, 47.5

Others, 413.8

Worldwide 2011 Q2 Smartphone sales by Operating System - Gartner

Market Share Handset Makers ( mn units 2010) : Strategy Analytics 2011

Sybase –Afaria/

Unwired

IBM DB2

EveryplaceRhomobile

Broadbeam/

MobileawareMotorola

SFA/FFA Distribution/RetailManagement

Dashboard

Patient

Monitoring

LegacyEnterprise

Systems

Applications

Middleware/

Platforms

LAN, PSTN, DSL, GSM, GPRS, CDMA, CDPD, UMTS, WLL, WiFi, Satellite

Blackberry

Enterprise Server

HIS Supply Chain Financial

Sierra/

Jasper/

Telenor

Credit Card/Claims

Mobility Landscape


PDA Smart-

phonesMobile

phones

Rugged

Terminal

POS/

Tablets

Laptop

RFID

readerBarcode

Scanner Camera

GPS

ReceiverPrinter Sensors

RS232, USB , Bluetooth, Zigbee, UWB, HomeRF, IrDA

OS/ Platforms

Mobile Devices

Peripherals

LAN, PSTN, DSL, GSM, GPRS, CDMA, CDPD, UMTS, WLL, WiFi, Satellite

Reference – iGP Mobility Deck

Mobility Challenges

• Real-time, Off-line, Near-real-time

• Security, Authentication, Authorization

• Technology Convergence

• Provisioning and device management

• Scalability

• Services – Messaging, location

based, notification

Application Challenges

• Commercial grade apps

Complexity of

• Applications

Enterprise

Systems

Applications

Middleware

Mobility Landscape and Challenges


• Commercial grade apps

• Content Aggregation

• Customizations and personalization

• Multi-device support

• Multi-lingual support

• Application categories

Device Challenges

• Device capabilities

• Programming models

• No dominant standard

• Wide range of target

environments

• Range of devices

• Connectivity options

• Technology options

• Vendors and

• Platforms

OS

Mobile Devices

Peripherals

Reference – iGP Mobility Deck

Handset / Mobile Device

Test Handset from OEM

Native Applications

Web Based Applications

Third Party Applications

Mobile Handset Testing /

Device QualificationMobile Handset App / Mobile WAP Testing

Mobile Testing Landscape


Test Handset from OEM

Test Branded Pre-Released Handsets

supplied by Telecom Operators

Examples

New / Upgrade of Handset from OEM

directly

Co-branded Handset with Telecom Operator

Test Apps that reside on the device and may

or may not require network connectivity

Examples

Messaging,

Multimedia, Games,

PIM etc

Test Applications that are accessible through

mobile browsers

Examples

Mobile Banking

Payment solutions

Test Third party Apps that are specifically

installed on the mobile handset

Examples

Facebook, Twitter

• User Interface

• Branding

• Mobile Phone Controls

• Mobile Handset Applications

• Partner Applications

• Multimedia and Graphics

• User Interface

• Branding

• Mobile Phone Controls

• Mobile Handset Applications

• Partner Applications


Types of Testing

• Functional• Regression • Integration • Automation

Mobile Handset Testing



• Integration with other systems


• Integration with other systems

Focus Areas

• Automation •Multi-lingual • Usability• Field / Coverage• Quality of Service• Performance Testing• Security Testing

The customer in this case could be a Telecom Operator or an OEM for e.g. Nokia

Please note – Types of testing highlighted in Italics are not offered as a service in this space by iGATEPatni today

1. User Interface and Branding : Verifying menus, navigations and the soft keys etc. are as per the

specifications given in the UI specification document.

2. Mobile phone controls : Verify that all the controls or buttons on the handset are mapped correctly

for instance pressing a dedicated camera key should launch the camera application only and not

any other application.

3. Mobile Handset application : Verifying the functional correctness of all the applications on the

handset for e.g. verifying that the alarm on the handset can be set and that it goes off at the given

Mobile Handset Testing – Focus Areas


handset for e.g. verifying that the alarm on the handset can be set and that it goes off at the given

time.

4. Partner applications : This test is to verify whether all the 3rd party applications present on the

handset work properly an example of this could be verifying that the 3rd party application like MSN

messenger is launching properly and is working as needed.

5. Multimedia and graphics : This test is to verify whether the graphics on the handset i.e. wallpapers

, screen savers, themes (both preloaded and downloaded) appear properly.

6. Integration with other systems : if required

Network and Non-Network Testing

Mobile Handset Testing

Requires Network

Doesn’t Require Network

DATA

• WAP | SMS/MMS | DOWNLOAD CONTENTS

• DOWNLOAD GAMES

• CONTACTS

• TOOLS (calculator,voice memo,stopwatch,etc.)

• SETTTINGS


• DOWNLOAD GAMES

• WAP deck and 3rd party apps

• INSTANT MESSAGING (like yahoo,msn.google)

• EMAIL(3rd part email apps)

CALLS

• MAKE/RECEIVE CALLS | VOICE MAIL | SPECIAL NUMBERS

• RECENT CALL LISTS | INTER CARRIER SWITCH

• ALARMS AND SCHEDULER APPS.

• RECEIVING VARIOUS ALERTS (low balance, promo alerts etc.)

• SETTTINGS

• CAMERA

• PRE LOADED CONTENTS

• GAMES

• HELP & INFORMATION CONTENT

• MEMORY

• BLUETOOTH

• MEDIA PLAYER (audio and video)

• LANGUAGE UI (in cases of another language being

present like Spanish)

Sample Handset Specification


Reference – Collateral from past engagement

Types of Testing

• Functional

• Compatibility

• Automation

Mobile App Testing


Mobile

App Testing

• Automation

• Regression

• Performance

• Security

• Usability

The customer in this case could be an Enterprise OR a content Provider or an App/WAP provider

Please note – Types of testing highlighted in Italics are not offered as a service in this space by iGATEPatni today

Popular Testing Tools used in this space

Commercial

• Device Anywhere

• Perfecto Mobile

• Test Quest

Open Source

• Robotium (Android Apps)

• FoneMonkey


• Test Quest

• Squish

• Jamo Solutions

• Eggplant

• FoneMonkey (iphone Apps)

• TestMaker

• T-Plan Robot

Please note – This is only a sample list of tools. This by no means is a list of most recommended tools by iGATEPatni in this space

Parameters Device Anywhere TestQuest Perfecto Mobile Eggplant Robotium

Device and Platform

Support

All Leading Handset OEM s

Android, Blackberry,

Windows, Symbian, iOS,

Win 7, WebOS

Android,

Blackberry, BREW,

UTS, Windows

Android, Blackberry,

Nokia, iOS,

Android, iphone,

Symbian, Windows

Blackberry

Android

Scripting Drag drop functionality

Ability to script in Java

No Scripting Yes Yes Yes

Functional Testing Yes Yes Yes Yes Yes

Automation Testing Yes Yes Yes Yes Yes

Feature Set comparison of Popular Tools


Automation Testing Yes Yes Yes Yes Yes

Performance Testing No Yes No No No

Test Management Built-in Test Case Manager Built-in Test Mgr No No No

Remote Connectivity Yes Yes Yes Yes No

Integration with

industry std. tools

HP QTP and QC

IBM Rational Quality

Not available HP QTP and QC No No

Licensing Model Subscription based SaaS*

[ 12 mth commitment ]

Cost Subscription based

SaaS*

Cost Opensource

* Each package is a collection of devices with some common characteristics such as network (e.g. Verizon Package), manufacturer (Samsung Package), operating system (Android package) etc.

Reference – Collateral from Individual Websites of respective toolset vendors | These are some of the key features ; however one needs to do a thorough POC to arrive at a recommendation

What is Remote Device Testing?


100 s of “real” devices / phones connected to live

network are “installed” in their Data centre available

for use based on a subscription package (SaaS)

Through the DeviceAnywhere Test Centre a tester can

remotely connect to these devices / phones via an https

web based connection in real time and can remotely

perform any action..

� No need not buy all the devices/phones ( including SIM card )

� Test multiple combinations of devices / operating systems

� Leverage feature of the Testing tool to automate and test manage the whole testing process

� Conduct this testing from a remote location / offshore – thus making it further attractive

Some key benefits of device testing tools

Test Planning Test Design Test Execution Test Closure

Analyze Requirements

Design Test Cases

Setup Test

Functional,

Automated

Non-

Document Closure Report

Approach for Testing


Document Test Strategy

Tool Selection and POC *

Setup Test Environment

Create Test Data

Non-Functional

Defect Management

Present Findings

Go Live!

using automated tools OR Emulators

using actual devices

Please note – this is a technical opinion of the author.

Key considerations for Drafting a Strategy

1. Clearly establish the scope of Testing – Functional, Automation etc

2. Agree the number and types of devices that need to be tested for example Blackberry Bold, Blackberry

Storm, iPhone etc . Do not leave any loose ends – for e.g. “ Test this on Blackberry” – which models??

3. Identify the testing toolset of choice – based on the following key criteria

• Budget for Testing

• Number of devices / operating systems / browsers ( including versions of each ) to be tested

• Duration of the programme

• Results of Technical Proof of Concept

4. If certifying a common functionality / WAP / APP across devices account for Regression testing of defects


4. If certifying a common functionality / WAP / APP across devices account for Regression testing of defects

across the devices/operating systems/browsers , else you will run out of time during re-tests!

5. Testing report should highlight the features that are working / not working across devices

6. Conduct automation post a thorough ROI calculation only

7. Some tools do not integrate with traditional Test Management and OR defect management tools –

therefore account for the overhead ( effort and cost ) of managing test cases / defects in a different toolset

8. Defect Management – Prioritize defects upfront with the Dev/Vendors and Business

9. In third party testing scenario ( OEM ) account for additional rounds of discussions for defect triage

Please note – this is a technical opinion of the author.

Case Studies


Activations

Port In/Out

Help/Self Care

Security

Change MIN/ESN

Top Ups

Register CC

Adhoc Top Up

VMU Web Portal

Internet

Sales

Handset

Plans

Services

Phones

Contact Center/ IVR

“Virgin Xtras”

Ringtones

Wallpapers

Handset

Virgin Mobile USA Customers

Virgin Mobile USA – Handset Testing

• First MVNO in US and UK

• Customer base ~ 5 M

• “Pay as you go” mobile

services and handsets

• Uses Sprint’s network

• Target segment 15-30 yr old


CRM Billing CDR

Distribution

Center

Credit Card

TransactionsTelcordia

Gateway

PromosPromos Paypal Top Up

Auto Top Up

VirginXtras Mobile Web

Messaging

Games

3rd Party Content

• Target segment 15-30 yr old

• Partnered with iGATEPatni

for QA and UAT of the IT stack

• Handset, IVR, Web-

Portal, CRM, DW and Billing

platforms

• 5 year relationship

• 52 person strong team in

onshore / off

Areas of Testing

Handset Testing

Handset

� Functionality

� User Interface

� Branding

� Mobile Phone controls

� Partner Applications

WAP

� WAP Enabled Application

� Third Party Applications

� Third Party Content

Integration

� Integration with

Mediation Managers

Billing system

Customer Care system


� Partner Applications

� Multimedia and Graphics

Functional Testing | Regression Testing | End to End Testing |

Compatibility Testing

VMU

KYOCERANOKIA UT STARCOMM

HANDSETUNDERSTANDING

DOCSTEST CASE PREP

REQUIREMENTS

UI SPEC &

HANDSET

Testing Process and Flow


END CUTOMER

VMU

FLASHING

TESTINGBUG REPORTING

NEW BUILD

PATNI

UNSTABLE

STABLE

HANDSET

HANDSET,UI SPEC ,HRD

Offshore

HANDSET PORTFOLIO

WDIG

GATEWAY

P100 P205

Handset Testing


Onsite

DATA Platform

Telcordia SCH110LG200 STIC LAB

�Voice

�Contacts

�SMS

�MMS

�Email

�Browsing

�Settings

�Calendar

�Alarm

�Camera functionalities

�Hands free Calling

�Content Download

�Family Monitor

�Family Alerts

�Family Locator

�Photo Upload and

Download


77523429 Mobile Testing

Documents

mobile handset

testing highlighted

regression testing

mobile devices

confidential september 27

technical opinion

hp qtp

reference collateral