Achieving 100% mobile test coverage perfecto mobile

Achieving 100% Test Coverage for Your Mobile Footprint – Issues, Options and

Strategies

An InfoStretch and Perfecto Mobile Webinar

November 12th, 2013

Agenda

Panelists Introduction

Mobile Testing Challenges

How Can You Optimize Quality for the Mobile App Lifecycle Planning

Definition

Execution

Automation

How Does Perfecto’s Mobile Cloud Solves these challenges

Q&A

Introduction to Panelists

Manish Mathuria

Founder and CTO at InfoStretch Corporation

[email protected]

Eran Kinsbruner

Director of Product Marketing at Perfecto Mobile

[email protected]

InfoStretch - The Mobile Application Lifecycle Enablers

Mobile

Applications

Requirements

Creative Design

and Development

Optimize Quality Performance and

Scalability

Sustained User

Experience

Matured Technology to Support Lifecycle Needs

MobiXpert Elemento Qmetry

Attesto Synchro

Mobile Services – Custom Services for Custom Needs at Your Fingertips

Mobile Strategy Mobile Development Mobile QA Mobile Sustenance

Alignment of Mobility Goals with

Business Strategy

Experience with a Vast Range

of Operating Systems and Form

Factors

SLA Driven Delivery with 30%

Reduction in Time to Market

Private Cloud with 3000+

Devices

200,000+ Apps Tested

with1500+ Device Test Cases

Readymade Test Packs in

Retail, Banking, Travel,

Hospitality and Entertainment

30 Day SLA for New Device

and OS Upgrades

Real Time App Store Review

Tracking, Remediation and

Verification

Proactive Identification of User

Experience Related Issues with

Intelligent Heuristics

Specialized mobile solutions in

retail, travel, digital

entertainment and hospitality

Reusable tools, frameworks

and technology platform for

easier maintenance

Developed and deployed over

300 apps in the app stores

worldwide

Powered by:

6

About Perfecto Mobile

Mobile Testing is NOT Easy

More Apps

Shorter release Iterations

New devices

Growing Team

Android Fragmentation and Portability

11,868 distinct Android devices, up from 3,997 Android devices last year

Delivering Quality in a Complex World is Hard

9

http://www.youtube.com/watch?v=X0aciNBunnQ

OS

Screen Device

Version Style

Mobile Quality IS a critical Challenge…

Mobile is a Market Platform

Plans to Scale

V 1.0

V 2.0

Success!

Disappointment!

That won’t happen again …

How do we manage this?

Qu

alit

y

Time

Let’s get it out the door …

KISS

Read Only

Forget KISS let’s Scale

Complex Txns

Phases of Mobile Quality

Planning Device Selection - Which OS/Device Combination is Ideal for My App?

Definition

Types of Tests - Which Ones Provide me High Coverage?

Processes - How Do I Manage All the Testing Processes? How Do I Handle Automation?

Execution

Devices - How Do we Acquire Them? And Data Plans?

Remote Access - How Do I Do This Securely and Exclusively

Conditioning - How Do I Expose the Device to Potential Defects?

Automation

Automation – Is there ROI?

Automation – Which Tools

Automation – Which Test Cases?

Release & Sustenance

Post Release Support - How do I Manage the New OS’s and Devices

Identifying Defects - How do I Respond and Identify Defects to Ensure High Customer Satisfaction?

End User Feedback and Usage – How do I Monitor this from App Stores

Planning – Device Selection Criteria

Type of the App

• Games

• Social

• Business

User Personas

• Business

• Teen

• Social

Geography

• Asia Pac

• Americas etc.

App Functions

• Streaming

• Screen Res

• GPS etc.

Device Popularity Form

Factor OS

Output: Device/OS/Test

Matrix

Objective: Maximum Coverage at Minimum Cost (&time)

Definition – With Test Management Tools

In-Depth Test Management Solution

Integrates with complementary ALM tools, Simple and Easy to use

CONTROL HUB

FOR QA

Execution - Mobile Labs

Geographies

• Americas

• Europe

• Asia Pac

Languages

• EFIGS, Arabic, Japanese, Chinese, Indian languages

3000 + Devices

• Data Plans

• Global Networks

Mobile Private Cloud Test Lab

Private Cloud

Client B Private Cloud

Client A Device Docking Module Device Cradle

Remote access for verification

Certification Guidelines

Audit trail for certification

Automated Tests/Tasks

Seamless provision of new devices

Readily available and deployable lab ( Cap-Ex Savings)

Capability to test for global coverage (Reduce Time to market)

Access to and testing on pre-market devices (Enhances competitiveness)

Secured, Dedicated, Automated

Mobile Testing and Automation

Native Platform

Automation

Technologies

Platform Independent

Mobile Automation

Mobile HTML5

Based Automation

Typical Challenges

1

2

3

How to Select Test Cases and Devices for

Mobile Automation?

Scripting Challenges and Dealing with

Fragmentation

Test Execution Challenges

Test Case Selection

• System/Peripheral Interaction

• Interaction between multiple apps/ OS-App /Multi Domain

• Location aware, peripheral, battery life, system alerts etc.

Tool and Resource Capability

• Best Practices for other GUI Automation apply

• UI components that will change less.

• Clear understood business processes

Stability/Change

• Less graphically intensive , more data and business scenarios

• Medium complexity test cases - > High - > Low

• Smoke - > High - > Medium - > Low

Capability/ Complexity / Priority

Scripting Challenges and Fragmentation

Widgets Render Different – Same OS

Inherent Device/Platform Differences

Form Factor Challenges

Droid X2 Droid Charge iPhone 4

Dealing with Fragmentation

Browser

Open

Browser

Go To Sign On

Make

Payment

Confirm

Payment Sign Off Text Verify

Make Payment Test case QTP/Selenium

Address

ATM

Search

Browser

Open

Select

Acct View

Schedule

Payment Sync Text

Quick

Links Nick Name

Browser

Go To Sign Off

ZIP ATM

Search

Make

Payment

Sync

Image

Confirm

Payment

Framework

Sign On

Component Library

iOS Android

HTC Aria

Android

HTC Droid BB Torch LG CU620

LG

VX9000

Device Implementations for Sign On (10+)

BB 8520

Devices

Develop a Strong Test Recovery

• Device characteristics

• Test script errors

• Automation tool instability

• Device instability

• Data instability

Automation Tests will break due to

• Develop agents that can restart test components on devices e.g. iwebdriver

• Enhance test case recovery, capability to restart test case.

Invest in Robust Test Recovery System

• Details trace of the test

• Immediately tells you the failures

Invest in Logging from the test cases

Automation Tool Selection Criteria

Are we using the right

Automation tool?

HW based decision Cross Platform support,

Device variations

Sharing/collaboration based decision

Cloud (Public/Private), HW Dependency (Local)

Technologhy based decision

Integration to QC, Script support, Emulator support,

DOM/OCR Object Recognition (Hybrid)

Type of Application Multi - Native, Hybrid,

Web based

Automation is a key for both Functional and Non Functional

Testing

Performance & End-User Experience – Key for Success

Source: LINK

Android defect database

Apple Bug Reporter

Pre-release testing (iOS7 Beta)

Per Device version

Invest in Continuous Integration

• Automate execution of test cases

• Integrated automated reporting, and notifications

• Prominent tools – Jenkins, hudson

What is CI?

• There is no Automation until its Automated

• Well justified effort, easy ROI, pays off in days

• Automated trending and device comparisons.

Why CI?

• Dedicated automation devices

Invest in a Test Automation lab of devices

Perfecto Mobile Continuous Integration Workflow

Submit Build Test …. In Minutes vs. Hours

Demo