Software_Quality_Days_Ashok_Karania_Mobile_Automation

©2015 InfoStretch Corporation. All rights reserved.  

Dr. Ashok Karania | January 21, 2015

Mobile Automation

Lesson from the Trenches

©2015 InfoStretch Corporation. All rights reserved.  

Agenda

§ Mobile  Landscape  &  Tes1ng  Challenges  

§ Automa1on  Challenges  

§ Lessons  From  The  Trenches  

§ Case  Study  

§ Q  &  A  

©2015 InfoStretch Corporation. All rights reserved.  

§  Apps Tested

§  Employees

100+ §  Apps

developed and deployed

3000+

Global Footprint

A leading provider of next-gen mobile application lifecycle services ranging from design and development to testing and sustenance.

About us

Locations Corporate HQ: Silicon Valley Offices: Conshohocken (PA), Ahmedabad (India), Pune (India), London (UK)

600+

250K

§  Active Mobile Customers Engagements 60+

About Infostretch

§  Real devices in mobile labs

©2015 InfoStretch Corporation. All rights reserved.  

Mobile Landscape & Testing Challenges

©2015 InfoStretch Corporation. All rights reserved.  

25%

1.7B

95% Of adults are online with

3+ devices Forrester 2013

Number of smartphones and tablets shipped worldwide will

more than double by 2017 IDC 2014

Of enterprises allow BYOD in some form

FierceMobileIT2014

31.7% The mobile applications market

is expected to increase 31.7% by 2017

IDC 2014

©2015 InfoStretch Corporation. All rights reserved.  

THE FUTURE IS MOBILE

WEARABLES GLASS FLEXIBLE

SMARTPHONES VIRTUAL

©2015 InfoStretch Corporation. All rights reserved.  

AND BEYOND

INTERNET OF

THINGS

BIONICS AND

ROBOTICS

COGNITIVE COMPUTING

3D PRINTERS TRANSPARENT

LCDs

CONNECTED CARS

©2015 InfoStretch Corporation. All rights reserved.  

YOU NEED A NEW VISION, NOT A NEW VERSION

CONNECTED AGILE COMMUNITY

NEXT GEN ENTERPRISE

©2015 InfoStretch Corporation. All rights reserved.  

Mobile Automation Is Imperative

§ Mobile Testing Is Everything We Do On Web & More & More

§ Enables To Meet Compressed Release and Development Cycles

§ Address Fragmentation of Devices, Platforms and Carriers

§ Generate Real World Testing Conditions

§  Improve Overall Testing Quality

§ Overall Time & Resource Efficiencies

§ Systematic

©2015 InfoStretch Corporation. All rights reserved.  

Automation Challenges from the Trenches

©2015 InfoStretch Corporation. All rights reserved.  

1. Device Fragmentation

~12K Android Devices

©2015 InfoStretch Corporation. All rights reserved.  

2. Why Real Devices Are A Nightmare

Fragile Slow Maintenance Issues Cost   Other App Inter-Operability

!

©2015 InfoStretch Corporation. All rights reserved.  

3. Simulating Real Life Conditions The Real Mobile End User Experience Can Be Different

§  Servers Will Be Loaded §  Devices Will Behave Differently §  Network Will React Differently

Apps Behave Differently In Special Conditions §  CPU Usage §  RAM §  Network Conditions §  Other Apps §  Environment conditions §  Light Conditions

We Need Real Life Testing Conditions!

©2015 InfoStretch Corporation. All rights reserved.  

4. Mobile Device Lab Is A Big Challenge §  Test Lab Creation

§  Test Lab Maintenance §  Team

§  Buy Or Cloud

§  Globally Available

§  Management §  Contracts §  OS Upgrades §  Procurement Policy §  Performance §  Security

©2015 InfoStretch Corporation. All rights reserved.  

Lessons from the Trenches

©2015 InfoStretch Corporation. All rights reserved.  

The  best  way  to  deal  with  mobile  device  

automa4on  challenge  is  to  not  automate    every  thing  on  the  mobile  devices ”

“  

©2015 InfoStretch Corporation. All rights reserved.  

How  To  

Eat  An  Elephant  § Priori1ze  Business  Needs  &  Analyze  Test  Cases  § Design  An  Automa1on  Framework  § Select  Right  Automa1on  Tool  

 License  VS  OS,  Integra1ons,  Exis1ng  Investment,  Mobile  Friendly,  Team  Experience  

§ Automate  All  Func1onal  Test  Cases  On  Simulators  Mobile  Web  Can  Be  Simulated  By  Tes1ng  On  Real  Browsers  Using  User  Agent  Strings  

Na1ve  Can  Be  Tested  By  Using  Simulators  

§ Automate  Device  Centric  &  Cri1cal  Func1onal  Test  Cases  On  Real  Devices  § Iden1fy  &  Integrate  Non-­‐Mobile  Elements  Of  Automa1on  

 Data  Condi1oning,  SeRngs  Pre-­‐Requisites,  Environment  SeRngs  

©2015 InfoStretch Corporation. All rights reserved.  

NATURE OF TEST CASES

Func1onal  

Non-­‐Func1onal  

Security  

Performance  

TYPE OF APP

Business  App  

Social  Networking  

Gaming  App  

U1lity  App  

VERIFICATION CRITERIA

Business  Rules  

Look  &  Feel  

Response  Time  

Content  

TARGET USERS

Enterprises  

Consumer    

Field  Engineers  

Test Case Analysis

1.  

©2015 InfoStretch Corporation. All rights reserved.  

Meeting Device Fragmentation - Device

Type of the App Games Social

Business

User Personas Business

Teen Social

Geography Asia Pac

Americas etc.

App Functions Streaming

Screen Res GPS etc.

Device Popularity Form Factor OS

Output: Device/OS/Test Matrix

©2015 InfoStretch Corporation. All rights reserved.  

iOS Android HTC Aria

Android HTC Droid BB Torch LG CU620 LG

VX9000

Device Implementations for Sign On (10+)

BB 8520

Devices

Browser Open

Browser Go To Sign On Make

Payment Confirm Payment Sign Off Text Verify

Make Payment Test case QTP/Selenium

Address ATM

Search

Browser Open

Select Acct View

Schedule Payment Sync Text Quick

Links Nick Name

Browser Go To Sign Off ZIP ATM

Search Make

Payment Sync

Image Confirm Payment

Framework

Sign On

Component Library

2. Mobile Test Automation Framework

©2015 InfoStretch Corporation. All rights reserved.  

Automation Tools Tool   Supported  

Pla.orms  Basis  of  Automa5on  

Scrip5ng  Language  

Instrumenta5on  Required  

Repor5ng  Format   Cost  

A   Object  Access   All  Major   No   HTML,  Junit,  CSV   Free  

P   Both   Na5ve,  Visual  Basic,  Java  

Either   HTML,  PDF,  Na5ve  

Paid  

K   Object  Access   Objec1ve-­‐C   Yes   XCode   Free  

C   Object  Access   Ruby,  C#,  JVM   Yes   HTML   Free  MR   Object  Access   Python   Yes   Junit   Free  

R   Object  Access   Java   Yes   Junit   Free  

MT   Both   JS,  Na1ve   Yes   HTML/Junit   Paid  

X   Both   C#,  Ruby   No   HTML   Paid  

T   GUI   N.A   Yes   Various   Paid  

E   GUI   Na1ve   No   Na1ve   Paid  

©2015 InfoStretch Corporation. All rights reserved.  

Cloud Options •  Real Devices – Perfecto •  Simulators – Sauce Labs

In-House Mobile Labs

Crowd Sourcing

Meeting The Device Lab Challenge

©2015 InfoStretch Corporation. All rights reserved.  

APPROACH QUALITY COST EFFECTIVENESS SECURITY PRIVACY ACCOUNTABILITY TIME TO

MARKET DEVICE

COVERAGE GLOBAL TESTING

Simulator/Emulator 3 5 3 4 4 4 3 5

Real Mobile Devices 4 2 5 5 5 2 5 4

Remote Mobile Access 4 4 4 4 4 5 4 4

Crowd sourcing 2 3 2 1 1 3 4 4

Managed Testing Services 5 3 5 5 5 4 5 4

Guide: Least Effective 0

Most Effective 5

Comparative Evaluation Of Device Testing Options

©2015 InfoStretch Corporation. All rights reserved.  

Create Real Life Conditions

Manage CPU Usage Manage Memory Manage Network Bandwidth Create & Manage Interruptions

Manage Location Of Device Manage Orientation Turn WIFI On or Off

©2015 InfoStretch Corporation. All rights reserved.  

Summary: How To Eat An Elephant One Bit At A Time

§ Prioritize Business Needs & Analyze Test Cases

§ Design An Automation Framework

§ Select Right Automation Tool

§ Automate All Functional Test Cases On Simulators

§ Automate Device Centric & Critical Functional Test Cases On Real Devices

§  Identify & Integrate Non-Mobile Elements Of Automation

©2015 InfoStretch Corporation. All rights reserved.  

Continue The Dialogue…

Email:  [email protected]  Linkedin:  ashokkarania  Blog:  blog.infostretch.com  Blog:  www.ashokkarania.com  

Visit  ATOS  Booth  Today  &  Tomorrow!    

©2015 InfoStretch Corporation. All rights reserved.  

CONNECT WITH US:

Thank You

Tel: 408.727.1100 Email: [email protected]

URL: www.infostretch.com