Mobile Test Automation: Lessons from the Trenches

(408) 727-1100 | in fo@infost retch.com | 2880 Lakes ide Dr ive Sui te 200, Santa Clara, CA

www. infostretch.com

Mobile Test Automation Lessons from the Trenches

Manish Mathuria

CTO

InfoStretch Corporation

Mohammad Sha

Director of QA

Sabre Airline Solutions



Manish MathuriaCTO, InfoStretch Corporation


LINKEDIN: manishmathuria

EMAIL: [email protected]

BLOG: blog.infostretch.com

mailto:[email protected]



AGENDA

1Mobile Test Automation

Tool Categories

Mobile Automation

– Best Practices

Case Study – Sabre Airline

Solutions

QUESTIONS

2 3 54

Mobile Landscape and Testing Challenges



MOBILE LANDSCAPEMOBILE TESTING CHALLENGES



Do you know how many babies are born each day? • 371,000 DAILY

Impressive, until you realize the number iPhones sold • 377,900 DAILY ( Luke Wroblewski (@lukew) January 24, 2012 ) • 515,000 DAILY ( Q2 2012 - BI)

Or the number of Android phones activated. • 1.3M DAILY (Eric Schmidt – 9/5/12)

And letʼs not forget the growth of tablets. • 277,800 DAILY (just iPad – 189k daily in Q2 – WSJ Oct 1/2012)

MOBILE ON THE FAST TRACK



ENTERPRISE MOBILITY EXPLODING



ENTERPRISE MOBILITY EXPLODING



RiskGap from requirements

Traditional SW/PC

Mobile Dev.

‘Traditional’

TimeSW Cycles

Mobile

Short Cycle for Mobile

Short Cycle Mobile dev.

MOBILE - SHORTER LIFE CYCLE



Product Version Product Release

timeline

Browsers

2005

2006

2007

2008

2011

Market

6 years

6 years2005-2011

MOBILE ALM PROCESS – CONTINUOUS QA



Product Version Product Release

timeline

Market

Android V2.3 V3V4

V2.3.3 V2.3.4 V2.3.5 V2.3.6V2.3.7V3.1 V3.2

Jan Mar

Jun Sep

Dec

1 year - 2011

1 year2011


www. infostretch.comWHAT DOES ALL THIS MEAN?

No Matter What Industry – There is an app coming near you!

It will be fast, high profile, high visibility (C level)

It will test your Dev and QA Teams

It will be here to stay

Think process, devices, Automation, Agile

Have Fun!


www. infostretch.com17



MOBILE AUTOMATIONMOBILE AUTOMATION TOOL CATEGORIES

Native Platform Automation Technologies

Platform Independent Mobile Automation

Mobile HTML5 Based Automation





TOOL CATEGORIESMOBILE AUTOMATION TOOL CATEGORIES

Mobile HTML5 Based Automation

Native Platform Automation Technologies

Platform Independent Mobile Automation – Type A

Platform Independent Mobile Automation – Type B

What

• Drives HTML and JS constructs

• Recognizes Web Controls

• Drives GUI widgets and Controls on the Native UI Layer

• Leverages screen coordinates, images and OCR to drive automation

• Drives GUI widgets and Controls on the Native UI Layer

Pros

• Cross device platforms

• Cross browsers• Object aware

automation• Non Intrusive to App

• Ultimate control as good object level UI layer access

• Cross platform scripting, one script many devices.

• Has access to the whole device

• Good control over GUI and executing app

• Object aware automation

Cons

• Limited to Web and Cross Platform HTML5 Apps

• Intrusive, often requires code access

• Written specifically for a device platform

• Limited to the context of the application

• Limited object awareness, reliance on image and OCR

• Less effective for complex automation

• Intrusive will likely take a special build.

• May or may not require jail breaking

• Limited to the context of the application



MOBILE AUTOMATIONLESSONS FROM THE TRENCHES



Dealing with special device conditions

Scripting Challenges and Dealing with Fragmentation

Test Execution Challenges

How to Select Test Cases and Devices for Mobile Automation?

LESSONS LEARNT

1

2

3

4


www. infostretch.com 21

(408) 727-1100 | market ing@infost retch.com | 2880 Lakes ide Dr ive Sui te 200, Santa Clara, CA

DEVICES SELECTIONHow do you identify the Devices for Mobile Automation?

Objective: Maximum Coverage at Minimum Cost (&time)

Type of the App• Games• Social• Business

User Personas• Business• Teen• Social

Geography• Asia Pac• Americas etc.

App Functions• Streaming• Screen Res• GPS etc.

Device Popularity Form Factor

OS

Output: Device/OS/Test

Matrix


www. infostretch.com 22


TEST CASE SELECTIONHow do you identify the test cases for Mobile Automation?

• System/Peripheral Interaction• Interaction between multiple apps/ OS-App /Multi Domain• Location aware, peripheral, battery life, system alerts etc.

Tool and Resource Capability

• Best Practices for other GUI Automation apply• UI components that will change less.• Clear understood business processes

Stability/Change

• Less graphically intensive , more data and business scenarios• Medium complexity test cases - > High - > Low• Smoke - > High - > Medium - > Low

Capability/ Complexity / Priority



Scripting Challenges / Fragmentation



Android 2.3Left: SamsungRight: Sony Ericson

WIDGETS WILL RENDER DIFFERENT EVEN ON SAME OS



Inherent Device/Platform Differences



Droid X2 Droid Charge iPhone 4

FORM FACTOR CHALLENGES







Desktop Browser Droid Charge

ACTUAL TARGET DEVICESTEST ON ACTUAL TARGET DEVICES





Dealing with Fragmentation

Design your test cases for exceptions!

ComponentQTP TestcasesDevice Implementations

Browser Open

Browser Go To Sign On Make

PaymentConfirm Payment Sign OffText Verify

Make Payment Test caseQTP/Selenium

Address ATM Search

Browser Open

Select Acct View

Schedule Payment Sync Text Quick Links Nick Name

Component Library

Browser Go To Sign On Sign Off ZIP ATM

SearchMake

Payment Sync Image Confirm Payment

Framework

iOS Android HTC Aria

Android HTC Droid BB Torch LG CU620 LG VX9000

Device Implementations for Sign On (10+)

BB 8520

Devices



Dealing with Special Device Conditions



• CPU Usage, RAM, Network Conditions, Other Apps, Environment conditions, Light Conditions

App Misbehave in Special Conditions…

• Android – system panel app / Task Manager• iPhone – System and Monitoring Tools

Handy Tools

• Remotely sets device in desired and specific conditions – CPU, RAM, Network, Light etc.

• Accessible from Automated scripts.

InfoStretch Device Stressor

What Special Conditions?



Execution Best Practices





• Device characteristics• Test script errors• Automation tool instability• Device instability• Data instability

Automation Tests will break due to

• Develop agents that can restart test components on devices e.g. if webdriver • Enhance test case recovery, capability to restart test case.

Invest in Robust Test Recovery System

• Details trace of the test• Immediately tells you the failures

Invest in Logging from the test cases

DEVELOP STRONG TEST RECOVERY





33

CI INTEGRATION - REFERENCE



AUTOMATING THE AUTOMATION

• Automate execution of test cases• Integrated automated reporting, and notifications• Prominent tools – Jenkins, hudson

What is CI?

• There is no Automation until its Automated • Well justified effort, easy ROI, pays off in Days• Automated trending and device comparisons.

Why CI?

• Dedicated automation devices

Invest in a Test Automation lab of devices

INVEST IN CONTINUOUS INTEGRATION (CI)





35

OUR CI INTEGRATION



CASE STUDY



Automated a mobile web based application that enables the ability for passengers to book and manage travel on their iOS and Android mobile devices through a simple and intuitive process flow.

Development and Manual Testers defined 33 user scenarios test cases for automation to detect regressions.

Test cases are categorized by a. Book My Flightb. Manage My Flightc. Check Flight Statusd. Feature Function Test – integrated testing with the

above 3 categories

Automation starts as soon as Manual tests verify each functionality.

Application compatibility testing with system notifications are manually tested.

BUSINESS PROBLEM



Automation Tool Selected: Selenium Remote Android and IOS web driver. Automated tests limited to the Sabre web application only. Selenium based solution does not support automated application compatibility i.e. system alerts – phone calls while application is running and Blackberry devices - Perfecto Mobile test automation can be used in this case.

Selenium Test Suites

Webdriver installed on each mobile devices

Jenkins CI


Test Code Repository

Monitor for regression

Sabre

Test on Firefox first – as baseline, then move to mobile devices


TEST ENVIRONMENT

http://images.search.yahoo.com/images/view;_ylt=A2KJkIaxRChQZSAA0jqJzbkF;_ylu=X3oDMTBlMTQ4cGxyBHNlYwNzcgRzbGsDaW1n?back=http://images.search.yahoo.com/search/images?p=Android+phones&n=30&ei=utf-8&y=Search&fr=yfp-t-701&tab=organic&ri=34&w=450&h=450&imgurl=www.vaxuo.com/wp-content/uploads/2011/03/HTC-Inspire-4G-Android-Phone.jpg&rurl=http://www.vaxuo.com/tablets/htc-inspire-4g-android-phone.html&size=72.3+KB&name=HTC+Inspire+4G+Android+Phone&p=Android+phones&oid=590356791832e685249cb0ae88735300&fr2=&fr=yfp-t-701&tt=HTC+Inspire+4G+Android+Phone&b=31&ni=77&no=34&ts=&tab=organic&sigr=11umtao3v&sigb=13johemjs&sigi=129vhq8a6&.crumb=mgrZsN41Xll

https://wiki.jenkins-ci.org/display/JENKINS/Meet+Jenkins

http://images.search.yahoo.com/images/view;_ylt=A2KJkK0xTShQAFAA7vmJzbkF;_ylu=X3oDMTBlMTQ4cGxyBHNlYwNzcgRzbGsDaW1n?back=http://images.search.yahoo.com/search/images?p=FIREFOX+BROWSER&fr=yfp-t-701&fr2=piv-web&tab=organic&ri=46&w=1233&h=1176&imgurl=nuduenka.files.wordpress.com/2010/03/firefox.png&rurl=http://nuduenka.wordpress.com/2010/03/08/browser-%25E0%25B8%2584%25E0%25B8%25B7%25E0%25B8%25AD%25E0%25B8%25AD%25E0%25B8%25B0%25E0%25B9%2584%25E0%25B8%25A3/&size=361.1+KB&name=FireFox+3&p=FIREFOX+BROWSER&oid=ff88d19dddb3cc0add37160fa9f30ea7&fr2=piv-web&fr=yfp-t-701&tt=FireFox+3&b=31&ni=56&no=46&ts=&tab=organic&sigr=13hhra4nc&sigb=139cqrlll&sigi=11gi2moe1&.crumb=mgrZsN41Xll



• Element ID Discovery – – Test code dependent on locator id changes and UI flow changes especially in agile scrum development

environment. Colocation is critical to promote communication. – Xpath not supported on all mobile devices.

• Selenium Web driver – connection reset when executing whole test suites on iOS and Android devices on a single run.– Web driver not responsive at times causing timeouts. – Still maturing - monthly releases

• On Android– Android Web driver – delay in populating in list box values. – Problems with launching web applications on Android emulator. – Compatibility across various devices – i.e. On Android 2.3 OS - Samsung Galaxy and Sony Ericson have

different list-box implementations (see next slide)

• On IOS– iOS Web driver does not always reset after test suite run and need device restart.– Installing Web driver on iOS devices requires “jail broken” device and valid provision profile on

device.

SOME TECHNICAL CHALLENGES



• Automation team to be co-located with development team especially with agile scrum teams.

• Establish policy to ensure locator ids backward compatibility between sprints.

• Due to the variability of devices and web driver behaviors, architect your automation with this theme “when things change, change it in ONE place”.

• Automation scripts should coordinate the highest level abstraction, (libraries) whenever possible to maintain agility and robustness across devices.

• Agreement on list of devices / vendors to be supported. Not all devices will be have the same way.

• Selenium iOS and Android web drivers are still evolving - expect to do debugging

to workaround web driver issues.

LESSONS LEARNT



QUESTIONS



NEXT STEPS

http://tinyurl.com/automationpresentationwww.infostretch.com/starwest-session-mobile-automation.pdf

email: info@infostretch call: (408) 727-1100

Mobile Test Automation: Lessons from the Trenches

Technology