Page 1
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
How to Build an Enterprise Grade Mobile Testing Strategy
Eran Kinsbruner Product Marketing @Perfecto Mobile
Antoine Aymer Product Marketing @HP
A
Page 2
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 2
How different is Mobile?
A
Page 3
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 3
Type
Click Double click Right click Drag
Tap & Multi tap Double / long tap
Drag Swipe
Pinch & zoom
Locate
Type
Rotate / Tilt
Snapshot
Authenticate
Communicate
Speak
A
Page 4
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 4
CES 2014, Smart Innovation
BYOD Inflight Entertainment
Contactless payment
Smart
e-bike
Smart gardener
Smart clothes
Smart nappy
Smart appliance
Augmented reality
A
Page 5
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 5
Starbucks’ app
Features: • Pay for your purchases • Track and redeem rewards • Reload Starbucks Card’s balance • Get directions to the nearest store • Explore coffees, beverages, food and
nutritional information • Send eGifts instantly to your contacts • Stay informed by being the first to
hear of special offers
Improve in-store customer experience
Generate customers loyalty
Increase sales
A
Page 6
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 6
But this is not enough… A
Page 7
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 7
Use case #1
SaaS-based applications are expected to generate a growth rate of 17.4%.
Sport App connected with smartwatch
Unit test
• Unit test • 100% emulation
Interoperability
• Data transfer via Bluetooth • 100% manual/human testing
Industrialisation
• Functional testing on major devices • 80% automated testing
A
Page 8
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 8
Why Mobile Testing looks so difficult?
Page 9
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 9
It is the jungle!
versus versus
Development platform
Various OS types, versions, device/tablet, OEMs
Evolving
Browsers
Page 10
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 10
DEVICE
TESTING
MARKET
Performance Network (>400 operators)
Contextual issues Peripherals Impact on back-end Emulator vs real mobile
Automation vs manual Device management What Devices to test on? Bug-fix latency Unavailable services Off-shore
Time to market
Fragmented landscape (OS, version, input, screen, style)
USER Know the user Users’ feedback
Page 11
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 11
Why Mobile Testing looks so difficult
A
48% lack
expertise
38% lack tools
Page 12
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 12
Mobile testing disciplines
Functional
• Validation of Functionality
• Smoke / Regressions Testing
• Offline access testing
• Negative Testing
Non Functional
• Network Strength / Outage / Recovery
• Different NW Types
• Peripheral Testing
• Services
Interoperability (IOP)
• Voice / SMS interrupts
• Notifications
• Battery /Cable Removal
Memory Leak
• Memory Usage
• Memory Leaks
• Garbage Collection
Performance Testing
• CPU Usage testing
• Network Usage
• Page Render time or activity Render time
Usability Testing
• User Experience
• Competitive Analysis
• Expert Review
Installation Testing
• New App Install
• Uninstall and Reinstall
• Upgrade testing
Security Testing
• OWASP Vulnerabilities
• Dynamic Testing
• Static Code Analysis
• Data Encryption
Language Testing
• Validation for Locales
• Images and Text
• Currencies, time zones etc.
• Context
Page 13
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 13
Definition of Fully Tested
Page 14
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 14
The Truth Is Out There! Which of the following statements reflect your development team's approach to testing mobile apps?
4%
21%
27%
31%
34%
55%
55%
55%
60%
We don't formally test or instrument our appsbefore deploying them; we fix changes as needed
We integrate crash reporting/analytics into ourdeployed application to monitor and fix problems
We automate our testing process as part of acontinuous integration process
We use a cloud-based service to test multipledevices
We test devices that are deployed on multiplecellular carriers
We do a quick test on an emulator/simulator or ona single device
We test across multiple devices that run the sameversion operating system
We test across multiple OS versions
We test our app across multiple device orientations
Source: Forrester, Q1 2013 Base: 765 Software developers developing mobile applications
A
Page 15
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 15
Mobile is Agile on Steroids
A
Page 16
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 16
The ideal Mobile testing picture
Cum
ula
tive
bu
sin
ess
valu
e
Time
Sprint 4 Sprint 3 Sprint 2 Sprint 1
CI CI CI CI
V1.0 V1.1 V1.2
Release, Iteration, Nightly, Continuous
End-to-End test cycle
On any device, by any team, from any location
Longevity testing (independent of sprint cycles)
Page 17
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Agile and Testing -- 5 o’clock on Friday the 30th, do you know if your code is “done”?
Page 18
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 18
The Truth Is Out There!
52% spend
more than an hour debugging/fixing
code
49% of
developers spend more than an hour a day testing new
code
Source: Forrester, Q1 2013 Base: 765 Software developers developing mobile applications
only 15% spend that much time writing test automation code
A
Page 19
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 19
Agile mandates automation
Page 20
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 20
Use case #2
SaaS-based applications are expected to generate a growth rate of 17.4%.
Top US Bank
Page 21
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 21
Use case #3
SaaS-based applications are expected to generate a growth rate of 17.4%. Type of Testing
3 weeks
Mobile banking, 12 Millions unique users native application and mobile enabled website
Impact
manual
Deployment
In-house
A
Page 22
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 22
Use case #3
SaaS-based applications are expected to generate a growth rate of 17.4%. Type of Testing
3 weeks
Mobile banking, 12 Millions unique users native application and mobile enabled website
Automated
Deployment
Cloud
48 hours
86% time savings
60% cost savings
Impact
A
Page 23
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 23
Automation requirements
Page 24
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 24
What can get in the way?
Silos App Style Cross-Platform
Differences Use cases
web store
Page 25
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 25
Cross-platform differences
Menu panel
Text colours
Images
Name position
Labels
Button images
Page 26
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 26
Building an Enterprise Grade Mobile Testing Strategy
Page 27
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 27
Sprint 4 Sprint 3 Sprint 2 Sprint 1
CI CI CI CI
Start
Transform Grow
Predictability, scalability
What fits my workflow?
Coverage, Reliability & Governance
Page 28
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 28
Finding the right balance
Coverage
Devices
Service Level
Governance
Existing Tools
Time to Market Locations &
Networks
A
Page 29
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 29
HP UFT Mobile
Page 30
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 30
na
tiv
e
#1 HP UFT Add-in #2 Deployment #3 Devices
Page 31
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 31
HP UFT Extension Integrated with HP ALM
Real mobile devices (and emulators) Perform almost any operations
Device agnostic Develop, run and schedule one
test across multiple platform Visual & Object analysis
Manageable Full IT control on authentication, access permission and activities
Predictable availability
Global Accessible from any location (VPN mandatory with MCE)
Secured Full tractability of actions. Device cleanups and encryption abilities Continuous testing
Support fast, agile delivery
Scalable From single, small project to hundreds of simultaneous users
Complex scenario advanced built-in functionalities such voice injection and recognition, NFC, location based #1 HP UFT Add-in
Page 32
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 32
#2 Deployment
Public Cloud
Virtual Private Cloud
MobileCloud Express
Page 33
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 33
Use case #4: Virgin Media
Virgin Media UK Offices V
irg
in M
edia
Co
rpo
rate
Net
wo
rk
On-premise lab
Team members
Core team
Off-shore
Virtual Private Cloud Dedicated devices in various geographies
Multiple devices in various geographies
Public Cloud
Page 34
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 34
#3 Devices
List of supported devices: http://www.perfectomobile.com/portal/cms/services/handsets.html
Wide device spectrum HTC, Apple, Nokia, Motorola, Samsung, SonyEricsson, LG,
BlackBerry, Palm, Kindle
Full device control Fast charging cycle, audio injection, reboot, clean/wipes
Scalable Additional handsets are added on-
demand
Global The handsets resides in various datacenters in the world: US, Canada, UK, France, Germany, Spain, South Africa, India.
Deploy your app Testers can deploy their application on the device, test it as if you were holding the handset in your hands.
Powerful replay all tests are video recorded to allow
easy recreation of defects
Page 35
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 35
SCO
PE
AU
TOM
ATIO
N
Choose the right devices
Define relevant tests across all target devices
Traditional non-mobile automation rules still
apply
2
(before the demo)
Be intimate with manual testing
Think about future maintenance
Implement continuous integration
Gain mobile objects experience
Always sanity test your previous version
1
3
4
6
5
7
8