Page 1
DISCLAIMER
This handout is made possible by the support of the American people through the United States Agency for International Development
(USAID). The contents are the sole responsibility of Mercy Corps and do not necessarily reflect the views of USAID or the United States
Government.
Think Tank Event: Mobile App Testing
Samer Desouky, TestPRO
Page 2
• Why Mobile Testing is Important
• Key Challenges
• App Types
• Test Environment
• Testing Challenges
• Tools
Topics
Page 3
• The Degree of meeting or exceeding the expectation.
• The process concerned with evaluation of software products to:
• Determine if the product satisfy specified requirements.
• Fit for purpose.
• Detect defects.
• Testing Measure Quality.
“ There are a million ways to do something wrong but only one way to do it right.”
Quality & Testing
Page 4
• Software Failures Cost up to
$60 Billion to US economy
• 16.2% completed on time and
on budget
• 31.1% of projects are
cancelled before they ever
get completed.
• 52.7% of projects will cost
189% of their original
estimates
http://www.articlesbase.com/technology-
articles/why-software-fails-5492086.html
Quality & Testing
Page 5
• A project is said to have failed when :
• Unable to complete in the allotted budget and resources.
• Delivers later than it was decided.
• Unable to meet the requirements.
• Failed projects could be of the following 4 Kinds:
• Failure caused by bad management
• Failure caused by poor team effort
• Failure caused by poor end product
• Failure caused by errors that could have been caught
Quality & Testing
Page 7
Dumb Phone
•Voice calling and text messaging
Feature Phone
•Additional functions
•Limited or no web access
•Limited ability to run 3rd app
Smart Phone
•Advanced computing capability and connectivity
•Combined mobile phone with PDA
Evolution
Page 9
• Smartphones are rapidly becoming the primary method of
interaction for consumers and businesses worldwide.
• International Data Corporation (IDC) predicts that 182.7 billion
mobile apps will be downloaded by 2015.
• That’s a 1600%
increase from
the 10.7 billion
apps downloaded
in 2010.
Penetration
Page 10
• Loss of Time & Money
• Damage to business reputation
• Injury or death
• Word-of-mouth
• Mobile testing services should drive 20% - 40% reduction in costs
while improving speed to market, productivity and quality.
Increase
Quality of Testing
Test Coverage
Release confidence
What if apps fail?
Reduce
Time to Market
Testing Resources
Defect Resolution Time
Overall Testing Costs
Page 11
• Cost of Quality divided into 2 categories:
• Cost of Achieving Quality :
• Cost of prevention
• Cost of detection
• Cost of Poor Quality:
• Cost of internal failure
• Cost of external failure
Cost of Quality
• A portion of the testing budget is a cost of detection and internal failure
which is typically well below the cost of external failure which is 10 –
1000 times.
Page 13
• Multiplicity
• User Expectation
Challenges
Page 16
• Location-intelligent mobile commerce Retail
• Magazines & newspapers going 100% digital Media
• Mobile bookings, check-ins, maps, deals Travel
• Tablets and apps in classrooms Education
• Patient records, physician notes, etc. Lifestyle
• Games and social media platforms Social
Industry
Page 18
• CTQs are the internal critical quality parameters that relate to the
wants and needs of the customer.
• CTCs are what is important to the customer.
• What’s CTQ to ensure the things that are CTC.
User Expectation
Page 19
Performance According to a recent study, 60% of mobile users will abandon your app or site if it doesn’t load within three seconds. Three seconds!
OS & device Having an app available on some devices and not others is a death sentence in the mobile world.
Security Personal data to remain personal – user expect their apps to be 100% free of viruses, malware and other threats.
Connection speed
Does your app run the same on 3G, 4G and Wi-Fi coverage.
Data usage Users are wising up to the fact that some apps drain battery faster than others.
Installation & launch
Is your application easy to install and launch?
User Expectation - CTCs
Page 21
• Native app
• An app build purely using the components build using the underlying platform
• Runs on device OS and firmware
• Calendar , Calculator , Camera
• Web app
• Browser app, web capable mobile devices
• It’s a web site which run on Mobile phones
• HTML, java script
• M.facebook.com
• Hybrid app
• Combination of Native and Web apps
• Has cross platform compatibility
• LinkedIN , Facebook
Types of mobile applications
Page 22
Similar
• Testing different
screen sizes and
form factors is
necessary
• Testing on real
devices is a must.
Types of mobile applications
Page 23
Types of mobile applications
Different
Page 25
Real Device
Simulators &
Emulators
Cloud-based
Test Device
Page 26
• Testing on real devices is not feasible and very expensive.
• Mobile emulators and simulators can help developers test their
apps on different devices without having to actually have them on
hand.
• These technologies serve as virtual pieces of hardware that
operate on a PC or laptop.
Test Device
Simulators
• Running a program to predict actual behavior
Emulators
• Running a program designed for a different system
Page 27
• Simulators & Emulators
Test Device
Page 28
• Real Devices
http://www.keynotedeviceanywhere.com/da-free-product-overview.php
Test Device
Page 29
Web browsers
http://gs.statcounter.com/#mobile_browser-ww-monthly-201303-201403
Android OS
http://developer.android.com/about/dashboards/index.html
Android Phones
http://www.appbrain.com/stats/top-android-phones
Test Device
• What target platforms and devices dose the app target?
• What are the popular devices in market?
Page 30
Sourcing Strategies
• In-House Testing
• Outsourcing Testing
• Beta Testing
• Crowd-sourced Testing
Sourcing Strategies
Page 31
• In-House:
• Pros:
• In-house team makes communication easier
• Professional testers who are tightly integrated into SDLC
• Cons:
• Capacity constraints
• Testing may not match user demographics
• Maintaining adequate collection of mobile devices is
prohibitively expensive
Sourcing Strategies
Page 32
• Outsourced Testing
• Pros:
• Professional testers on demand
• Access to range of devices
• More test types
• Cons:
• Hidden costs
• Not easily scalable
• Challenges in communication and time zone
• Still lab-based testing
Sourcing Strategies
Page 33
• Beta Testing
• Pros:
• No direct costs
• Lives outside the test lab environment
• Range of mobile devices, carriers, OS, locations
• Cons:
• Users are not professional testers
• Provide subjective feedback which cost for the company
• Puts unfinished product in front of customers
Sourcing Strategies
Page 34
• Crowd-sourced Testing
• Pros:
• Lives outside the test lab across real devices, carriers, OS
and locations
• Professional testers who mirror end users
• Access to range of mobile devices & connections
• Cons:
• Direct cost
• Requires communication efforts between testers &
developers
Sourcing Strategies
Page 36
Functionality
•Performing as specified in the specification
Usability
•Follow the design guideline for a particular OS
•MWBP http://www.w3.org/TR/mobile-bp/#cl
Interrupt
•SMS, call, shutdown, network outage
Security
•Device file system , encryption
Performance and Load
•Start-up Time, Response , Network, Memory
Challenges
Page 37
• Functional
• User Experience
• Performance and Load
• Security
• Interrupt
• Localization
Test Challenges
Page 38
• Gestures
• Orientation
• Social media integration
• Network connectivity
• Push notifications
• Media
• Local Settings
• Installation permissions
• Third-party services
• Failure handling
Functional Test
Page 40
Different interface, size of elements, usability of UI components
Orientation
Page 41
• Authorization via social networks
• Share
• Post messages: images, links
• Comments
• Login/logout
Social networks integration
Page 42
• Switching between mobile network and Wi-Fi
• Size of requests/responses
• Upload/download - Pause, Cancel
• Loss of connection
Network connection types
Page 43
• Receiving
• Open corresponding object
• Errors handling (e.g. object corresponding to received
notification don’t exist any more, empty notification,
etc.)
Push notifications
Page 44
Interaction with media content (photo, video,
sound) and external hardware:
• Usage of photos from library/camera
• Usage of microphone
• Audio/video playback
Media content
Page 45
• Language
• Date/Time format
• Volume
Usage of locale settings
Page 46
The No.1 rule is to follow the design guidelines specified
for the particular OS platform.
• Simplicity
• Context
• Good navigation
• Font sizes and Button sizes
• Portrait and landscape orientation considerations
User Experience
Page 47
• Start-up time
• Response time
• Network
• Memory consumption
• Battery consumption
Performance & Load
Page 48
• Requirements, prototypes, mock-ups
• Creating API tests
• Creating application’s Mindmaps ®
Start Testing
Page 49
• Cluttered
• Confusing
• Lost
• Quick and easy
• Location and Privacy data
• Test End to End Flow on each device
Start Testing
Page 51
Tools - Presentation
Page 52
Tools - Automation
Page 53
Tools - Automation
Page 54
Tools - Performance
Page 55
• Why Mobile Testing is Important
• Key Challenges
• App Types
• Test Environment
• Testing Challenges
• Tools
Topics
Page 57
Testing is our Profession
www.testproeg.com
Contact Us
+2 0222756841
+2 01021902447
+2 01000190709
30 Ahmad El-Zomor St., Nasr City, Cairo, Egypt
E-Mail: [email protected]