© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 1 2 December 2014 Ville-Veikko Helppi Technical Product Manager [email protected] Learn the Best Practices in Mobile Game Testing W E B I N A R
Jul 07, 2015
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 1
2 December 2014
Ville-Veikko HelppiTechnical Product Manager
Learn the Best
Practices in Mobile
Game Testing
W E B I N A R
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 2
Agenda
• Different Types of Testing and How Game
Testing Differs from Regular App Testing?
• Frameworks, Tools and Technologies – What is
Important in Mobile Game Testing Today?
• Top Priorities for Mobile Game Testing
• Infrastructure for Successful Game Testing
• Hands-on Example (Image Recognition)
• Testdroid Update
• Q&A
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 3
Let's Define 'Testing'...
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 4
Different Types of Testing?
• Black-box testing vs. White-box testing
• Methods used in Mobile Game Testing
– Functional
– Compatibility
– Performance
– Localization
– Regression
– Load
– Smoke
– Acceptance
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 5
Defining Some Ways to Test
• Functionality of the game-play, featuresFunctional Testing
• Done always when new features /regressions are includedRegression Testing
• How game runs on different configurationsCompatibility Testing
• Different languages, geo-focused materialsLocalization Testing
• Endurance test to determine if system can handle the loadSoak Testing
• Measures the capacity of the system Stress Testing
• Simplest form of performance testing, measures how system handles certain loadsLoad Testing
• Isolation of the environment (e.g. from network) to see how game worksHermetic Testing
Feature
-based
testing
Performance
testing
End-user
testing
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 6
The Old-Fashion Thinking
Deterministic systems
Identifiable objects
Tool support(s)
User-driven actions
Simple inputs/outputs
Virtualization
Random setups/results
No handle on objects
No standards
User reacts to input
Complex inputs/outputs
Real hardware
<>
<>
<>
<>
<>
<>
High-level of automation Low-level of automation
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 7
Test Automation as a Solution
Random setups/results -> Easy to generate random events
No handle on objects -> Variety of options (e.g. image
recognition)
No standards -> Variety of frameworks (cross-platform
etc.)
User reacts to input -> Test scripts can handle random
events
Complex inputs/outputs -> Test scripts can generate input/content
Real hardware -> Availability of real mobile hardwareTest Automation brings real value
with variety of options to mobile game testing
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 8
Challenges that Test Automation Tackles
• Proliferation of Devices, Brands, Operators,
Platforms
• Coupling the Testing and Development process
(Continuous Integration/Deployment/Delivery)
• Different Types of Testing (as automated)
– Stress, Load, Performance, Functionality etc.
• Test script & test asset usability
• Cons of Manual testing
– Tedious effort, Expensive human resources, Error-prone
(as done by humans) & Not reusable (done once)
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 9
OS
versions
ChipsetsCPU + GPU
Tens of
OEMs
Memory
Displays (resolutions,
physical
hw)
OEM
mods
Other
hardwar
e (connectivity,
calibration)
Relation
to other
software
Significant Benefits of Using Automation
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 10
Correct behaviour
across platforms and
browsers
Integration with web
back-ends
Typically need to
fully utilize HW
(CPU+GPU)
Resource (e.g.
battery)
consumption
OpenGL ES 2/3
Functionality
and usability
Screen
orientations,
connectivity,
user profiles
Robustness
Robustness and
security!
Brand
Compliances,
verification with back-
ends and data
Different 'Verticals' Require Different Practices
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 11
Importance of
Mobile Game Testing
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 12
Important Aspects in Game Testing
• User Interface and Functionality
– UI layouts/elements, Menus, Resolution,
Orientation, Overall visuality
• Graphics Performance
• Usability and User Experience
– Responsiveness, Real Occuring Events
• Multi-player/User
• Social Integrations
• Security and Liabilities
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 13
User Interface and Functional
• Compared to regular app testing, mobile game
testing may seem to be more challenging
– e.g. OpenGL ES content not action-identifiable
• Graphics and UI – Identical look&feel across all
devices and variants
• Usability controls in the context of game
• Navigation/game-play, Progress, Delays
between "game-stages"
• Animations, Fonts, Other graphics assets
• Screen Resolution and Orientation
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 14
Performance Testing
• Performance, Load, Stress, Spike, Soak
• Measure the real
user experience
• Execute tests with
all possible devices
(SW&HW)
• Discover the impact
of server loads, app
performance
(CPU/GPU
utilization)
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 15
• Studies: Majority of
users leave game due
sluggish, general
slowness or other bad
performance behavior
• It's very important to
understand how game
behaves on different
devices, under
different circumstances
Performance Makes +/- UX
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 16
Building the User Experience
UnderstandingHow quickly gamer assimilates what the
game is about, how to fully play with it, what
are the functionalities.
UsabilityThe main User Interface, menus, design and
how usable those are
Playing ExperienceIntuition to follow the game and go forward
with its instructions and game-play
MonetizationAre gamers willing to buy – invest – in game
• Graphics performance
capability is the most
important thing for mobile
games when forming UX
• Too many games fail to
work properly with low-
end/mid-range devices due
hardware performance
• The real understanding of
game's performance
capability can be only
figured out by testing on
real devices
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 17
• End-to-end between tests, AI or
more simple implementations
• Optimize the data communication
between server and clients
• Test simultaneously on several
(different) devices
• Check what happens if game gets
interrupted (platform)
• Location-based features – how to
properly mock up those?
• Hermetic environment – what
happens?
Multi-Player/User Features
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 18
• Important part of the after-
engagement (how gamers
share their results)
• Many of OEM branded
devices provide certain
SoMe apps pre-installed
(version?)
• Connectivity, login and
data transferring are
tested in context of game
• Hermetic environment –
what happens?
Social Integrations
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 19
Security and Liabilities
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 20
4 Top Priorities in
Mobile Game Testing
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 21
Localization
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 22
Device Diversity
By OpenSignal, Inc. 2014 - http://opensignal.com/reports/2014/android-fragmentation/
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 23
Real Devices vs. Emulators
• Emulators/simulators won’t help you to test…
– User Experience and Usability
– Hardware
– Software
– Infrastructure
0 % = the percentage of your app users
that use emulator to run your app!
Why Real Devices are Must-to-Have?
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 24
Manual vs. Automation
Smaller coverage, More money burnt & time wasted, Error-prone
Large coverage, quickly
completed, Less money & time wasted, Exact
results.
Manual Automation
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 25
Infrastructure for
Mobile Game Testing
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 26
High-Level Infrastructure
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 27
Architecture / Infrastructure
GAME
Gam
e
Engin
e
Android
iOS
...
API
&
services
Server
Database
App Middleware Platform Connectivity/Network Back-end
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 28
Isolated, Hermetic Testing
GAME
Gam
e
Engin
e
Android
iOS
...
API
&
services
Server
Database
Hermetic Environment
App Middleware Platform Connectivity/Network Back-end
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 29
Compatibility & Porting Testing
GAME
Gam
e
Engin
e
Android
iOS
...
API
&
services
Server
Database
Real Devices
App Middleware Platform Connectivity/Network Back-end
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 30
Smoke/Reference Testing
GAME
Gam
e
Engin
e
Android
iOS
...
API
&
services
Server
Database
Reference Environment
App Middleware Platform Connectivity/Network Back-end
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 31
Back-End Performance Testing
GAME
Gam
e
Engin
e
Android
iOS
...
API
&
services
Server
Database
Massive no. of users and significant load
App Middleware Platform Connectivity/Network Back-end
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 32
Continuous Integration
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 33
Game Testing Process
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 34
Hands-on Example:
Integrating Development with
Testing Environment
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 35
• Continuous Integration (e.g. Jenkins)
• Appium framework (for Android and iOS)
• No need for devices – we'll use devices on
Testdroid Cloud service
• Test scripts / test cases for game
Setup
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 36
Continuous Integration
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 37
Appium Intro
Family Tree: Android Test Automation Frameworks
Appium works for both, Android and iOS
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 38
© Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 38
• Uses Selenium Webdriver (W3C standard) as a
scripting framework
• Supports native Android, native iOS and mobile
web:
– Android via uiautomator (API level >=16) and
Selendroid (API level <16)
– iOS via UI Automation
– Mobile web as Selenium driver for Android and iOS
• You can write your Appium scripts on almost any
programming language
(Haskell/Go/Clojure/Java/Ruby)
Appium Basics
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
39
Test Script
Test Case
Desired Capabilities
{
“device”: “Android”,
“app”: “/Users/user/ApiDemos.apk”
“app-package”: “com.example.android.apis”
“app-activity”: “.ApiDemos”
}
Desired Capabilities / Android
Example
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
40
Test Script
Test Case
Desired Capabilities
{
“app”: “com.bitbar.testdroid.BitbarIOSSample”
}
Example
Desired Capabilities / iOS
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
41
Test Script
Test Case
Desired Capabilities
Appium @ Localhost
WebDriver
http://localhost_4723/wd/hubAppium Server
4723
Device
Example
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
Test Script
Test Case
Desired Capabilities
From Localhost to Testdroid
Cloud
WebDriver
http://localhost_4723/wd/hub
*Testdroid Caps
http://appium.testdroid.com/wd/hub
Example
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
Test Script
Test Case
Desired Capabilities
From Localhost to Testdroid
Cloud
WebDriver {
“testdroid_username”: “[email protected]”,
“testdroid_password”: “p4s$w0rd”,
“testdroid_project”: “My First Project”,
“testdroid_testrun”: “Test 1”,
“testdroid_device”: “iPad Mini 7.0.4 A1432”,
“testdroid_app”: “http://domain.com/app_v1.ipa”
.
.
“app”: “com.bitbar.testdroid.BitbarIOSSample”
}
Example
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
44
Example
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 45
350+ Android & 20+ iOS Devices
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
46
Example
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
47
Example
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
48
Example
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
49
Example
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
50
Example
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
51
Example
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 52
Example
Projects and Test Runs
AppiumBroker
5 Appium Ready
sessionid
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
53
Test Script
Behind the Scene
Device Cluster
Device 1
Device 2
Device 3
1 WebDriver Session Request@ http://appium.testdroid.com/wd/hub/
Desired Caps, .apk / .ipa
2 Configure project
3
Wait for device to become available
4 Start Appium
Session Map
WebDriver Session response6
sessionid
7 Test Run
sessionid Proxy
Appium
Device 1
Example
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 54
Image Recognition
The basic example of Image Recognition
in Mobile Game Testing: Template
Matching
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 55
Code Example
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 56
Code Example
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 57
Code Example
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 58
Code Example
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 59
• Test Script contains the following 'steps':
1. Identify the used platform (Android / iOS)
2. Search for 'Goldmine' graphical element
(using find_image to search for screen)
3. Script runs test to go to shop
4. Script runs test to buy a cannon
5. Script runs test to place the cannon (all
these three are defined in .png files)
6. Script runs test to start a battle!
7. Game is brought down and test ends.
Example: Clash of Clans
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 60
VIDEO
Image Recognition with Real Android and
iOS Devices in Local Environment
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 61
Testdroid Update
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 62
New UI in Testdroid Cloud 2.0
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 63
Testdroid Cloud's Interactive
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 64
Testdroid
PrivateCloud -
Running user-
selected devices
at any of our
hosting facilities
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 65
NEW HOSTING CENTER!Mountain View, CA
San Francisco
and
Mountain View
datacenter
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 66
Option 1:
Public Device Cloud
on-demand devices
(multitenant)
Mobile app testing on
over 400+ real
Android and iOS
devices hosted by
Bitbar
Option 2:
Private Device Cloud
reserved devices
Hosted by Bitbar in the
US and/or Europe
Devices chosen by and
reserved only for the
Customer
Option 3:
On-premise Device
Cloud
Automated mobile
app testing on real
Android and iOS de-
vices hosted by the
customer, usually
30-500 devices
Testdroid Deployment Options
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 67
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 68
Video recording & slides of this webinar will
be soon available at Vimeo and Slideshare!
Please visit at www.testdroid.com for more
information.
THANK YOU!
W E B I N A R
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 69
We operate the largest global device
cloud with over 400 devices
instantly available for developers
Devices from the US, Europe, China, Japan and Korea
– global devices for global customers