[email protected]PRESENTATION OVERVIEW PRESENTATION OVERVIEW − Why to test at all? − Richness of Android devices – a real opportunity − Android success story in numbers − Urban myths about mobile testing − Options for professional grade testing − Typical application implementations pitfalls − Some automated testing frameworks for Android 1
With more and more applications competing for users attention it is getting increasingly important to ensure that your app works on all Android devices as it should. If you launch your app without proper testing you risk wasting your marketing and development efforts by getting bad ratings and having your users disappointed just because your app does not work on their device.
In his presentation Jouko Kaasila from Bitbar goes through the most common approaches to tackle the testing challenge from application developer's point of view and explain why testing for device compatibility is a must for any serious Android application developer. He will also give examples of most common urban myths about testing and go through some best practices to minimize the cost and effort related to setting up your test infrastructure.
Jouko Kaasila is a co-founder of Bitbar, the company behind Testdroid Android testing tools. He has more than 14 years of experience in mobile software from early mobile services through Smartphone operating systems to Android platform. He is a serial entrepreneur with strong passion to build and help others to build businesses around mobile technologies and he is always keen to exchange views and experiences in this field.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
− Richness of Android devices – a real opportunity
− Android success story in numbers
− Urban myths about mobile testing
− Options for professional grade testing
− Typical application implementations pitfalls
− Some automated testing frameworks for Android
1
WHY TO TEST AT ALL?WHY TO TEST AT ALL?
Beta test approach
2
Systematic testing
Testing criteria: Some level of basic acceptance criteria needs to be created: What devices are supported, what use cases are important, what is the expected performance
Setting up test automation: Test cases are converted to test scripts, that are executed every time when the app changes
Testing throughout the development cycle: Testing starts when first features are implemented and continues even when the app is released
Pre launch activities: Light manual testing for catching most obvious bugs. Usually done on one or max 2-3 devices.
Beta phase: App is distributed to 10-100 beta users who will report bugs and any other issues
Launch: Based on the feedback from beta testing a new version is created and launched to the app stores.
WHY TO TEST AT ALL?WHY TO TEST AT ALL?
3
Some quotes:
Summary: Developers need to avoid treating end-users as beta testers and conduct pre-launch app testing to avoid critical reviews due to poor usability once released to market
“If there is a major bug that is not caught before the launch, then a lot of expensive development and marketing work is quickly wasted”
"The first wave of users gives the app bad ratings, which then undermine [the number of] downloads and the app would [eventually] sink into oblivion."
“Mobile design and development fits nicely with iterative development. Looping around the cycle of requirements, design, build and test in smaller chucks means that testing need not simply be a cost “tacked-on” to the end of a project. There are some types of such as penetration and security testing that should only be done towards the end of the project, but for functionality, performance and ease-of-use testing we recommend that this be done iteratively during the mobile project.”
“Notable from the survey is that 18% of responding companies say that they do not have enough time to test mobile apps, and 65% do not have the right tools. Also 52% cite lack of devices as a reason not to do mobile testing”
THE RICHNESS OF ANDROID ECOSYSTEMTHE RICHNESS OF ANDROID ECOSYSTEM
Fragmentation is innovation
Source: Open Signal Maps, May 2012
6Graphical representation of fragmentation in the Android ecosystem. Each rectangle represents the market share of one Android device. Samsung Galaxy S2, the most successful Android to date, has less than 10% market share.
Extensive use of web views− Webkits differ from one device to another− All sorts of timeout due to most content coming from server− Issues in http headers across devices− CSS&Javascript cause their own problems
Use cross platform tools− Especially the ones that generate the code on server side are very difficult to debug− Wrapping HTML5 to an app is quite ok− The time you save on development, you loose on debugging
HOW TO SET UP A PROFESSIONAL GRADE TESTINGHOW TO SET UP A PROFESSIONAL GRADE TESTING
13
Set up Continuous Integration and version control
Automate most common use cases− Address the basics first− Establish a repeatable baseline− Keep adding new tests as your functionality grows− Do not try to automate everything – there is still time and place for manual checks as well− Maintain your test library as your code evolves (store both in version control)
Define your target devices:− Depending your ambition level anything from 20 to 50 will do.− Test automatically and often with these devices− To get a reference point run occasionally tests with all possible devices (especially before any
TEST AUTOMATION FRAMEWORKS FOR ANDROID - TEST AUTOMATION FRAMEWORKS FOR ANDROID - OVERVIEWOVERVIEW
16
This is an overview of OPEN SOURCE Android testing frameworks – there are tens of closed proprietary testing tools/frameworks that are not covered here
AUTOMATION FRAMEWORKS FOR ANDROID – AUTOMATION FRAMEWORKS FOR ANDROID – MONKEY RUNNERMONKEY RUNNER
19
What:
- An API toolkit for writing programs that control an Android device or emulator from outside of Android code
Usage:
Pros:
- Only way to create tests that span across application boundaries
Cons:
- UI elements can only be clicked with x,y clicks that do not scale => you have to create one script for each resolution. There is no validation mechanism for any events created by clicks
AUTOMATION FRAMEWORKS FOR ANDROID – AUTOMATION FRAMEWORKS FOR ANDROID – MONKEY TALKMONKEY TALK
22
What:
- Cross platform testing tool/framework for Android and iOS
Usage:
Pros:
- Same script works for your Android and iOS app (if the apps are identical)
Cons:
- Uses proprietary scripting language. Requires you to wrap your app with a testing library so you will be testing a different binary that the one you are going to ship.