Testdroid: Different android test automation frameworks - what works you the best?

1

webinar

Different Android Test Automation Frameworks – What works you the best?

21 November 2013

Ville-Veikko HelppiTechnical Product [email protected]

mailto:[email protected]

3

Agenda• Different Ways to Automate Your App Testing• The Benefits and Characteristics of the Top 5

Android Test Automation Frameworks• 10 Tips for Professional Mobile App Testing• Testdroid Update• Q&A

webinar

© Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved.

4



webinar


5

Different Approaches To Testing webinar


6

Different Ways of Doing Test Automation

webinar


Automatic test exercisers Record and PlaybackHand written test scripts

Benefits:

Accurate, specific to your testing needs, plenty of options with frameworks, tools

Fast to create, accurate, not as sensitive to human-errors as hand-written tests, tools avail’ty

Fastest & extremely automated, excellent for smoke testing/quick testing, availability

Tradeoffs:

Takes a lot of time, ties resources to write test cases/scripts, error-prone (humans)

Compelling Recorder+Playback tools available for only few test automation frameworks

Not accurate as real test cases

7

Option #1: Cloud-Based Testingwebinar


Developers

Source code

Repository

Build failed

Polling

Notifying

Successful build

QA & Testers

Building

Error reporting

Test Cases

Application

Users / Consumers

Rele

ase

Com

mit

8

Option #2: Continuous Integrationwebinar


Developers

Source code

Repository

Build failed

Polling

Notifying

Successful build

QA & Testers

Building

Error reporting

Test Cases

Application

Users / Consumers

Rele

ase

Com

mit

Manualwebinar

© Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 9

Automation

Smaller coverage, More money burnt & time wasted, Error-prone

Large coverage, quickly

completed, Less money & time wasted, Exact

results.

10

How to Improve the App Quality• Automate generic things as much as you can• During the development your app changes –

the same must apply for testing!• Carefully select (testing) technology & partner• Use all 24 hours per day with test automation• Cloud-based platform is the only solution to

get you quickly covered in the global markets

webinar


11



webinar


12

Android Instrumentation Framework

webinar


• InstrumentationTestRunner – the primary plumbing for running tests on Android

• Android Instrumentation Framework is built on top of JUnit – a standard test framework on for any Java development

• Mock objects – methods for creating mock system objects such as content, service and intent.

13

Family Tree of Test Frameworkswebinar


JUnit

Android Instrumentation Framework

Robotium Espresso

UI Automator

Calabash

Appium

14

“It’s like Selenium but for Android”

Robotium: Basicswebinar


• Open source library extending JUnit with plenty of useful methods for Android UI testing

• Powerful and robust automatic black-box test cases for Android apps

• Supports native, hybrid and mobile web testing

• Semi-active open source community with monthly releases

15

Robotium: Code Examplewebinar


// Public void for the operation public void testRecorded() throws Exception { // Wait for the text 'Hello!' to be shown for newbie if (solo.waitForText("Hello!")) { // R class ID identifier for 'Sign in' - and click it solo.clickOnView(solo.findViewById("com.twitter.android.R.id.sign_in")); // R class ID identifier for entering username solo.enterText((EditText)

solo.findViewById("com.twitter.android.R.id.login_username"),"username");

// R class ID identifier for entering password solo.enterText((EditText)

solo.findViewById("com.twitter.android.R.id.login_password"),"password"); // R class ID identifier for clicking log in solo.clickOnView(solo.findViewById("com.twitter.android.R.id.login_login")); // Wait until log in is done solo.waitForActivity("HomeTabActivity"); } // Activate the text field to compose a tweet solo.clickOnView(solo.findViewById("com.twitter.android.R.id.menu_compose_tweet")); // Type the tweet solo.enterText((EditText) solo.findViewById("com.twitter.android.R.id.edit"), "Testdroid"); // Tweeting! solo.clickOnView(solo.findViewById("com.twitter.android.R.id.composer_post")); }

16

Robotium: ExtSolo - Extension Library

webinar


• Includes useful methods that have not been merged to Robotium, for instance:– Automatic scaling of x,y clicks for any resolution– Multi-path drags– Automatic screenshots on test failure– Mock locations– Change device language– Control WiFi connection

• You can download the library at http://testdroid.com/products/apis-plugins-and-libraries

http://testdroid.com/products/apis-plugins-and-libraries



17

Calabash: Basicswebinar


• Behavior driven test framework for native Android, native iOS and mobile web

• Tests are described in Cucumber and then converted to Robotium or Frank in run time

• Supports currently about 80 different natural language commands (controllers). New controllers can be implemented in Ruby or Java

• If your app uses custom UI Widgets you have to create controllers for those in order to test your app

• Calabash tools include a command line inspector for finding right UI element names/ids

18

Calabash: Architecturewebinar


• Calabash installs an HTTP server as an instrumentation package that listens commands from Calabash server

• Tests are executed on server side• Each test scenario is described in

Cucumber• Ruby Client library converts Cucumber

commands to either Robotium or Frank method calls

• Webview support is implemented with Javascript injection to the Webview

Android device

Your app

Calabash HTTP server

Test server

Step definitions, Ruby client

library

Features

19

Calabash: Code Examplewebinar


Feature: Login feature Scenario: As a valid user I can log into my app I wait for text "Hello" Then I press view with id "Sign in" Then I enter text "username" into "login_username" Then I enter text "password" into "login_password" Then I wait for activity "HomeTabActivity" Then I press view with id "menu_compose_tweet" Then I enter text "Testdroid" into field with id "edit" Then I press view with id "composer_post"

20

uiautomator: Basicswebinar


• Google’s test framework for testing native Android apps across device

• Works only on Android API level >=16• Runs JUnit test cases with special privileges (test

cases can span across different processes)• No support for web view (only handle to web view

elements as canvas object)

21

uiautomator: Code Examplewebinar


// Public void for the operationpublic void testSignInAndTweet() throws Exception { // Starting application: getUiDevice().wakeUp(); // Press Home button to ensure we're on homescreen getUiDevice().pressHome(); // Select 'Apps' and click button new UiObject(new UiSelector().description("Apps")).click(); // Select 'Twitter' and click new UiObject(new UiSelector().text("Twitter")).click(); // Locate and select 'Sign in' UiSelector signIn = new UiSelector().text("Sign In"); // If button is available, click UiObject signInButton = new UiObject(signIn); if (signInButton.exists()) { signInButton.click(); // Set the username new UiObject(new

UiSelector().className("android.widget.EditText").instance(0)).setText("username"); new UiObject(new

UiSelector().className("android.widget.EditText").instance(1)).setText("password"); new UiObject(new UiSelector().className("android.widget.Button").

text("Sign In").instance(0)).click(); // Wait Sign in progress window getUiDevice().waitForWindowUpdate(null, 2000); // Wait for main window getUiDevice().waitForWindowUpdate(null, 30000); } new UiObject(new UiSelector().description("New tweet")).click(); // Typing text for a tweet new UiObject(new UiSelector().className("android.widget.LinearLayout").instance(8)).

setText("Awesome #Testdroid!"); // Tweeting! new UiObject(new UiSelector().text("Tweet")).click();}

22

Appium: Basicswebinar


• Uses Selenium Webdriver (W3C standard) as a scripting framework

• Supports native Android, native iOS and mobile web:– Android via uiautomator (API level >=16) and Selendroid

(API level <16)– iOS via UI Automation– Mobile web as Selenium driver for Android and iOS

• You can write your Appium scripts on almost any programming language (Haskell/Go/Clojure/Java/Ruby)

23

Appium: How it works?webinar


• Appium is an HTTP server that creates and handles WebDriver sessions

• It starts an Appium server on the device that is listening commands from the main Appium process

• It receives json requests from client libraries over HTTP

• On Android Appium executes these commands as either uiautomator or Selendroid commands depending on the API level of the device

• Tests are driven from a Selenium script on Appium server

Android device

Your app

Bootstrap.jar

Test server

UI Automator controller or

Selendroid driver

Selenium script

24

Appium: Test Lifecyclewebinar


1. Appium installs Bootstrap.jar as an instrumentation package which contains an uiautomator server and a tcp server

2. The commands from a Selenium script are executed on the main Appium server that relays the commands over TCP/IP to the tcp server running on the device

3. The UI Automation server converts the Selenium commands to uiautomator commands on the fly

4. If the device has API level <=17 Appium installs a Selendroid server that converts the Selenium commands to Android Instrumentation (JUnit) commands

5. All test verification and result processing is happening on server side in the main Appium Server

25

Appium: Code Examplewebinar


# wait for hellosleep(3)textFields = driver.find_elements_by_tag_name('textField')assertEqual(textFields[0].get_attribute("value"), "Hello")# click sign-in buttondriver.find_elements_by_name('Sign in')[0].click()# find the text fields again, and enter username and passwordtextFields = driver.find_elements_by_tag_name('textField')textFields[0].send_keys("twitter_username")textFields[1].send_keys("passw0rd")# click the Login button (the first button in the view)driver.find_elements_by_tag_name('button')[0].click()# sleep sleep(3)# click the first button with name "Compose"driver.find_elements_by_name('Compose')[0].click()# type in the tweet messagedriver.find_elements_by_tag_name('textField')[0].send_keys(”#Testdroid is awesome!")# press the Send buttondriver.find_elements_by_name('Send')[0].click()# exitdriver.quit()

26

Espresso: Basicswebinar


• The latest Android test automation framework from Google• A custom Instrumentation Testrunner with special privileges• Works on API levels 8 (Froyo), 10 (Gingerbread), 15-19 (IJK)• Thin layer on top of Android Instrumentation Framework• Uses the Hamcrest matcher library

https://github.com/hamcrest

https://github.com/hamcrest

27

Espresso: Basics 2webinar


• Easy API for extending the framework:– You can write new matchers: onView(myCustomMatcher<View>)– You can write new actions: perform(myCustomAction)– You can write new checks: check(myCustomAssertion)

• Reliable: Synchronizes with the UI thread• It’s fast because there is no need for any sleeps (tests run on

same millisecond when the app becomes idle)• No support for web views• https://code.google.com/p/android-test-kit/

https://code.google.com/p/android-test-kit/

https://code.google.com/p/android-test-kit/

28

Espresso: Code Examplewebinar


public void testEspresso() { // Check if view with the text 'Hello.' is shown onView(withText("Hello.")).check(matches(isDisplayed())); // R class ID identifier for 'Sign in' - and click it onView(withId(getInstrumentation().getTargetContext().getResources() .getIdentifier("com.twitter.android:id/sign_in", null, null))).perform(click()); // R class ID identifier for entering username onView(withId(getInstrumentation().getTargetContext().getResources()

.getIdentifier("com.twitter.android:id/login_username", null, null))).perform((typeText("username"))); // R class ID identifier for entering password onView(withId(getInstrumentation().getTargetContext().getResources()

.getIdentifier("com.twitter.android:id/login_password", null, null))).perform((typeText("password"))); // R class ID identifier for clicking log in onView(withId(getInstrumentation().getTargetContext().getResources()

.getIdentifier("com.twitter.android:id/login_login", null, null))).perform(click()); // Activate the text field to compose a tweet

onView(withId(getInstrumentation().getTargetContext().getResources() .getIdentifier("com.twitter.android:id/menu_compose_tweet", null, null))).perform(click()); // Type the tweet onView(withId(getInstrumentation().getTargetContext().getResources() .getIdentifier("com.twitter.android:id/edit", null, null))).perform((typeText(”#Testdroid"))); // Tweeting! onView(withId(getInstrumentation().getTargetContext().getResources() .getIdentifier("com.twitter.android:id/composer_post", null, null))).perform(click()); }

29

Summary – Comparison Matrixwebinar


Robotium uiautomator Espresso Appium Calabash

Android Yes Yes Yes Yes Yes

iOS No No No Yes Yes

Mobile web Yes (Android)

Limited to x.y clicks

No Yes (Android & iOS)

Yes (Android)

Scripting Language

Java Java Java Almost any Ruby

Test creation tools

Testdroid Recorder

UI Automator viewer

Hierarchy Viewer

Appium.app CLI

Supported API levels

All 16 => 8, 10, 15-19 All All

Community Contributors Google Google Active Pretty quiet

30



webinar


32

Tip #1: Test early, Test oftenwebinar


33

Tip #2: Plan What to Automatewebinar


People

Infrastructure

Tools

Training

Time

MAN

UAL

AUTO

MATED

34

Tip #3: Use only Real Deviceswebinar


35

Tip #4: Use Atomic Test Unitswebinar


Test iteration #1 Test iteration #2

36

Tip #5: Create Hermetic Testswebinar


37

Tip #6: Use All Possible Deviceswebinar


Testdroid Cloud’s 250+ Android devices = 93-95% global Android volumes!

38

Tip #7: Separate Apps and Testswebinar


Your App

39

Tip #8: Output Everything to Logswebinar


40

Tip #9: End-to-End Testingwebinar


41

Tip #10: Integrate Dev & Testingwebinar


Com

mit

Developers

Source code

Repository

Build failed

Polling

Notifying

Successful build

QA & Testers

Building

Error reporting

Test Cases

Application

Rele

ase

Users / Consumers

42



webinar


43

Testdroid Productswebinar


Complete Solution for Mobile Apps/Games Testing

44

webinar


Testdroid – Because it is important to know what app testing costs!

45

webinar


Testdroid Blog and Webinars– Because it is important to how to automate your testing!

46



webinar


47© Copyrights by Bitbar Technologies Ltd. 2013

All rights reserved.

webinar

Testdroid: Different android test automation frameworks - what works you the best?

Technology

testing copyrights

bitbar technologies

real test cases copyrights

test casesscripts

testing needs

standard test framework

mobile web testing

errorprone copyrights