Best Practices in Mobile CI (webinar)

Best Practices for Mobile CI

April 28, 2015

© Sauce Labs, Inc.

Presenters

Kevin Rohling

Co-founder / VP of Product at Emberlight

Founder/CEO of cisimple now [email protected]

Kristian Meier

Sales Engineer at Sauce Labs

WHAT IS CONTINUOUS INTEGRATION FOR MOBILE?

AUTOMATICALLY…

Build Test Deploy

EVERY COMMIT.

CONTINUOUS INTEGRATION

HOW DOES IT WORK

1. Developer commits code

CONTINUOUS INTEGRATION

HOW DOES IT WORK

2. CI Server picks up the change.

CONTINUOUS INTEGRATION

HOW DOES IT WORK

3. Code is automatically compiled

CONTINUOUS INTEGRATION

HOW DOES IT WORK4. Tests are run

CONTINUOUS INTEGRATION

HOW DOES IT WORK

5. Application is deployed

CONTINUOUS INTEGRATION

WHY?

• Reduce the need for human-based testing

• Faster Feedback

• Automate EVERYTHING

WHY DO I USE CI?

• No Touch Configuration

WHY DO I USE CI?

• No Touch Configuration

• Automated OTA Distribution

WHY DO I USE CI?

• No Touch Configuration

• Automated OTA Distribution

• Code Validation: Automated Builds and Tests

ONE SIZE DOES NOT FIT ALL

Business Drivers Engineering Challenges

Early StartupInitial Traction,

FundingStability, UX

Consulting FirmClient Status Updates,

DeadlinesAd Hoc Distribution, Unit Testing

Large Product TeamRetention, Growth,

RevenueAutomation, Communication

PR

OC

ES

S

MOBILE IS HARDER

• Infrastructure

MOBILE IS HARDER

• Infrastructure

• Compilation/Code Signing

MOBILE IS HARDER

• Infrastructure

• Compilation/Code Signing

• Testing

MOBILE IS HARDER

• Infrastructure

• Compilation/Code Signing

• Testing

• Deployment

SIMULATORS/EMULATORS vs DEVICES

SIMULATORS:

• Used by iOS, included w/ Xcode

• Execute i386 instruction set, not ARM

• Very fast compared to Emulators

• Do not have access to certain hardware functions

• (GPS, Bluetooth Radio, Accelerometers, etc)

SIMULATORS/EMULATORS vs DEVICES

EMULATORS:

• Used by Android, included w/ Android SDKs

• Execute ARM (or native device instruction set)

• Very slow compared to Simulators

• Do not have access to certain hardware functions

• (GPS, Bluetooth Radio, Accelerometers, etc)

SIMULATORS/EMULATORS vs DEVICES

DEVICES:

• Reproduces the actual performance experienced by your users

• The only way to catch manufacturer/OEM bugs

• Very expensive to configure and maintain compared

• Full access to hardware functions:

• (GPS, Bluetooth Radio, Accelerometers, etc)

WHEN TO USE DEVICES vs SIMULATORS/EMULATORS?

• Simulators and Emulators are an excellent solution for running automated tests

during development. They are inexpensive and will reliably catch most problems.

• Physical Devices can be used on a lower frequency (i.e. pre-release, weekly,

daily, etc.). They are the only way to catch performance problems, test certain

hardware features and find OEM issues. In the least, devices should be used

before every release.

UNIT TESTING

According to a recent survey:

-77% of developers said app quality is “very important or mission critical”

-80% of mobile developers test their apps manually

-Only 13% of mobile developers use automated testing

UNIT vs FUNCTIONAL TESTING

• Unit Testing - testing small pieces of code

• Functional Testing - testing button clicks and UI interaction

UNIT/FUNCTIONAL TESTING

BENEFITS:

• Instant Gratification (GREEN is good!)

• Repeatable

• Can automatically test every commit

UNIT/FUNCTIONAL TESTING

CHALLENGES:

• Unit Tests are not users

• LOTs of work to write and maintain them

• Which framework to use?

• What language/framework do your developers know?

• Open Source/Community Support

• 3rd Party Framework requirements

UNIT/FUNCTIONAL TESTING FRAMEWORKS

WHICH ONE TO USE?

CI TOOLS/SERVICES FOR MOBILE

Good: Open Source, Lots of plugins

Bad: Self-Hosted, DIY solution

Jenkins

CI TOOLS/SERVICES FOR MOBILE

Good: Hosted solution, OS X support, Lots of plugins

Bad: Tedious setup process

Travis CI

CI TOOLS/SERVICES FOR MOBILE

Good: Hosted solution, Designed specifically for mobile, Easy setup

Bad: Less flexible than other solutions

Ship.io

CI TOOLS/SERVICES FOR MOBILE

Good: Integrated w/ Xcode, Apple-Supported

Bad: Self-Hosted, iOS Only

Xcode CI

Using the Sauce Labs Cloud with CI

Sauce is the ideal Automated Testing Platform for CI

© Sauce Labs, Inc.

Appium

• “Selenium for mobile apps” - uses the

WebDriver spec to simulate user actions on

iOS and Android

• Leverage your existing Selenium test

architecture to help speed apps to market

• Maximize test code reuse across platforms

makes developers more productive

• Development led by Sauce Labs

Appium is an open

source framework

that automates

testing of native,

hybrid and mobile

web apps.

© Sauce Labs, Inc.

The Sauce Labs Platform

• Point your existing test

scripts to Sauce Labs

• We spin up a new,

pristine VM every time

• Run tests in parallel to

reduce test time

• Capture videos, logs and

test metadata

• Choose from over 500

browsers / OS

combinations

Demo

© Sauce Labs, Inc.

Learn more about Sauce Labs

• Sign up for a free trial of Sauce

Labshttps://saucelabs.com/signup/trial

• Watch a video

https://www.youtube.com/user/saucelabs