Automated Testing on AWS Device Farm

Keisuke Nishitani, Solutions Architect, Amazon Web

Services Japan K.K.

2015.11.21

Automated Testing on AWS Device Farm

{

"Name" : "","Twitter" : "@Keisuke69","Profile" : {

"Role" : "Solution Architect","Customers": [ "Web Services", "Start-up

],"Services" : [

"AWS Lambda","Amazon API Gateway","All Mobile Services

]}

}

AWS Device Farm Q&A

AWS Device Farm Q&A

48%

32%

2.74 2014 4 80%

Failing to meet mobile app user expectations - A mobile app user survey from Dimensional Research (2015/3/30)http://go.saas.hp.com/apppulse-mobile-survey

Mobile Marketing Costs See Unprecedented Year-over-Year Growth, According to the Fiksu Indexes (2015/5/29)https://www.fiksu.com/about/news-releases/fiksu-indexes-mobile-marketing-costs-see-unprecedented-year-over-year-growth

OS

Open Signal - Android Fragmentation Visualized (2014/8)http://opensignal.com/reports/2014/android-fragmentation/

OS

Android 11,868 2014 8 18,796

Android 38% Jelly Bean, 39 % KitKat, 12% Lolipop, 10% Gingerbread Ice Cream Sandwich

Open Signal - Android Fragmentation Visualized (2014/8)http://opensignal.com/reports/2014/android-fragmentation/

Android Developers (2015/6)http://developer.android.com/intl/zh-CN/about/dashboards/index.html

IT QA 2012 18% 2014 26% 35%

10 4 IT

51%

Capgemini - World Quality Report 2014-15https://www.capgemini.com/thought-leadership/world-quality-report-2014-15

77% of Mobile App Purchasers Have Concerns When Buying Mobile Apps -Harris (2013/8/20)http://www.businesswire.com/news/home/20130820006244/en/77-Mobile-App-Purchasers-Concerns-Buying-Mobile#.Vaz9yaad1N0

OS

AWS Device Farm Q&A

AndroidiOSFire OS

YOUR

APP+

AWS Device FarmAWS AndroidiOSFire OS

OS

AWS

AWS

AppiumJUnit, TestNG Calabash Instrumentation Android

JUnit, Espresso, Robotium uiautomator XCTest(Xcode5), KIF iOS

QA

https://github.com/appium/sample-code AndroidTest.java

https://github.com/appium/sample-code AndroidContactsTest.java

http://calaba.sh/

Fuzz Fuzz UI (seed)(seed)

Fuzz

App Exproler

Android

View

AWS Device Farm

- WiFiBluetoothGPSNFC

CPU

AWS Jenkins Jenkins CI API

Jenkins Jenkins CI AWS Device Farm

AWS Device Farm Jenkins Pluginhttps://github.com/awslabs/aws-device-farm-jenkins-plugin

Jenkins

AWS Device Farm SSL

AWS 30

WiFiBluetooth

4GB 5 WiFi

SMS

Google Play

Google

PhoneGap, Titanium, Xamarin, Unity,

AWS Device Farm Q&A

Sier/

OS

AWS Device FarmOSAWS

AWS Device Farm

Fuzz

IHS Sphera First Report

Android/iOS

QA OS

Android 4.07000

Development/testing process

Develop SCMcommit Automated

buildtrigger Automated

UI test(simulators)

always

Ismilestone

?

no

Beta deployto

manual testers

AutomatedUI test

(Device Farm)

yes

Development/testing process

Develop SCMcommit Automated

buildtrigger Automated

UI test(simulators)

always

Ismilestone

?

no

Beta deployto

manual testers

AutomatedUI test

(Device Farm)

yes

PMease QuickBuild Java AndroidiOS HockeyAppTestFlight Appium UI AWS Device Farm

Development/testing process

Develop SCMcommit Automated

buildtrigger Automated

UI test(simulators)

always

Ismilestone

?

no

Beta deployto

manual testers

AutomatedUI test

(Device Farm)

yes

Appium

Ex: Ruby, Python, Java, JavaScript, PHP, C#

Ex: JUnit for Java, NUnit for C# UI

UIAutomation iOS UIAutomator Android

Development/testing process

Develop SCMcommit Automated

buildtrigger Automated

UI test(simulators)

always

Ismilestone

?

no

Beta deployto

manual testers

AutomatedUI test

(Device Farm)

yes

HockeyApp AndroidiOSWindows iOS

TestFlight iOS

Development/testing process

Develop SCMcommit Automated

buildtrigger Automated

UI test(simulators)

always

Ismilestone

?

no

Beta deployto

manual testers

AutomatedUI test

(Device Farm)

yes

Etsy

Etsy iOS, Android4 2 28QA

WebMobile

CI

1020 build, lint, ,

810

iOS: 50 Mac mini / MacBook Pro Android: 26 Linux containers

/

nightly build Admin

Android Espresso + Genymotion KIF(iOS) simulator/emulator1 Build Flow Plugin

AWS Device Farm

Jenkins plugin

30

EspressoAndroidKIFiOS

Google AnalyticsdevicesOS

AWS Device Farm Device Pool

4 10 30

pass/fail

Jumbotron

AWS Device Farm and Etsy> Device Farm tests results visualization

API

graphite

AWS Device Farm Q&A

- Device Minutes () 250 Device Minutes 1 Device Minute 0.17

- 250

Android

2502=$500

AWS Device Farm Q&A

Android, iOSFire OS

Fuzz

Jenkins CI

AWS Device Farm

AWS Device Farm Q&A

Thanks