Content:
I. History of iOS Automation Frameworks (5 min)
II. Path to Current Approach at Badoo (10 min)
III. Summary (5 min) IV. Movie: Test Run (3 min)
self.introduce =>
{ name: "Viktar Karanevich", title: ["SSTAE", "SSDET", "SQAAE"], languages: ["Java", "Ruby"], experience: "iOS Automation 7+ years" }
The original, largest and leading dating network:
330M+ registered users
800+ automated tests on iOS platform
50M+ downloads on iOS
Jun 2010UI Automation
Sep 2014UI Automationxcrun simctl
Apple
"xcrun simctl" is introduced in xCode 6
It is command line utility to control the Simulator(create, delete, erase, install, uninstall, ...).
Jun 2010UI Automation
Sep 2015UI Automationxcrun simctlUI Testing
Apple
UI Testing is introduced as replacementfor UIAutomation in xCode 7
Sep 2014UI Automationxcrun simctl
Jun 2010UI Automation
Sep 2016UI Testing
xcrun simctl
Apple
Sep 2015UI Automationxcrun simctlUI Testing
Sep 2014UI Automationxcrun simctl
UI Automation removed in xCode 8
Jun 2010UI Automation
Sep 2014xcrun simctl
Sep 2016UI Testing
Sep 2016UI Testing
Apple
Calabash
Nov 2009UI Spec
Calabash
Nov 2009UI Spec
Feb 2011Frank
Jan 2012Calabash
Step DefinitionsRuby Client Library
Features
Developer host
Calabash server
Simulator
Your app
UIQuery
Touch Actions
Calabash
Apr 2013Calabash +
UIAutomationLink: https://github.com/calabash/run_loop
Step DefinitionsRuby Client Library
Features
Developer host
Calabash server
Simulator
Your app
UIQueryRun loop:
UIAutomation
Calabash
Sep 2016Calabash +
DeviceAgentLink: https://github.com/calabash/run_loop/tree/develop/lib/run_loop/device_agent
Developer host Simulator
Step DefinitionsRuby Client Library
XCTest Runner
Features
Calabash server
Your app
UIQuery
Device Agent
Calabash
Nov 2009UI Spec
Feb 2011Frank
Jan 2012Calabash
iOS Automation Frameworks
Apr 2013Calabash +
UIAutomation
Sep 2016Calabash +
DeviceAgent
Jun 2010UI Automation
Sep 2016UI Testing
xcrun simctl
Sep 2015UI Automationxcrun simctlUI Testing
Sep 2014UI Automationxcrun simctl
Apple
3 years 1 year
Calabash
Nov 2009UI Spec
Feb 2011Frank
Jan 2012Calabash
iOS Automation Frameworks
Apr 2013Calabash +
UIAutomation
Sep 2016Calabash +
DeviceAgent
Jun 2010UI Automation
Sep 2016UI Testing
xcrun simctl
Sep 2015UI Automationxcrun simctlUI Testing
Sep 2014UI Automationxcrun simctl
Apple
3 years 1 year
II. Before Parallel RunTest Suite RunNightly Cucumber ReportNightmare!
15htotal time
1simulator
80data
~500tests
II. Before Parallel RunTest Suite RunNightly Cucumber ReportNightmare!
80data
fail tests
1simulator
Key Issue: How to speed up Test Run?
15htotal time
~500tests
1simulator
https://techblog.badoo.com
Tim's talk at Badoo: "Parallel Calabash Testing on iOS"
II. Clustering Hosts
- Parallel run on multi (virtual) hosts
- Mac Pro and hosts: test suite - ~1h 30 min
Developer host Simulator
Selenium DriverWDA Runner Your app
FB UIA WDA
Key Change: Reimplemented UIAutomation dependencyLink: https://github.com/facebook/WebDriverAgent
III. Full Migration to FB UIA WDA
Issues:- all tests have calabash code- custom app backdoors- workarounds and other
III. Full Migration to FB UIA WDA
Full Migration Costs: ~x hours
Is it possible to do Fast Migration?
III. Calabash + FacebookFast migration: Facebash!
Step DefinitionsRuby Client Library
Features
Developer host
Calabash server
Simulator
Your app
UIQueryRun loop:
UIAutomation
III. Calabash + FacebookFast migration: Facebash!
Step DefinitionsRuby Client Library
Features
Developer host
Calabash server
Simulator
Your app
UIQueryRun loop:
UIAutomation
Developer host Simulator
Step DefinitionsRuby Client Library
WDA Runner
Features
Calabash server
Your app
UIQuery
FB UIA WDA
IV. iOS 10 Update:from UIAutomation WDA to XCTest WDA
Link: https://github.com/facebook/WebDriverAgent
Developer host Simulator
Selenium DriverYour app
FB UIA WDA
WDA Runner
IV. iOS 10 Update:from UIAutomation WDA to XCTest WDA
Link: https://github.com/facebook/WebDriverAgent
Developer host Simulator
Selenium DriverYour app
FB UIA WDA
Developer host Simulator
Your app
FB XCTest WDA
WDA Runner
Selenium Driver
FB simctl
IV. iOS Hybrid 2.0iOS 10 update
Developer host Simulator
Step DefinitionsRuby Client Library
WDA Runner
Features
Calabash server
Your app
UIQuery
FB UIA WDA
IV. iOS Hybrid 2.0iOS 10 update
Developer host Simulator
Step DefinitionsRuby Client Library
WDA Runner
Features
Calabash server
Your app
UIQuery
FB UIA WDA
Developer host Simulator
Step DefinitionsRuby Client Library
FB simctl
Features
Calabash server
Your app
UIQuery
FB XCTest WDA
V. Summary: multi-desktops vs multi-simulators
Test Suite: 500 testsNightly Cucumber ReportHappy morning!
80data
Mac PRO
Test Suite Run Time
0 min
225 min
450 min
675 min
900 min
1 sim 10 sims 16 sims
75 min120 min
900 min
MultiDesktops
MultiSimulators
Developer host Simulator
Step DefinitionsRuby Client Library
FB simctl
Features
Calabash server
Your app
UIQuery
FB XCTest WDA
"Face"book + Cala"bash" = Facebash
V. Summary: current approach
- easy to set up Parallel Run on 1 Desktop- keep using of backdoors- full access to internal AUT state- cross iOS app automation- iOS simulator control with FBSimCtl- FB WDA supports Selenium protocol- FB WDA independent of Calabash/Appium
CalabashCalabashFacebookFacebookFacebookFacebook
V. Summary: PROs
- support framework parts: calabash server, FB WDA, fbsimctl, ...- "poor" documentation- using of mixed locator strategies
V. Summary: PROs and CONs
- Parallel Run in Headless Mode- Simulator Start/Shutdown/Reset optimization- Cold/Hot start iOS app with relogin- record video per the each test in Parallel Run
V. TODO