Aug 12, 2015
About Myself• Rajdeep Varma
• 5 years in Quality Engineering. I work at ThoughtWorks
• Passionate about test automation and agile practices
• Tested a number of web apps and recently started testing mobile apps
I will talk about
• Our Problem:
A. Long running automation
B. Running Automation on Multiple devices
• Our Solution: parallel_calabash
Typical CI Pipeline Of Mobile Project
Git PushCompile
+ Unit Test
Build App UI Tests Distribute
“Fast Feedback”
Stats
• 5 git repo
• Avg. 30 commits a day
• Approx. 100 Automated UI tests.
• “GitPush” TO “TestedOK” > 1 Hrs
With slow tests
Git Push
UI Tests
Deploy to next stage
Umm…slow tests—Chai Break—damn! test failed :(
Fixed!
<pain>its so slow<pain/>— Pray —
LOL
:(
Patience Is Not Always A Virtue
Choose One
a) Lots of automated tests for good coverage
b) Faster automation run
c) BOTH!
How?
http://sharpreflections.com/wp-content/uploads/2012/06/multi_core_cpu.png
Our tool-set
• MobileWeb: Cucumber + waitr-webdriver + parallel_tests
• AndroidApp: Calabash
Options
• Multiple parallel builds in CI
• Proprietary cloud based tools
• Create a tool that matches our need
http://images.clipartpanda.com/choice-clipart-fork-choice-hi.png
parallel_calabash
Demo
Single Device Parallel
Time
CPU
Usa
ge
How it works? <feature_grouper>
1_digit_addition.feature
2_digit_addition.feature
3_digit_addition.feature
4_digit_addition.feature
5_digit_addition.feature
6_digit_addition.feature
7_digit_addition.feature
8_digit_addition.feature
Detect Devices
Feature Grouper
1_digit_addition 2_digit_addition
3_digit_addition 4_digit_addition
5_digit_addition 6_digit_addition
7_digit_addition 8_digit_addition
How it works? <process spawner>
1_digit_addition 2_digit_addition
3_digit_addition 4_digit_addition
5_digit_addition 6_digit_addition
7_digit_addition 8_digit_addition
ENV[ADB_DEVICE_ARG] = emulator-5554ENV[TEST_PROCESS_NUMBER] = 1
ProcessSpawner
ENV[ADB_DEVICE_ARG] = emulator-5556ENV[TEST_PROCESS_NUMBER] = 2
ENV[ADB_DEVICE_ARG] = emulator-5558ENV[TEST_PROCESS_NUMBER] = 3
ENV[ADB_DEVICE_ARG] = emulator-5560ENV[TEST_PROCESS_NUMBER] = 4
SUMMARIZE
RESULT
stdout
Reportingconfig/cucumber.yml
default: --format html --out my_report_<%=ENV['DEVICE_INFO']%>.html
Reports from 3 devices:
my_report_GT_N7100.html
my_report_Samsung_Galaxy_Note_2___4_3___API_18___720x1280_1.html
my_report_Samsung_Galaxy_Note_2___4_3___API_18___720x1280_2.html
Optimization in test distribution
• Issue with distribution based on cucumber “featue” files
• Distribution based on “Scenario” instead of features
• --group-by-scenarios
Advantages of fast tests
• Fast Feedback
• Faster development cycle for BDD environment
• Modern machines have multiple cores and parallelization can make Maximum use of Hardware
• Faster tests makes developers & Testers happy
I will talk about
• Our Problem:
A. Long running automation
B. Running Automation on Multiple devices
• Our Solution: parallel_calabash
Device Fragmentation
parallel_calabash -a <path/to/apk> features --concurrent
what did we achieve?
• 70% increase in testing velocity.
• Scalability
• Reporting
• Process specific action using ENV[TEST_PROCESS_NUMBER]
Choose One
a) Lots of automated tests for good coverage
b) Faster automation run
c) BOTH!
TIME
https://github.com/rajdeepv/parallel_calabash https://rubygems.org/gems/parallel_calabash
parallel_calabash
References:
• https://github.com/grosser/parallel_tests
• http://martinfowler.com/
• http://continuousdelivery.com/
• http://watirmelon.com/
?