Transcript
Cookpad Tech Kitchen #2~ continuous delivery for iOS ~
Cookpad(domestic)
2
Cookpad(domestic)
3
2016年12月期 第2四半期 https://cf.cpcdn.com/info/assets/wp-content/uploads/20160809153218/2016.2Qir.pdf
Cookpad(world)
4
Cookpad(world)
5
2016年12月期 第2四半期 https://cf.cpcdn.com/info/assets/wp-content/uploads/20160809153218/2016.2Qir.pdf
my.name // Kazuaki Matsuo my.role // Test Engineer my. // @Kazu_cocoa my. // KazuCocoa my. // kazucocoa.wordpress.com
let my = Profile()
6
Agenda✓About our development style ✓ relationship between departments
✓ Dev process / flow ✓Ice break ✓ Test strategy ✓ Test tools / tips ✓Testing strategy and what test we run ✓ UI Testing
8
Our relationship
~ departments ~
9
Relationship(2014)
mobile first department
Mobile app engineersService engineers10
Relationship(2015)
mobile infrastructure
11
Mobile app engineersService engineers
Relationship(2016)
mobile infrastructure
12
Mobile app engineersService engineers
Our relationship
~ roles in iOS development ~
13
Owner
test engineers
designers
directors
iOS engineers
Our roles(2016)
14
Our dev cycleOur dev process
15
previous release
Deadline of Pull Request
Code Freeze submit release
about a month
5 days2 weeks 3 days 4 days
Dev cycle
🍎🍏
16
Dev process
🍎
design/implement/check
🍏
17
submit release
Today’s focus point
18
Today’s focus pointDev Flow ~ PR ~
Pact for iOS
UI Testing
19
Today’s focus point
UI Testing
20
Today’s focus point
Pact for iOS
21
Today’s focus pointDev Flow ~ PR ~
22
What I talked
23
✓ About Cookpad
✓ Our relationship between departments
✓ Our role in iOS development
✓ Our development cycle and process
✓ Who is going to proceed talking each topics
My role and work
~ Focus on technical activities ~
24
~ pyramid ~
25
Our test strategy
Small
Medium
Large
Enormous
manual
Our test strategy
26http://techlife.cookpad.com/entry/2016/08/13/test-size-for-mobile
Small
Medium
Large
Enormous
http://techlife.cookpad.com/entry/2016/08/13/test-size-for-mobile
Our test strategy
27
manual
✓ Automated tests ✓ running tests, judgement the results ✓ GUI Testing ✓ HTTP Requests check ✓ …
✓ Manual tests ✓ new features ✓ difficult to run automatically ✓ exploratory testing
What we checkin test size L/E
28
✓ Automated tests ✓ running tests, judgement the results ✓ GUI Testing ✓ HTTP Requests check ✓ …
✓ Manual tests ✓ new features ✓ difficult to run automatically ✓ exploratory testing
What we checkin test size L/E
29
~ GUI Test / Scenario Test ~ ~ L/E size ~
UI Testing
30
✓ Run tests based on scenarios
✓ We cover over 80% screen transitions
✓ Frequency we run tests
✓ arbitrary timing or before release the app
Test cases
31
✓ Scenario based automated test
✓ Capture HTTP requests
✓ Screen captures
✓ Image diff
Our automated tests
32
✓ Unexpected broken scenario
✓ Unexpected broken network traffic
✓ Unexpected broken layout
✓ Crashes
What we can find out
33
Unexpected brokennetwork traffic
34
Unexpected brokennetwork traffic
35
Example of traffics
36
only for iOS8.2Unexpected broken
layoutsOrder of Dialog Button
is revered
37
only for iOS8.2Unexpected broken
layouts
Here is diff
38
Expected layoutsexcept for iOS8.2
This order is correct
39
✓ Easy to figure out what kind of unexpected defects caused
✓ Regression test
✓ Humans can do other creative activities, other tasks and so on while running the automated tests
Advantage ofautomated tests
40
✓ If we run all tests with manual
✓ Scenario/GUI diff
✓ It takes around a few hours per a device at least
✓ We run several devices per release
✓ We can reduce human resources to do this kind of tasks
Costs
41
iOS 8.0 8.1 8.2 8.3 8.4 9.0 9.1 9.2 9.3 10.0
iPhone ◯ ◯ ◯ ◯ ◯ ◯ ◯ ◯ -
iPad ◯ ◯ ◯ ◯ ◯ ◯ ◯ ◯ -
iPad Pro ◯ ◯ ◯ ◯ -
✓ ◯: select devices we run tests
✓ 6~8 devices per a release
✓ iOS8.0 was removed from Xcode
✓ Do test partially with manual
Combination
42
http://www.slideshare.net/KazuMatsu/20141018-selenium-appiumcookpad
Architecture
Test scenario / wrapper / frameworks
43
✓ We should update scenario every release
✓ BTW, other libraries and wrappers are updated only to solve unstable environment
✓ We can update scenario within a day
✓ Include libraries and wrappers
✓ We can do other tasks except for automated test
Maintenance costs
44
✓ Currently, we try convert Objective-C to Swift
Code changes
45
✓ Keeping 99.9% crash free users
Crashes(Answer)
46
✓ What is our motivation to enhance test automation
✓ 2014/1~: I joined to Cookpad ✓ I’m first test engineer
✓ Test is not only my activity
✓We’d like to ensure minimum quality to release the application
Motivation
47
~ our work ~
UI Testing
48
✓ include libraries and scenarios
✓ Appium 0.18.0 ~>
How long have wedeveloped this tests?
49
✓ We should define concrete plans to tests ✓ What kind of test should we plan? ✓ We should consider software design,
configuration and so on against software tests
✓ I used Mindmap to arrange test targets
How to design tests
50
Maindmap
51
Example of mind maps
52
~ tool stack for iOS ~
UI Testing
53
✓ UI Testing ✓ Appium (JavaScript) ✓ Turnip/RSpec (Ruby) ✓ EarlGrey (Objective-C/Swift) ✓ WebDriverAgent(Objective-C)
✓ image diff ✓ kobold (JavaScript)
✓ Others ✓ wiremock (Jave) ✓ http_proxy (Elixir)
Tool stack for iOS
54
✓ UI Testing ✓ Appium (JavaScript) ✓ Turnip/RSpec (Ruby) ✓ EarlGrey (Objective-C/Swift) ✓ WebDriverAgent(Objective-C)
✓ image diff ✓ kobold (JavaScript)
✓ Others ✓ wiremock (Jave) ✓ http_proxy (Elixir)
Tool stack for iOS
55
Appium
56
✓ Backend differ from:
✓ Appium 1.6.x
✓ WebDriverAgent for xcuitest strategy
✓ UIAutomator for appium strategy
✓ Before Appium 1.5.x
✓ UIAutomator for appium strategy
✓ Appium try to solve problems between iOS <=> Appium Server
✓ Users only know about interface against Appium server
Appium
57
EarlGrey
58
✓ XCUITest supports over iOS9 ✓ Cookpad supports iOS8.0
✓ Lifecycle of XCUITest is same as XCTest ✓ So, we can’t clean environment
every tests. ✓ e.g. Permission dialog ✓ This make tests flaky.
✓ BTW, we should do XCUITest based UI Testing in the future.
How about XCUITest ?
59
Others✓ image diff ✓ kobold (JavaScript)
✓ Others ✓ wiremock (Jave) ✓ http_proxy (Elixir)
60
End
61
top related