20160913 cookpad ios_en

Cookpad Tech Kitchen #2~ continuous delivery for iOS ~

Cookpad(domestic)

2

Cookpad(domestic)

3

2016年12月期 第2四半期 https://cf.cpcdn.com/info/assets/wp-content/uploads/20160809153218/2016.2Qir.pdf

Cookpad(world)

4

Cookpad(world)

5

2016年12月期 第2四半期 https://cf.cpcdn.com/info/assets/wp-content/uploads/20160809153218/2016.2Qir.pdf

my.name // Kazuaki Matsuo my.role // Test Engineer my. // @Kazu_cocoa my. // KazuCocoa my. // kazucocoa.wordpress.com

let my = Profile()

6

Agenda✓About our development style ✓ relationship between departments

✓ Dev process / flow ✓Ice break ✓ Test strategy ✓ Test tools / tips ✓Testing strategy and what test we run ✓ UI Testing

8

Our relationship

~ departments ~

9

Relationship(2014)

mobile first department

Mobile app engineersService engineers10

Relationship(2015)

mobile infrastructure

11

Mobile app engineersService engineers

Relationship(2016)

mobile infrastructure

12

Mobile app engineersService engineers

Our relationship

~ roles in iOS development ~

13

Owner

test engineers

designers

directors

iOS engineers

Our roles(2016)

14

Our dev cycleOur dev process

15

previous release

Deadline of Pull Request

Code Freeze submit release

about a month

5 days2 weeks 3 days 4 days

Dev cycle

🍎🍏

16

Dev process

🍎

design/implement/check

🍏

17

submit release

Today’s focus point

18

Today’s focus pointDev Flow ~ PR ~

Pact for iOS

UI Testing

19

Today’s focus point

UI Testing

20

Today’s focus point

Pact for iOS

21

Today’s focus pointDev Flow ~ PR ~

22

What I talked

23

✓ About Cookpad

✓ Our relationship between departments

✓ Our role in iOS development

✓ Our development cycle and process

✓ Who is going to proceed talking each topics

My role and work

~ Focus on technical activities ~

24

~ pyramid ~

25

Our test strategy

Small

Medium

Large

Enormous

manual

Our test strategy

26http://techlife.cookpad.com/entry/2016/08/13/test-size-for-mobile

Small

Medium

Large

Enormous

http://techlife.cookpad.com/entry/2016/08/13/test-size-for-mobile

Our test strategy

27

manual

✓ Automated tests ✓ running tests, judgement the results ✓ GUI Testing ✓ HTTP Requests check ✓ …

✓ Manual tests ✓ new features ✓ difficult to run automatically ✓ exploratory testing

What we checkin test size L/E

28

✓ Automated tests ✓ running tests, judgement the results ✓ GUI Testing ✓ HTTP Requests check ✓ …

✓ Manual tests ✓ new features ✓ difficult to run automatically ✓ exploratory testing

What we checkin test size L/E

29

~ GUI Test / Scenario Test ~ ~ L/E size ~

UI Testing

30

✓ Run tests based on scenarios

✓ We cover over 80% screen transitions

✓ Frequency we run tests

✓ arbitrary timing or before release the app

Test cases

31

✓ Scenario based automated test

✓ Capture HTTP requests

✓ Screen captures

✓ Image diff

Our automated tests

32

✓ Unexpected broken scenario

✓ Unexpected broken network traffic

✓ Unexpected broken layout

✓ Crashes

What we can find out

33

Unexpected brokennetwork traffic

34

Unexpected brokennetwork traffic

35

Example of traffics

36

only for iOS8.2Unexpected broken

layoutsOrder of Dialog Button

is revered

37

only for iOS8.2Unexpected broken

layouts

Here is diff

38

Expected layoutsexcept for iOS8.2

This order is correct

39

✓ Easy to figure out what kind of unexpected defects caused

✓ Regression test

✓ Humans can do other creative activities, other tasks and so on while running the automated tests

Advantage ofautomated tests

40

✓ If we run all tests with manual

✓ Scenario/GUI diff

✓ It takes around a few hours per a device at least

✓ We run several devices per release

✓ We can reduce human resources to do this kind of tasks

Costs

41

iOS 8.0 8.1 8.2 8.3 8.4 9.0 9.1 9.2 9.3 10.0

iPhone ◯ ◯ ◯ ◯ ◯ ◯ ◯ ◯ -

iPad ◯ ◯ ◯ ◯ ◯ ◯ ◯ ◯ -

iPad Pro ◯ ◯ ◯ ◯ -

✓ ◯: select devices we run tests

✓ 6~8 devices per a release

✓ iOS8.0 was removed from Xcode

✓ Do test partially with manual

Combination

42

http://www.slideshare.net/KazuMatsu/20141018-selenium-appiumcookpad

Architecture

Test scenario / wrapper / frameworks

43

✓ We should update scenario every release

✓ BTW, other libraries and wrappers are updated only to solve unstable environment

✓ We can update scenario within a day

✓ Include libraries and wrappers

✓ We can do other tasks except for automated test

Maintenance costs

44

✓ Currently, we try convert Objective-C to Swift

Code changes

45

✓ Keeping 99.9% crash free users

Crashes(Answer)

46

✓ What is our motivation to enhance test automation

✓ 2014/1~: I joined to Cookpad ✓ I’m first test engineer

✓ Test is not only my activity

✓We’d like to ensure minimum quality to release the application

Motivation

47

~ our work ~

UI Testing

48

✓ include libraries and scenarios

✓ Appium 0.18.0 ~>

How long have wedeveloped this tests?

49

✓ We should define concrete plans to tests ✓ What kind of test should we plan? ✓ We should consider software design,

configuration and so on against software tests

✓ I used Mindmap to arrange test targets

How to design tests

50

Maindmap

51

Example of mind maps

52

~ tool stack for iOS ~

UI Testing

53

✓ UI Testing ✓ Appium (JavaScript) ✓ Turnip/RSpec (Ruby) ✓ EarlGrey (Objective-C/Swift) ✓ WebDriverAgent(Objective-C)

✓ image diff ✓ kobold (JavaScript)

✓ Others ✓ wiremock (Jave) ✓ http_proxy (Elixir)

Tool stack for iOS

54

✓ UI Testing ✓ Appium (JavaScript) ✓ Turnip/RSpec (Ruby) ✓ EarlGrey (Objective-C/Swift) ✓ WebDriverAgent(Objective-C)

✓ image diff ✓ kobold (JavaScript)

✓ Others ✓ wiremock (Jave) ✓ http_proxy (Elixir)

Tool stack for iOS

55

Appium

56

✓ Backend differ from:

✓ Appium 1.6.x

✓ WebDriverAgent for xcuitest strategy

✓ UIAutomator for appium strategy

✓ Before Appium 1.5.x

✓ UIAutomator for appium strategy

✓ Appium try to solve problems between iOS <=> Appium Server

✓ Users only know about interface against Appium server

Appium

57

EarlGrey

58

✓ XCUITest supports over iOS9 ✓ Cookpad supports iOS8.0

✓ Lifecycle of XCUITest is same as XCTest ✓ So, we can’t clean environment

every tests. ✓ e.g. Permission dialog ✓ This make tests flaky.

✓ BTW, we should do XCUITest based UI Testing in the future.

How about XCUITest ?

59

Others✓ image diff ✓ kobold (JavaScript)

✓ Others ✓ wiremock (Jave) ✓ http_proxy (Elixir)

60

End

61