Top Banner
Canary Analyze All the Things Roy Rapoport @royrapoport June 12, 2014 Significant contributions by Chris Sanden, @chris_sanden 1
183

Canary Analyze All the Things

Aug 11, 2014

Download

Data & Analytics

royrapoport

A Canary Analysis presentation for QCon/NY 2014
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: Canary Analyze All the Things

Canary Analyze All the ThingsRoy Rapoport @royrapoport June 12, 2014

Significant contributions by Chris Sanden, @chris_sanden1

Page 2: Canary Analyze All the Things

Oh, the Places We’ll Go!

• Introductions

• Proposed Use Case and Definition

• Continuous Improvement / MVP Model

• Issues, Solutions

• Cloud Considerations

• The Road at Netflix

2

Page 3: Canary Analyze All the Things

A Word About Me …

3

Page 4: Canary Analyze All the Things

A Word About Me …

•About 20 years in technology

3

Page 5: Canary Analyze All the Things

A Word About Me …

•About 20 years in technology•Systems engineering, networking, software development, QA, release management

3

Page 6: Canary Analyze All the Things

A Word About Me …

•About 20 years in technology•Systems engineering, networking, software development, QA, release management

•Time at Netflix: 1809 days

3

Page 7: Canary Analyze All the Things

A Word About Me …

•About 20 years in technology•Systems engineering, networking, software development, QA, release management

•Time at Netflix: 1809 days 4y:11m:14d

3

Page 8: Canary Analyze All the Things

A Word About Me …

•About 20 years in technology•Systems engineering, networking, software development, QA, release management

•Time at Netflix: 1809 days •At Netflix:

4y:11m:14d

3

Page 9: Canary Analyze All the Things

A Word About Me …

•About 20 years in technology•Systems engineering, networking, software development, QA, release management

•Time at Netflix: 1809 days •At Netflix:•Systems Engineering, Service Delivery in IT/Ops

4y:11m:14d

3

Page 10: Canary Analyze All the Things

A Word About Me …

•About 20 years in technology•Systems engineering, networking, software development, QA, release management

•Time at Netflix: 1809 days •At Netflix:•Systems Engineering, Service Delivery in IT/Ops•Troubleshooter and Builder of Python Things[tm] in Product Engineering

4y:11m:14d

3

Page 11: Canary Analyze All the Things

A Word About Me …

•About 20 years in technology•Systems engineering, networking, software development, QA, release management

•Time at Netflix: 1809 days •At Netflix:•Systems Engineering, Service Delivery in IT/Ops•Troubleshooter and Builder of Python Things[tm] in Product Engineering

•Current role: Insight Engineering in Product Engineering

4y:11m:14d

3

Page 12: Canary Analyze All the Things

A Word About Me …

•About 20 years in technology•Systems engineering, networking, software development, QA, release management

•Time at Netflix: 1809 days •At Netflix:•Systems Engineering, Service Delivery in IT/Ops•Troubleshooter and Builder of Python Things[tm] in Product Engineering

•Current role: Insight Engineering in Product Engineering•Real-Time Operational Insight

4y:11m:14d

3

Page 13: Canary Analyze All the Things

A Word About Netflix…

4

Page 14: Canary Analyze All the Things

A Word About Netflix…Just the Stats

4

Page 15: Canary Analyze All the Things

A Word About Netflix…

•16 years

Just the Stats

4

Page 16: Canary Analyze All the Things

A Word About Netflix…

•16 years•2000+ employees

Just the Stats

4

Page 17: Canary Analyze All the Things

A Word About Netflix…

•16 years•2000+ employees•48 million users

Just the Stats

4

Page 18: Canary Analyze All the Things

A Word About Netflix…

•16 years•2000+ employees•48 million users•5x10^9 hours/quarter

Just the Stats

4

Page 19: Canary Analyze All the Things

A Word About Netflix…

5

Page 20: Canary Analyze All the Things

A Word About Netflix…Freedom and Responsibility Culture

5

Page 21: Canary Analyze All the Things

A Word About Netflix…

•Optimize speed of innovation Constrain availability Cost will be what cost will be

Freedom and Responsibility Culture

5

Page 22: Canary Analyze All the Things

A Word About Netflix…

•Optimize speed of innovation Constrain availability Cost will be what cost will be •Hire smart (experienced) people Get out of their way

Freedom and Responsibility Culture

5

Page 23: Canary Analyze All the Things

A Word About Netflix…

•Optimize speed of innovation Constrain availability Cost will be what cost will be •Hire smart (experienced) people Get out of their way•Anti-process bias

Freedom and Responsibility Culture

5

Page 24: Canary Analyze All the Things

A Word About Netflix…

6

Page 25: Canary Analyze All the Things

A Word About Netflix…Technology and Operations

6

Page 26: Canary Analyze All the Things

A Word About Netflix…

•Service Oriented Architecture

Technology and Operations

6

Page 27: Canary Analyze All the Things

A Word About Netflix…

•Service Oriented Architecture•Decentralized Operations. You

Technology and Operations

6

Page 28: Canary Analyze All the Things

A Word About Netflix…

•Service Oriented Architecture•Decentralized Operations. You•Build

Technology and Operations

6

Page 29: Canary Analyze All the Things

A Word About Netflix…

•Service Oriented Architecture•Decentralized Operations. You•Build•Test

Technology and Operations

6

Page 30: Canary Analyze All the Things

A Word About Netflix…

•Service Oriented Architecture•Decentralized Operations. You•Build•Test•Deploy

Technology and Operations

6

Page 31: Canary Analyze All the Things

A Word About Netflix…

•Service Oriented Architecture•Decentralized Operations. You•Build•Test•Deploy•Set up alerting and monitoring

Technology and Operations

6

Page 32: Canary Analyze All the Things

A Word About Netflix…

•Service Oriented Architecture•Decentralized Operations. You•Build•Test•Deploy•Set up alerting and monitoring•Wake up at 2AM

Technology and Operations

6

Page 33: Canary Analyze All the Things

Oh, the Places We’ll Go!

• Introductions

• Proposed Use Case and Definition

• Continuous Improvement / MVP Model

• Issues, Solutions

• Cloud Considerations

• The Road at Netflix

7

Page 34: Canary Analyze All the Things

Why Canary Analysis?

8

Page 35: Canary Analyze All the Things

So You’ve Just Done a Release

9

Page 36: Canary Analyze All the Things

So You’ve Just Done a Release> curl http://WhatDoesTheFooSay.prod.netflix.net/api/v1/cat

9

Page 37: Canary Analyze All the Things

So You’ve Just Done a Release> curl http://WhatDoesTheFooSay.prod.netflix.net/api/v1/cat{“response”: “meow”}

9

Page 38: Canary Analyze All the Things

So You’ve Just Done a Release> curl http://WhatDoesTheFooSay.prod.netflix.net/api/v1/cat{“response”: “meow”}

9

Page 39: Canary Analyze All the Things

So You’ve Just Done a Release

10

Page 40: Canary Analyze All the Things

So You’ve Just Done a Release> curl http://WhatDoesTheFooSay.prod.netflix.net/api/v1/dog

10

Page 41: Canary Analyze All the Things

So You’ve Just Done a Release> curl http://WhatDoesTheFooSay.prod.netflix.net/api/v1/dog{“response”: “woof”}

10

Page 42: Canary Analyze All the Things

So You’ve Just Done a Release> curl http://WhatDoesTheFooSay.prod.netflix.net/api/v1/dog{“response”: “woof”}

10

Page 43: Canary Analyze All the Things

So You’ve Just Done a Release

11

Page 44: Canary Analyze All the Things

So You’ve Just Done a Release> curl http://WhatDoesTheFooSay.prod.netflix.net/api/v1/fox

11

Page 45: Canary Analyze All the Things

So You’ve Just Done a Release> curl http://WhatDoesTheFooSay.prod.netflix.net/api/v1/fox{“response”: “wa-pa-pa-pa-pa-pa-pow”}

11

Page 46: Canary Analyze All the Things

So You’ve Just Done a Release> curl http://WhatDoesTheFooSay.prod.netflix.net/api/v1/fox{“response”: “wa-pa-pa-pa-pa-pa-pow”}

The correct answer to “what does the fox say?” is left an exercise for the reader

11

Page 47: Canary Analyze All the Things

You Need Better Testing!

12

Page 48: Canary Analyze All the Things

You Need Better Testing!

Well, yeah

12

Page 49: Canary Analyze All the Things

You Need Better Testing!

“I’m going to push to production, though I’m pretty sure it’s going to kill the system”

13

- Said no one, ever*

* Hopefully

Page 50: Canary Analyze All the Things

Rate of Change 1 10 100 1000

0

1

2

3

4

5

6

Avai

labi

lity

(nin

es)

Detour Rate of Change vs Availability

14

Page 51: Canary Analyze All the Things

Rate of Change 1 10 100 1000

0

1

2

3

4

5

6

Avai

labi

lity

(nin

es)

Detour Rate of Change vs Availability

14

Page 52: Canary Analyze All the Things

Rate of Change 1 10 100 1000

0

1

2

3

4

5

6

Avai

labi

lity

(nin

es)

Detour Rate of Change vs Availability

14

Page 53: Canary Analyze All the Things

Rate of Change 1 10 100 1000

0

1

2

3

4

5

6

Avai

labi

lity

(nin

es)

Detour Rate of Change vs Availability

14

Page 54: Canary Analyze All the Things

Rate of Change 1 10 100 1000

0

1

2

3

4

5

6

Avai

labi

lity

(nin

es)

Detour Rate of Change vs Availability

14

Page 55: Canary Analyze All the Things

Rate of Change 1 10 100 1000

0

1

2

3

4

5

6

Avai

labi

lity

(nin

es)

Detour Rate of Change vs Availability

Operations Engineering

14

Page 56: Canary Analyze All the Things

You Need Better Testing!Deployments!

Canary Analysis!!

• A deployment process where • a new change (in behavior, code, or both) • is rolled out into production gradually, • with checkpoints along the way to examine the new (canary) systems • (optionally versus the old (baseline) systems) • and make go/no-go decisions.

15

Page 57: Canary Analyze All the Things

Canary Analysis Is Not

16

Page 58: Canary Analyze All the Things

Canary Analysis Is Not

•A replacement for any sort of software testing

16

Page 59: Canary Analyze All the Things

Canary Analysis Is Not

•A replacement for any sort of software testing•A/B Testing

16

Page 60: Canary Analyze All the Things

Canary Analysis Is Not

•A replacement for any sort of software testing•A/B Testing•Releasing 100% to production and hoping for the best

16

Page 61: Canary Analyze All the Things

Version Control System

1000 servers @ 1.0.1

Customers

Build & Deployment

System

Automated Canary Analysis

One Possible Process

17

Page 62: Canary Analyze All the Things

Version Control System

1000 servers @ 1.0.1

Customers

Build & Deployment

System

1 server @ 1.0.2

Automated Canary Analysis

One Possible Process

17

Page 63: Canary Analyze All the Things

Version Control System

1000 servers @ 1.0.1

Customers

Build & Deployment

System

Automated Canary Analysis

10 servers @ 1.0.2

One Possible Process

17

Page 64: Canary Analyze All the Things

Version Control System

1000 servers @ 1.0.1

Customers

Build & Deployment

System

Automated Canary Analysis

1000 servers @ 1.0.2

One Possible Process

17

Page 65: Canary Analyze All the Things

Version Control System

1000 servers @ 1.0.1

Customers

Build & Deployment

System

Automated Canary Analysis

1000 servers @ 1.0.2

One Possible Process

18

Page 66: Canary Analyze All the Things

Version Control System Customers

Build & Deployment

System

Automated Canary Analysis

1000 servers @ 1.0.2

One Possible Process

18

Page 67: Canary Analyze All the Things

Version Control System

1000 servers @ 1.0.1

Customers

Build & Deployment

System

Automated Canary Analysis

1000 servers @ 1.0.2

One Possible Process

19

Page 68: Canary Analyze All the Things

Version Control System

1000 servers @ 1.0.1

Customers

Build & Deployment

System

Automated Canary Analysis

1000 servers @ 1.0.2

One Possible Process

19

Page 69: Canary Analyze All the Things

Oh, the Places We’ll Go!

• Introductions

• Proposed Use Case and Definition

• Continuous Improvement / MVP Model

• Issues, Solutions

• Cloud Considerations

• The Road at Netflix

20

Page 70: Canary Analyze All the Things

Are We There Yet?

21

Page 71: Canary Analyze All the Things

Are We There Yet?

• We’re not

21

Page 72: Canary Analyze All the Things

Are We There Yet?

• We’re not

• You’re probably not either

21

Page 73: Canary Analyze All the Things

Minimally …

22

Page 74: Canary Analyze All the Things

Minimally …

• Observability

22

Page 75: Canary Analyze All the Things

Minimally …

• Observability

• Partial traffic routing

22

Page 76: Canary Analyze All the Things

Minimally …

• Observability

• Partial traffic routing

• Decision-making

22

Page 77: Canary Analyze All the Things

Better Yet …

23

Page 78: Canary Analyze All the Things

Better Yet …

• Focus on the Goal

23

Page 79: Canary Analyze All the Things

Better Yet …

• Focus on the Goal

• Current Baseline Matters

23

Page 80: Canary Analyze All the Things

Better Yet …

• Focus on the Goal

• Current Baseline Matters

26% fewer errors in canary

23

Page 81: Canary Analyze All the Things

Better Yet …

• Focus on the Goal

• Current Baseline Matters

• Observability segregation

26% fewer errors in canary

23

Page 82: Canary Analyze All the Things

Hold On a Minute!

26% fewer errors in canary

24

Page 83: Canary Analyze All the Things

Hold On a Minute!

26% fewer errors in canary

Mission Accomplished

24

Page 84: Canary Analyze All the Things

Hold On a Minute!

26% fewer errors in canary

Mission Accomplished

30% fewer requests handled in canary

25

Page 85: Canary Analyze All the Things

Hold On a Minute!

26

Page 86: Canary Analyze All the Things

Hold On a Minute!

26

Page 87: Canary Analyze All the Things

Hold On a Minute!

27

Page 88: Canary Analyze All the Things

Hold On a Minute!

• Absolute numbers are relatively unimportant

27

Page 89: Canary Analyze All the Things

Hold On a Minute!

• Absolute numbers are relatively unimportant

• Relative numbers matter

27

Page 90: Canary Analyze All the Things

Hold On a Minute!

• Absolute numbers are relatively unimportant

• Relative numbers matter• Error rate

27

Page 91: Canary Analyze All the Things

Hold On a Minute!

• Absolute numbers are relatively unimportant

• Relative numbers matter• Error rate• RPS per CPU cycle

27

Page 92: Canary Analyze All the Things

Hold On a Minute!

• Absolute numbers are relatively unimportant

• Relative numbers matter• Error rate• RPS per CPU cycle

27

Page 93: Canary Analyze All the Things

Requests Rate Comparison

So You’ve Got Your Graphs requests

28

Page 94: Canary Analyze All the Things

Requests Rate Comparison

So You’ve Got Your Graphs requests

28

Page 95: Canary Analyze All the Things

Requests Rate Comparison

So You’ve Got Your Graphs requests

Type RAM Cores CostBaseline m3.medium 3.75GB 3 $.11/hrCanary m1.small 1.7GB 1 $.06/hr

28

Page 96: Canary Analyze All the Things

So You’ve Got Your Graphs

29

Page 97: Canary Analyze All the Things

Automating …

30

Page 98: Canary Analyze All the Things

Automating …

• Decision

30

Page 99: Canary Analyze All the Things

Automating …

• Decision

• Execution

30

Page 100: Canary Analyze All the Things

A Quick Recap

31

Page 101: Canary Analyze All the Things

A Quick Recap

• Observe

31

Page 102: Canary Analyze All the Things

A Quick Recap

• Observe

• Segregate metrics

31

Page 103: Canary Analyze All the Things

A Quick Recap

• Observe

• Segregate metrics

• Partial deploy

31

Page 104: Canary Analyze All the Things

A Quick Recap

• Observe

• Segregate metrics

• Partial deploy

• Compare to Baseline

31

Page 105: Canary Analyze All the Things

A Quick Recap

• Observe

• Segregate metrics

• Partial deploy

• Compare to Baseline

• Absolutes are never right

31

Page 106: Canary Analyze All the Things

A Quick Recap

• Observe

• Segregate metrics

• Partial deploy

• Compare to Baseline

• Absolutes are never right

• Automate decision

31

Page 107: Canary Analyze All the Things

A Quick Recap

• Observe

• Segregate metrics

• Partial deploy

• Compare to Baseline

• Absolutes are never right

• Automate decision

• Automate execution

31

Page 108: Canary Analyze All the Things

Oh, the Places We’ll Go!

• Introductions

• Proposed Use Case and Definition

• Continuous Improvement / MVP Model

• Issues, Solutions

• Cloud Considerations

• The Road at Netflix

32

Page 109: Canary Analyze All the Things

To Save You Some Time …

Not all metrics are created equal

33

Page 110: Canary Analyze All the Things

To Save You Some Time …

Not all metrics are created equal

Focus on System and Application Metrics

33

Page 111: Canary Analyze All the Things

To Save You Some Time …

Not all metrics are created equal

Focus on System and Application Metrics

Weight by category (system, latency, etc)

33

Page 112: Canary Analyze All the Things

To Save You Some Time …

Outliers are out, lying

34

Page 113: Canary Analyze All the Things

To Save You Some Time …

Outliers are out, lying

Use a group of servers

34

Page 114: Canary Analyze All the Things

To Save You Some Time …

Outliers are out, lying

Use a group of servers

Balance fidelity with customer impact

34

Page 115: Canary Analyze All the Things

To Save You Some Time …

Exercise without warmup can result in injury

35

Page 116: Canary Analyze All the Things

To Save You Some Time …

Exercise without warmup can result in injury

Repeat canary analysis frequently

35

Page 117: Canary Analyze All the Things

To Save You Some Time …

Exercise without warmup can result in injury

Repeat canary analysis frequently

Both traffic and startup time are factors

35

Page 118: Canary Analyze All the Things

To Save You Some Time …

vive la différence!

36

Page 119: Canary Analyze All the Things

To Save You Some Time …

vive la différence!

Hot-OK, Cold-OK

36

Page 120: Canary Analyze All the Things

To Save You Some Time …

vive la différence!

Hot-OK, Cold-OK

Let Application Owners Choose

36

Page 121: Canary Analyze All the Things

To Save You Some Time …

Signal is better than no1$#[NO CARRIER]

37

Page 122: Canary Analyze All the Things

To Save You Some Time …

Signal is better than no1$#[NO CARRIER]

Ignore weak signals

37

Page 123: Canary Analyze All the Things

Oh, the Places We’ll Go!

• Introductions

• Proposed Use Case and Definition

• Continuous Improvement / MVP Model

• Issues, Solutions

• Cloud Considerations

• The Road at Netflix

38

Page 124: Canary Analyze All the Things

Good News

39

Page 125: Canary Analyze All the Things

Good News

39

Page 126: Canary Analyze All the Things

Good News

• Software-Defined Everything

39

Page 127: Canary Analyze All the Things

Good News

• Software-Defined Everything

• Incremental Pricing

39

Page 128: Canary Analyze All the Things

Bad News

40

Page 129: Canary Analyze All the Things

Bad News

40

Page 130: Canary Analyze All the Things

Bad News

• Capacity Management

40

Page 131: Canary Analyze All the Things

Bad News

• Capacity Management

• Unpredictable Inconsistency

40

Page 132: Canary Analyze All the Things

Oh, the Places We’ll Go!

• Introductions

• Proposed Use Case and Definition

• Continuous Improvement / MVP Model

• Issues, Solutions

• Cloud Considerations

• The Road at Netflix

41

Page 133: Canary Analyze All the Things

Numbers

42

Page 134: Canary Analyze All the Things

Numbers

• 752 services in production

42

Page 135: Canary Analyze All the Things

Numbers

• 752 services in production

• In-house telemetry platform

42

Page 136: Canary Analyze All the Things

Numbers

• 752 services in production

• In-house telemetry platform

• A few metrics

42

Page 137: Canary Analyze All the Things

Numbers

• 752 services in production

• In-house telemetry platform

• A few metrics

42

Page 138: Canary Analyze All the Things

43

Page 139: Canary Analyze All the Things

Been there.Done that.Manually. Artisanally.

43

Page 140: Canary Analyze All the Things

Been there.

• Started in the Data Center

Done that.Manually. Artisanally.

43

Page 141: Canary Analyze All the Things

Been there.

• Started in the Data Center

• Manual, dashboard-driven

Done that.Manually. Artisanally.

43

Page 142: Canary Analyze All the Things

Been there.Done that.Manually.

44

CPU

Requests

Errors

Page 143: Canary Analyze All the Things

Been there.Done that.Manually.

45

Page 144: Canary Analyze All the Things

Been there.Done that.Manually.

45

Page 145: Canary Analyze All the Things

Been there.Done that.Manually.

46

Page 146: Canary Analyze All the Things

Been there.Done that.Manually.

46

Page 147: Canary Analyze All the Things

Been there.Done that.Manually.

47

Page 148: Canary Analyze All the Things

Been there.Done that.Manually.

47

Page 149: Canary Analyze All the Things

Been there.Done that.Manually.

48

Page 150: Canary Analyze All the Things

Been there.Done that.Manually.• Context vs Precision

48

Page 151: Canary Analyze All the Things

Been there.Done that.Manually.• Context vs Precision

• No …

48

Page 152: Canary Analyze All the Things

Been there.Done that.Manually.• Context vs Precision

• No …

• Repeatability

48

Page 153: Canary Analyze All the Things

Been there.Done that.Manually.• Context vs Precision

• No …

• Repeatability

• Trending

48

Page 154: Canary Analyze All the Things

Been there.Done that.Manually.• Context vs Precision

• No …

• Repeatability

• Trending

• Manual effort is manual

48

Page 155: Canary Analyze All the Things

So Now What?

49

Page 156: Canary Analyze All the Things

So Now What?

• Automate Analysis

49

Page 157: Canary Analyze All the Things

So Now What?

• Automate Analysis

• Took Some Effort

49

Page 158: Canary Analyze All the Things

So Now What?

• Automate Analysis

• Took Some Effort

• Approach and analytics

49

Page 159: Canary Analyze All the Things

So Now What?

• Automate Analysis

• Took Some Effort

• Approach and analytics

• Presentation matters

49

Page 160: Canary Analyze All the Things

Automated Canary Analysis

50

Page 161: Canary Analyze All the Things

Automated Canary Analysis

51

Page 162: Canary Analyze All the Things

Automated Canary Analysis

51

Page 163: Canary Analyze All the Things

Automated Canary Analysis

52

Page 164: Canary Analyze All the Things

Automated Canary Analysis

53

Page 165: Canary Analyze All the Things

Automated Canary Analysis

54

Page 166: Canary Analyze All the Things

For Our Next Trick …

55

Page 167: Canary Analyze All the Things

For Our Next Trick …

• Configuration GUI

55

Page 168: Canary Analyze All the Things

For Our Next Trick …

• Configuration GUI• Deployment System Integration

55

Page 169: Canary Analyze All the Things

For Our Next Trick …

• Configuration GUI• Deployment System Integration• ACA All The Things

55

Page 170: Canary Analyze All the Things

For Our Next Trick …

• Configuration GUI• Deployment System Integration• ACA All The Things

• OpenConnect firmware updates

55

Page 171: Canary Analyze All the Things

For Our Next Trick …

• Configuration GUI• Deployment System Integration• ACA All The Things

• OpenConnect firmware updates• Client software changes

55

Page 172: Canary Analyze All the Things

For Our Next Trick …

• Configuration GUI• Deployment System Integration• ACA All The Things

• OpenConnect firmware updates• Client software changes• Configuration changes in production

55

Page 173: Canary Analyze All the Things

Summary

56

Page 174: Canary Analyze All the Things

Summary

• Canary Analysis makes your changes

56

Page 175: Canary Analyze All the Things

Summary

• Canary Analysis makes your changes• Safer

56

Page 176: Canary Analyze All the Things

Summary

• Canary Analysis makes your changes• Safer• Faster

56

Page 177: Canary Analyze All the Things

Summary

• Canary Analysis makes your changes• Safer• Faster• Easier

56

Page 178: Canary Analyze All the Things

Summary

• Canary Analysis makes your changes• Safer• Faster• Easier

• Most people can start doing it

56

Page 179: Canary Analyze All the Things

Summary

• Canary Analysis makes your changes• Safer• Faster• Easier

• Most people can start doing it• Everyone can do it better

56

Page 180: Canary Analyze All the Things

Summary

• Canary Analysis makes your changes• Safer• Faster• Easier

• Most people can start doing it• Everyone can do it better

56

Page 181: Canary Analyze All the Things

• https://www.flickr.com/photos/cseeman

• https://www.flickr.com/photos/ransomtech

• https://www.flickr.com/photos/dougbrown47

• https://www.flickr.com/photos/andresthor/

• https://www.flickr.com/photos/dougbrown47

• https://www.flickr.com/photos/pkdesigns

Questions, Attributions, Feedback

57

Page 182: Canary Analyze All the Things

• https://www.flickr.com/photos/cseeman

• https://www.flickr.com/photos/ransomtech

• https://www.flickr.com/photos/dougbrown47

• https://www.flickr.com/photos/andresthor/

• https://www.flickr.com/photos/dougbrown47

• https://www.flickr.com/photos/pkdesigns

Questions, Attributions, Feedback

@royrapoport

57

Page 183: Canary Analyze All the Things

• https://www.flickr.com/photos/cseeman

• https://www.flickr.com/photos/ransomtech

• https://www.flickr.com/photos/dougbrown47

• https://www.flickr.com/photos/andresthor/

• https://www.flickr.com/photos/dougbrown47

• https://www.flickr.com/photos/pkdesigns

Questions, Attributions, Feedback

@royrapoport ?57