Tomer Shiri - A Bug's Life - How to Avoid Production Catastrophes

a bug’s life

How to avoid production catastrophes

Tomer Shiritomer@chegg.com

Remove before flight

Who’s to blame?

How to avoid?

Don’t release anything that has local changes

Avoid Pre release checklists completely

Avoid manual configurations

Use build configurations

avoid manual operations

Demo time

Highway 64

What when wrong?

All integers are MAX_INT

How these values age generated?

Configfile

@dynamic properties

Seems reasonable?

Whats ‘c’?

Whats ‘B’?

Solution?

That’s an edge case, man

Have you ever swizzled private methods?

Removed that annoying UIScrollView from Apple’s views?

Used KVM in order to access / change private ivars?

Don’t rely on internal implementation

The overachiever

High traffic!

Some web services stopped functioning.

Oath API getting hammered by the clients

When an API requests fails, it stops all other requests and tries to obtain a new auth_token.

When two threads tries to do that in the same time, they are failing each other.

Solution?

There is no good solution for that.

Threading bug are hard to hunt.

Use kill switches

Prepare for the worst

avoid manual operations

Everything that can compile - should compile

Don’t rely on internal implementation

Prepare for the worst

Tomer Shiri

tomer@chegg.com

@tomertomershiri

github.com/tomersh