Production Ready Services at Netflix

@jonahhorowitz

Production Ready ServicesIncreasing Microservice Availability

at Netflix

@jonahhorowitz

Netflix Architecture

@jonahhorowitz

@jonahhorowitz

Determining Microservice Availability

Service A Service B

?

@jonahhorowitz

Microservice Availability Report

@jonahhorowitz

Microservice Availability Report

@jonahhorowitz

Production Ready

@jonahhorowitz

Characteristics of a Production Ready Service● Reliable

○ Chaos Monkey Enabled○ Automated Deployment Pipelines○ Automated Canary Analysis

@jonahhorowitz

Characteristics of a Production Ready Service● Reliable

○ Chaos Monkey Enabled○ Automated Deployment Pipelines○ Automated Canary Analysis

● Scalable○ Proactive autoscaling○ Reactive autoscaling

@jonahhorowitz

Characteristics of a Production Ready Service● Reliable

○ Chaos Monkey Enabled○ Automated Deployment Pipelines○ Automated Canary Analysis

● Scalable○ Proactive autoscaling○ Reactive autoscaling

● Performant○ Automated Canary Analysis○ Proper Instance Type○ Well-Tuned GC○ Well-Tuned connections, threads

(ezconfig)

@jonahhorowitz

Characteristics of a Production Ready Service● Reliable

○ Chaos Monkey Enabled○ Automated Deployment Pipelines○ Automated Canary Analysis

● Scalable○ Proactive autoscaling○ Reactive autoscaling

● Performant○ Automated Canary Analysis○ Proper Instance Type○ Well-Tuned GC○ Well-Tuned connections, threads

(ezconfig)● Monitored

○ High Quality Alerts■ Upstream Failure %■ Downstream Failure %

○ Dashboards■ Monitoring Releases■ Troubleshooting

@jonahhorowitz

Team Engagements

@jonahhorowitz

Drill SergeantAutomated Checks

● Deployment Pipelines● Operating System● Canary Analysis● Chaos Monkey

Manual Checks

● Alerts● Dashboards● Autoscaling Rules (for now)● Java/Tomcat/Apache Tuning (for

now)

@jonahhorowitz

Jonah HorowitzSenior Site Reliability Engineer

@jonahhorowitz

https://netflix.github.io/https://jobs.netflix.com/

Velocity Ignite Talk tomorrow:7:00pm – 8:30pm Santa Clara Convention Center