© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. SUMMIT The Software Development Process at Amazon Jonathan Weiss Amazon Web Services
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMM I T
The Software Development Process at Amazon
Jonathan WeissAmazon Web Services
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMM I T
Amazon is hundreds of different businesses
Over 80 announcements at re:Invent 2018 alone
No signs of slowing down
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMM I T
The Software Development Lifecycle
developers customers
releasetestbuild
plan monitor
delivery pipeline
feedback loop
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMM I T
Ten Lessons Learned
1. Microservices
2. DevOps Teams
3. Self-Service Tools
4. Continuous Delivery
5. PessimisticDeployments
6. Automated Testing
7. Optimize ECT Loop
8. Monitor Everything
9. Measure Everything
10. Listen to Customers
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMM I T
Ten Lessons Learned
1. Microservices
2. DevOps Teams
3. Self-Service Tools
4. Continuous Delivery
5. PessimisticDeployments
6. Automated Testing
7. Optimize ECT Loop
8. Monitor Everything
9. Measure Everything
10. Listen to Customers
SUMM I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMM I T© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMM I T
Monolith development lifecycle
developers
releasetestbuild
delivery pipelineapp
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMM I T
Service-Oriented Architecture (SOA)
Single-purpose Primitive
Connected w/ APIs
Highly decoupled
“Microservices”
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMM I T
Ten Lessons Learned
1. Microservices
2. DevOps Teams
3. Self-Service Tools
4. Continuous Delivery
5. PessimisticDeployments
6. Automated Testing
7. Optimize ECT Loop
8. Monitor Everything
9. Measure Everything
10. Listen to Customers
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMM I T
Two-pizza teams
Decentralized
Agility
Autonomy
Accountability
Ownership
“DevOps”
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMM I T
Missing Tools
developers delivery pipelineservices
???
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMM I T
Ten Lessons Learned
1. Microservices
2. DevOps Teams
3. Self-Service Tools
4. Continuous Delivery
5. PessimisticDeployments
6. Automated Testing
7. Optimize ECT Loop
8. Monitor Everything
9. Measure Everything
10. Listen to Customers
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMM I T
Two-pizza teams
Decentralized
Agility
Autonomy
Accountability
Ownership
“DevOps”
Self-service
Technology-agnostic
Encourage best practices
Single-purpose services
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMM I T
Deployment service
No downtime deployments
Health tracking
Versioned artifacts and rollbacks
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMM I T
Ten Lessons Learned
1. Microservices
2. DevOps Teams
3. Self-Service Tools
4. Continuous Delivery
5. PessimisticDeployments
6. Automated Testing
7. Optimize ECT Loop
8. Monitor Everything
9. Measure Everything
10. Listen to Customers
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMM I T
Pipelines
Continuous delivery
Automated release process
Faster & more reliable releases
Adopted by virtually all teams
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMM I T
Microsservice development lifecycle
developers delivery pipelinesservices
releasetestbuild
releasetestbuild
releasetestbuild
releasetestbuild
releasetestbuild
releasetestbuild
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMM I T
Thousands of teamsx Microservice architecture
x Continuous deliveryx Multiple environments
= millions of deployments
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMM I T
Ten Lessons Learned
1. Microservices
2. DevOps Teams
3. Self-Service Tools
4. Continuous Delivery
5. PessimisticDeployments
6. Automated Testing
7. Optimize ECT Loop
8. Monitor Everything
9. Measure Everything
10. Listen to Customers
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMM I T
Release Pipelines
ProductionStagingBuildSource
Run Build
and Unit
Tests
Deploy to
Staging
Environment
Deploy to
Production
Environment
Pull Source
Code
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMM I T
Release Pipelines
ProductionStagingBuildSource
Run Build
and Unit
Tests
Deploy to
Staging
Environment
Deploy
to One
Box
Pull Source
Code
Deploy
to 1st
AZ
Deploy
to 1st
AZ
Deploy
to One
Box
Deploy
to 2nd
AZ
Deploy
to 2nd
AZ
Deploy
to 3rd
AZ
Deploy
to 3rd
AZ
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMM I T
Ten Lessons Learned
1. Microservices
2. DevOps Teams
3. Self-Service Tools
4. Continuous Delivery
5. PessimisticDeployments
6. Automated Testing
7. Optimize ECT Loop
8. Monitor Everything
9. Measure Everything
10. Listen to Customers
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMM I T
Release Pipelines
ProductionStagingBuildSource
Run Build
and Unit
Tests
Deploy to
Staging
Environment
Deploy
to One
Box
Pull Source
Code
Deploy
to 3rd
AZ
Deploy
to 1st
AZ
Deploy
to 2nd
AZ
Deploy
to One
Box
Deploy
to 3rd
AZ
Deploy
to 1st
AZ
Deploy
to 2nd
AZ
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMM I T
Release Pipelines
ProductionStagingBuildSource
Run Build
and Unit
Tests
Deploy to
Staging
Environment
Deploy
to One
Box
Pull Source
Code
Deploy
to 3rd
AZ
Deploy
to 1st
AZ
Deploy
to 2nd
AZ
Deploy
to One
Box
Deploy
to 3rd
AZ
Deploy
to 1st
AZ
Deploy
to 2nd
AZ
Functional
Tests
Load / Perf
Tests
Other
Tests
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMM I T
Release Pipelines
ProductiongammabetaBuildSource
Run Build
and Unit
Tests
Deploy to
Staging
Environment
Deploy
to One
Box
Pull Source
Code
Deploy
to 3rd
AZ
Deploy
to 1st
AZ
Deploy
to 2nd
AZ
Deploy
to One
Box
Deploy
to 3rd
AZ
Deploy
to 1st
AZ
Deploy
to 2nd
AZ
Integration
Tests
Load / Perf
Tests
Other
Tests
Deploy to
Staging
Environment
Functional
Tests
Other
Tests
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMM I T
Ten Lessons Learned
1. Microservices
2. DevOps Teams
3. Self-Service Tools
4. Continuous Delivery
5. PessimisticDeployments
6. Automated Testing
7. Optimize ECT Loop
8. Monitor Everything
9. Measure Everything
10. Listen to Customers
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMM I T
The Edit-Compile-Test Loop
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMM I T
The Edit-Compile-Test Loop
ECT Loop
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMM I T
The Edit-Compile-Test Loop
ECT Loop
Code Review
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMM I T
The Edit-Compile-Test Loop
ECT Loop
Code Review
Staging
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMM I T
The Edit-Compile-Test Loop
ECT Loop
Code Review
Staging
Production
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMM I T
Catching Problems in the Development Lifecycle
ProductionStagingCode
ReviewECT
Ideal Place Great Place Good Place Last Resort
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMM I T
Ten Lessons Learned
1. Microservices
2. DevOps Teams
3. Self-Service Tools
4. Continuous Delivery
5. PessimisticDeployments
6. Automated Testing
7. Optimize ECT Loop
8. Monitor Everything
9. Measure Everything
10. Listen to Customers
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMM I T
The Software Development Lifecycle
developers customers
releasetestbuild
plan monitor
delivery pipeline
feedback loop
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMM I T
Ops Culture
Ops Meetings
Ops Dashboards
• resource monitoring
• application monitoring
• canary monitoring
• client-side monitoring
• “order drop rate”
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMM I T
Ten Lessons Learned
1. Microservices
2. DevOps Teams
3. Self-Service Tools
4. Continuous Delivery
5. PessimisticDeployments
6. Automated Testing
7. Optimize ECT Loop
8. Monitor Everything
9. Measure Everything
10. Listen to Customers
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMM I T
Data Culture
WBR Meetings
Metrics Decks
• feature adoption
• segment growth
• funnel analysis
• cost structure
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMM I T
Ten Lessons Learned
1. Microservices
2. DevOps Teams
3. Self-Service Tools
4. Continuous Delivery
5. PessimisticDeployments
6. Automated Testing
7. Optimize ECT Loop
8. Monitor Everything
9. Measure Everything
10. Listen to Customers
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMM I T© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Listen
It's easy to be right if you just listen to customers.
1. Ask them wantthey want
2. Get feedback on what you build
3. Course correct if need to
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMM I T
Feature vs. Need
Wenger 16999 Swiss Army Knife Gianthttps://www.amazon.com/Wenger-16999-Swiss-Knife-Giant/dp/B001DZTJRQ
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMM I T
Ten Lessons Learned
“Working Backwards” description
by Werner Vogels, Amazon CTO
www.allthingsdistributed.com/2006/11/working_backwards.htm
l
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMM I T
Ten Lessons Learned
1. Microservices
2. DevOps Teams
3. Self-Service Tools
4. Continuous Delivery
5. PessimisticDeployments
6. Automated Testing
7. Optimize ECT Loop
8. Monitor Everything
9. Measure Everything
10. Listen to Customers
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMM I T
Thank you!
SUMM I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Jonathan Weiss@jweiss