Microservice Scars:Lessons we learnt
Abhaya Chauhan@AbhayaChauhan
www.abhayachauhan.comJoshua Toth@TothJoshuaJ
Question Policy
Let’s go on a Journey
Context
PageUp
Context
Talent Management
SaaS
Context
100% Cloud based
AWS + Azure
Context
19 years
Context
25% growth year on year
Context
PageUp = Market Leaders
Context
The Monolith
Context
Fragility
The Monolith
Context
Agility
The Monolith
Context
Complexity
The Monolith
Context
Innovation
The Monolith
Context
Deployment
The Monolith
Context
Production Hardened
The Monolith
Context
#Scale
Context
Large Projects
Context
Big Rewrite
Context
If you can’t build a monolith, what makes you think
microservices are the answer?
Simon Brown
Context
First rule of Distributed Computing
Context
Don’t do it.
Context
Why Microservices?
Context
Reduce time to market
Context
Reduce blast radius
Context
EnableEvolutionary Architecture
Context
Sprint Zero
Decisions
Sprint Zero
The stack
Sprint Zero
Sprint Zero
Sprint Zero
Frontend
Sprint Zero
Backend
Frontend
Sprint Zero
Backend
DB
Frontend
Sprint Zero
Elastic Beanstalk
S3
.Net
Sprint Zero
Self hosted
Sprint Zero
Angular
Sprint Zero
DynamoDB
Sprint Zero
Pact
Sprint Zero
Infancy
Sprint Zero
Limitations
Sprint Zero
Investment
Sprint Zero
Deployment
Sprint Zero
Continuous
Sprint Zero
Phabricator
Sprint Zero
Phabro slides had to be removed.
Please contact us for any queries
Sprint Zero
Pipeline
Sprint Zero
Sprint Zero
Pipeline
Sprint Zero
Pipeline
Sprint Zero
F# Make (Fake)
Sprint Zero
Monolith Scripts
Sprint Zero
Sharing != caring
Sprint Zero
Clear separation
Sprint Zero
Development Practices
Sprint Zero
TDD
Sprint Zero
BDD
Sprint Zero
Sprint Zero
Elastic Beanstalk
Full stack
Elastic Beanstalk
Visualization
Elastic Beanstalk
Monitoring
Elastic Beanstalk
Monitoring
Elastic Beanstalk
Monitoring
Elastic Beanstalk
Monitoring
Elastic Beanstalk
Elastic Beanstalk
Elastic Beanstalk
Elastic Beanstalk
Rebuilds
Elastic Beanstalk
No Downtime
Elastic Beanstalk
The Hack
Elastic Beanstalk
Elastic Beanstalk
Elastic Beanstalk
Elastic Beanstalk
Speed
Elastic Beanstalk
Care
Elastic Beanstalk
Synchronization
Elastic Beanstalk
50 shades of “fuck we can’t roll this back”
Grey
Elastic Beanstalk
Scaling
Elastic Beanstalk
Authentication
IdentityServer3(Locksmith)
OAuth2
Immediate return
OAuth2
Questions
Messaging
Messaging
Messaging
Messaging
Request Response Pain
Messaging
Resource intensive
Messaging
Fault Prone
Messaging
1 to 1 communication
Messaging
Event Driven Architecture
Messaging
Webhooks
Messaging
Messaging
Messaging
:`(
Messaging
Queuing (SQS)
Messaging
Resource Intensive
Fault Prone
1 - many Communication
Messaging
Resource Intensive
Fault Prone
1 - many Communication
Messaging
Resource Intensive
Fault Prone
1 - many Communication
Messaging
Resource Intensive
Fault Prone
1 - many Communication
Messaging
Simple Notification Service (SNS)
Messaging
Push messages
Messaging
Mobile Devices
Messaging
Mobile DevicesSMS
Messaging
Mobile DevicesSMS
Emails
Messaging
Mobile DevicesSMS
EmailsHTTP Endpoints
Messaging
Mobile DevicesSMS
EmailsHTTP Endpoints
Lambda
Messaging
Mobile DevicesSMS
EmailsHTTP Endpoints
LambdaQueues
Messaging
Messaging
Messaging
Messaging
Messaging
SNS + SQS = Amazing Service Bus
Resource Intensive
Fault Prone
1 - many Communication
Messaging
Circuit Breaker
Messaging
Messaging
CloudWatch
Messaging
Historical Events
Messaging
A store for events
Messaging
DynamoDb
Fully Managed NoSQL DB
DynamoDb
Fast and Predictable
DynamoDb
Distributed
DynamoDb
Key-ValueDocument
DynamoDb
Hash (Range)
Partition 1 Partition 2 Partition 3 Partition N
Read & Write capacity equally distributed
Max 400kb
DynamoDb
1 Read unit = 4KB/s1 Write unit = 1KB/s
DynamoDb
CQRS
DynamoDb
DynamoDb
Oh shit!
DynamoDb
Write is 20x more expensive
DynamoDb
DynamoDb
{"Key":"Mauritania","Value":"42803"},{"Key":"Mauritius","Value":"42804"},{"Key":"Mexico","Value":"42805"},{"Key":"Micronesia","Value":"42806"},{"Key":"Monaco","Value":"42807"},{"Key":"Mongolia","Value":"42808"},{"Key":"Montenegro","Value":"42809"},{"Key":"Montserrat","Value":"42810"},{"Key":"Morocco","Value":"42811"},{"Key":"Mozambique","Value":"42812"},{"Key":"Namibia","Value":"42813"},{"Key":"Nauru","Value":"42814"},{"Key":"Nepal","Value":"42815"},{"Key":"Netherlands","Value":"42816"},{"Key":"Netherlands Antilles","Value":"42817"},{"Key":"New Caledonia","Value":"42818"},{"Key":"New Zealand","Value":"42819"},{"Key":"Nicaragua","Value":"42820"},{"Key":"Niger","Value":"42821"},{"Key":"Nigeria","Value":"42822"},{"Key":"Niue","Value":"42823"},{"Key":"Northern Mariana Islands","Value":"42824"},{"Key":"Norway","Value":"42825"},{"Key":"Oman","Value":"42826"},{"Key":"Pakistan","Value":"42827"},{"Key":"Palau, Republic Of","Value":"42828"},{"Key":"Palestine","Value":"42829"},{"Key":"Panama","Value":"42830"},{"Key":"Papua New Guinea","Value":"42831"},{"Key":"Paraguay","Value":"42832"},{"Key":"Peru","Value":"42833"},{"Key":"Philippines","Value":"42834"},{"Key":"Poland","Value":"42835"},{"Key":"Portugal","Value":"42836"},{"Key":"Puerto Rico","Value":"42837"},{"Key":"Qatar","Value":"42838"},{"Key":"Republic Of Korea","Value":"42839"},{"Key":"Republic Of Moldova","Value":"42840"},{"Key":"Republic Of Yemen","Value":"42841"},{"Key":"Reunion","Value":"42842"},{"Key":"Romania","Value":"42843"},{"Key":"Russian Federation","Value":"42844"},{"Key":"Rwanda","Value":"42845"},{"Key":"Saint Kitts And Nevis","Value":"42846"},{"Key":"Saint Lucia","Value":"42847"},{"Key":"Sao Tome & Principe","Value":"42848"},{"Key":"Saudi Arabia","Value":"42849"},{"Key":"Senegal","Value":"42850"},{"Key":"Serbia","Value":"42851"},{"Key":"Serbia & Monten","Value":"42852"},{"Key":"Seychelles","Value":"42853"},{"Key":"Sierra Leone","Value":"42854"},{"Key":"Singapore","Value":"42855"},{"Key":"Slovakia","Value":"42856"},{"Key":"Slovenia","Value":"42857"},{"Key":"Solomon Islands","Value":"42858"},{"Key":"Somalia","Value":"42859"},{"Key":"South Africa","Value":"42860"},{"Key":"South Sudan","Value":"42861"},{"Key":"Spain","Value":"42862"},{"Key":"Sri Lanka","Value":"42863"},{"Key":"St. Vincent And The Grena","Value":"42864"},{"Key":"Sudan","Value":"42865"},{"Key":"Surinam","Value":"42866"},{"Key":"Swaziland","Value":"42867"},{"Key":"Sweden","Value":"42868"},{"Key":"Switzerland","Value":"42869"},{"Key":"Syrian Arab Republic","Value":"42870"},{"Key":"Tajikistan","Value":"42871"},{"Key":"Tanzania, United Rep. Of","Value":"42872"},{"Key":"Tfyr Of Macedonia","Value":"42873"},{"Key":"Thailand","Value":"42874"},{"Key":"Timor-Leste","Value":"42875"},{"Key":"Togo","Value":"42876"},{"Key":"Tokelau Islands","Value":"42877"},{"Key":"Tonga","Value":"42878"},{"Key":"Trinidad And Tobago","Value":"42879"},{"Key":"Tunisia","Value":"42880"},{"Key":"Turkey","Value":"42881"},{"Key":"Turkmenistan","Value":"42882"},{"Key":"Turks And Caicos Island","Value":"42883"},{"Key":"Mauritania","Value":"42803"},{"Key":"Mauritius","Value":"42804"},{"Key":"Mexico","Value":"42805"},{"Key":"Micronesia","Value":"42806"},{"Key":"Monaco","Value":"42807"},{"Key":"Mongolia","Value":"42808"},{"Key":"Montenegro","Value":"42809"},{"Key":"Montserrat","Value":"42810"},{"Key":"Morocco","Value":"42811"},{"Key":"Mozambique","Value":"42812"},{"Key":"Namibia","Value":"42813"},{"Key":"Nauru","Value":"42814"},{"Key":"Nepal","Value":"42815"},{"Key":"Netherlands","Value":"42816"},{"Key":"Netherlands Antilles","Value":"42817"},{"Key":"New Caledonia","Value":"42818"},{"Key":"New Zealand","Value":"42819"},{"Key":"Nicaragua","Value":"42820"},{"Key":"Niger","Value":"42821"},{"Key":"Nigeria","Value":"42822"},{"Key":"Niue","Value":"42823"},{"Key":"Northern Mariana Islands","Value":"42824"},{"Key":"Norway","Value":"42825"},{"Key":"Oman","Value":"42826"},{"Key":"Pakistan","Value":"42827"},{"Key":"Palau, Republic Of","Value":"42828"},{"Key":"Palestine","Value":"42829"},{"Key":"Panama","Value":"42830"},{"Key":"Papua New Guinea","Value":"42831"},{"Key":"Paraguay","Value":"42832"},{"Key":"Peru","Value":"42833"},{"Key":"Philippines","Value":"42834"},{"Key":"Poland","Value":"42835"},{"Key":"Portugal","Value":"42836"},{"Key":"Puerto Rico","Value":"42837"},{"Key":"Qatar","Value":"42838"},{"Key":"Republic Of Korea","Value":"42839"},{"Key":"Republic Of Moldova","Value":"42840"},{"Key":"Republic Of Yemen","Value":"42841"},{"Key":"Reunion","Value":"42842"},{"Key":"Romania","Value":"42843"},{"Key":"Russian Federation","Value":"42844"},{"Key":"Rwanda","Value":"42845"},{"Key":"Saint Kitts And Nevis","Value":"42846"},{"Key":"Saint Lucia","Value":"42847"},{"Key":"Sao Tome & Principe","Value":"42848"},{"Key":"Saudi Arabia","Value":"42849"},{"Key":"Senegal","Value":"42850"},{"Key":"Serbia","Value":"42851"},{"Key":"Serbia & Monten","Value":"42852"},{"Key":"Seychelles","Value":"42853"},{"Key":"Sierra Leone","Value":"42854"},{"Key":"Singapore","Value":"42855"},{"Key":"Slovakia","Value":"42856"},{"Key":"Slovenia","Value":"42857"}
DynamoDb
{"Key":"Mauritania","Value":"42803"},{"Key":"Mauritius","Value":"42804"},{"Key":"Mexico","Value":"42805"},{"Key":"Micronesia","Value":"42806"},{"Key":"Monaco","Value":"42807"},{"Key":"Mongolia","Value":"42808"},{"Key":"Montenegro","Value":"42809"},{"Key":"Montserrat","Value":"42810"},{"Key":"Morocco","Value":"42811"},{"Key":"Mozambique","Value":"42812"},{"Key":"Namibia","Value":"42813"},{"Key":"Nauru","Value":"42814"},{"Key":"Nepal","Value":"42815"},{"Key":"Netherlands","Value":"42816"},{"Key":"Netherlands Antilles","Value":"42817"},{"Key":"New Caledonia","Value":"42818"},{"Key":"New Zealand","Value":"42819"},{"Key":"Nicaragua","Value":"42820"},{"Key":"Niger","Value":"42821"},{"Key":"Nigeria","Value":"42822"},{"Key":"Niue","Value":"42823"},{"Key":"Northern Mariana Islands","Value":"42824"},{"Key":"Norway","Value":"42825"},{"Key":"Oman","Value":"42826"},{"Key":"Pakistan","Value":"42827"},{"Key":"Palau, Republic Of","Value":"42828"},{"Key":"Palestine","Value":"42829"},{"Key":"Panama","Value":"42830"},{"Key":"Papua New Guinea","Value":"42831"},{"Key":"Paraguay","Value":"42832"},{"Key":"Peru","Value":"42833"},{"Key":"Philippines","Value":"42834"},{"Key":"Poland","Value":"42835"},{"Key":"Portugal","Value":"42836"},{"Key":"Puerto Rico","Value":"42837"},{"Key":"Qatar","Value":"42838"},{"Key":"Republic Of Korea","Value":"42839"},{"Key":"Republic Of Moldova","Value":"42840"},{"Key":"Republic Of Yemen","Value":"42841"},{"Key":"Reunion","Value":"42842"},{"Key":"Romania","Value":"42843"},{"Key":"Russian Federation","Value":"42844"},{"Key":"Rwanda","Value":"42845"},{"Key":"Saint Kitts And Nevis","Value":"42846"},{"Key":"Saint Lucia","Value":"42847"},{"Key":"Sao Tome & Principe","Value":"42848"},{"Key":"Saudi Arabia","Value":"42849"},{"Key":"Senegal","Value":"42850"},{"Key":"Serbia","Value":"42851"},{"Key":"Serbia & Monten","Value":"42852"},{"Key":"Seychelles","Value":"42853"},{"Key":"Sierra Leone","Value":"42854"},{"Key":"Singapore","Value":"42855"},{"Key":"Slovakia","Value":"42856"},{"Key":"Slovenia","Value":"42857"},{"Key":"Solomon Islands","Value":"42858"},{"Key":"Somalia","Value":"42859"},{"Key":"South Africa","Value":"42860"},{"Key":"South
{"Key":"Papua New Guinea","Value":"42831"},{"Key":"Paraguay","Value":"42832"},{"Key":"Peru","Value":"42833"},{"Key":"Philippines","Value":"42834"},{"Key":"Poland","Value":"42835"},{"Key":"Portugal","Value":"42836"},{"Key":"Puerto Rico","Value":"42837"},{"Key":"Qatar","Value":"42838"},{"Key":"Republic Of Korea","Value":"42839"},{"Key":"Republic Of Moldova","Value":"42840"}
Static (Read Optimised) Volatile (Write Optimised)
Oh shit #2
DynamoDb
DynamoDb
Recovery Point Objective = 30
minutes
DynamoDb
Recovery Time Objective = 1 day
DynamoDb
Distributed Triggers
DynamoDb
DynamoDb
DynamoDb
DynamoDb
DynamoDb
S3 Versioned Buckets
DynamoDb
DynamoDb
DynamoDb
DynamoDb
DynamoDb Version Control
DynamoDb
Risk Profile
Low
High
Team
Skillset
Frontend
First Microservice
Quality
NoSql
Infrastructure
Security / DR
Estimation
Risk Profile at startup
PM 101 -Risk Management
Risk Profile: Team
1. Team availability2. Experience level of the Team3. Domain knowledge of the Team4. Technical skillset of the Team5. Team flow, size and track record
Risk Profile: Product
1. Product Complexity2. Product Criticality3. Product Stability4. Problem Clarity5. User perspection / expectations
Risk vs Returns
Risk vs Business Value
Retrospective
Monolith Gateway
Stakeholders
More than you think
Stakeholders
Metrics
Stakeholders
Transparency
Stakeholders
Support Balance
Business value
Team goals
Bake some goods
Time to iterate
Don’t underestimate
users
Pair pair pair
Questions
Thank you
https://github.com/PageUpPeopleOrg
https://en.wikipedia.org/wiki/Test-driven_developmenthttps://en.wikipedia.org/wiki/Behavior-driven_developmenthttps://angularjs.org/http://fsharp.github.io/FAKE/http://owin.org/https://aws.amazon.com/documentation/https://twitter.com/simonbrownhttp://phabricator.org/https://github.com/realestate-com-au/pacthttps://github.com/IdentityServer/IdentityServer3http://martinfowler.com/bliki/CQRS.html