Continuous Delivery for databases: microservices, team structures, and Conway's Law Matthew Skelton, Skelton Thatcher Consulting BBC:develop conference, Thursday 13 th November 2014, London, UK #bbcdevelop
Jul 02, 2015
Continuous Delivery for databases:
microservices, team structures, and Conway's Law
Matthew Skelton, Skelton Thatcher Consulting
BBC:develop conference,
Thursday 13th November 2014, London, UK
#bbcdevelop
Continuous DeliveryDatabases
MicroservicesTeam structuresConway’s Law
Developers & Database Administrators (DBAs)
Siege
http://galleryhip.com/castle-siege-wallpaper.html
http://theattractivearts.com/2013/11/life-after-love/
Shared context and language for Developers and DBAs to facilitate
a move to microservices and Continuous Delivery
BUILDING BLOCKS
Scale (magnification)Distributed transactions
Event streams
OUTCOMES(New software architectures)
New team topologiesNew responsibilities(Continuous Delivery)
Matthew Skelton
•Building & operating commercial software systems since 1998
•Cybernetics + Neuroscience + Music• control engineering• psychology• ‘network’ and group interactions
@matthewpskelton
Skelton ThatcherIntroduction and Services Overview
FORCES / CONTEXT
‘Cloud’ changes the way we mustdesign, deliver, and operate
our software systems
Changes for ‘cloud’
FORCES
Deployability(Elasticity)
(Conway’s Law)
DEPLOYABILITY
Single applicationSingle user journey
Single feature
DATABASE DEPLOYABILITYMinimize changes in ProductionReduce accidental complexity
Archive, distinguish, and split dataName things transparently
Source Business Intelligence from a data warehouseValue more highly the need for change
Avoid Production-only tooling and config
http://bit.ly/DatabaseDeployability
MICROSERVICES
Single applicationSingle user journey
Single feature
DATABASE DEPLOYABILITYEnd the siege mentality
BUILDING BLOCKS
BUILDING BLOCKS
Distributed transactionsScale (magnification)
Event streams
BETWEEN ORGANISATIONS
Asynchronous data reconciliationEventual consistency
BETWEEN MICROSERVICES
Asynchronous data reconciliationEventual consistency
Familiar patterns, different ‘magnification’
EVENT STREAMS
Event sourcingNoSQL, CQRS, Reactive
Pub-Sub messaging
EVENT STREAMSRDBMS database transaction log
Familiar patterns, different ‘magnification’
‘Fractal architecture’
FAMILIAR PATTERNS
Distributed transactionsEvent streams
Asynchronous data reconciliationEventual consistency
CHANGES
CHANGES
Team topologyResponsibilities
Conway’s Law
Mel Conway, 1968“organizations which design systems ... are constrained to produce designs which are copies of the communication structures of these organizations”
http://www.melconway.com/Home/Conways_Law.html
Ruth Malan, 2008“if the architecture of the system and the architecture of the organization are at odds, the architecture of the organization wins”
http://traceinthesand.com/blog/2008/02/13/conways-law/
Topologies and responsibilities
There is no single ‘right’ team topology, but there are several
‘bad’ topologies for any one organisation
Make the team topology & responsibilities match the
required software architecture
RECAPITULATION
Both Devs and DBAs know that:
Distributed transactions do not scale well
Event streams are the foundation of reliable systems
Shared concepts:
Distributed transactionsEvent streams
Asynchronous data reconciliationEventual consistency
Familiar concepts, different scale & context
New team topologies needed
http://theattractivearts.com/2013/11/life-after-love/
We did not really mention:
Microservices technologiesTeam topologies in depth
…
Continuous Delivery for databasesWe need finer-grained, more frequent and isolated deployments
Microservices allow these more fine-grained deployments
We need to support the microservice database model: a larger number of smaller databases with eventual consistency and asynchronous (back office) data reconciliation
This is what we've been doing at an organisational level anyhow (Master Data, inter -bank reconciliation) so we apply the same principles at a Business Service level
This needs team restructuring to be effective (Conway's Law)
What team topologies might work well?
Continuous Delivery for databases
Further details (article, slides):http://bit.ly/CDforDB
Further readingBuild Quality Inbuildqualityin.com
Continuous Delivery & DevOps experience reportsForewords by Dave Farley and Patrick Debois
70% of royalties go to Code Club (@CodeClub)
Contributors include several presenters at BBC:develop 2014:- Amy Phillips (Songkick)- Phil Wills (Guardian)
Discount for #bbcdevelop !https://leanpub.com/buildqualityin/c/BBCdevelop2014
Further readingRuth Malan (@ruthmalan) – Conway’s Law round-up: http://www.ruthmalan.com/Journal/2014/2014JournalMay.htm#Conways_Law
Allan Kelly on Conway’s Law: https://vimeo.com/channels/londoncd/85378217
Common database deployment blockers and Continuous Delivery headaches (Simple Talk) http://bit.ly/DatabaseDeployability
Team Topologies: http://bit.ly/DevOpsTopologies
Matt Hilbert on ‘Next Generation DBA’: http://bit.ly/NextGenDBA
Chris Richardson ‘Microservices: Decomposing Applications for Deployability and Scalability’ http://www.slideshare.net/chris.e.richardson/microservices-decomposing-applications-for-deployability-and-scalability-jax
ThanksBBC for inviting the talk
Audience for listening, asking questions
People at Red Gate Software for joining up some of the dots