Continuous Delivery for databases - microservices, team structures, and Conway's Law - develop:BBC 2014

Continuous Delivery for databases:

microservices, team structures, and Conway's Law

Matthew Skelton, Skelton Thatcher Consulting

BBC:develop conference,

Thursday 13th November 2014, London, UK

#bbcdevelop

Continuous DeliveryDatabases

MicroservicesTeam structuresConway’s Law

Developers & Database Administrators (DBAs)

Siege

http://galleryhip.com/castle-siege-wallpaper.html

http://theattractivearts.com/2013/11/life-after-love/

Shared context and language for Developers and DBAs to facilitate

a move to microservices and Continuous Delivery

BUILDING BLOCKS

Scale (magnification)Distributed transactions

Event streams

OUTCOMES(New software architectures)

New team topologiesNew responsibilities(Continuous Delivery)

Matthew Skelton

•Building & operating commercial software systems since 1998

•Cybernetics + Neuroscience + Music• control engineering• psychology• ‘network’ and group interactions

@matthewpskelton

Skelton ThatcherIntroduction and Services Overview

FORCES / CONTEXT

‘Cloud’ changes the way we mustdesign, deliver, and operate

our software systems

Changes for ‘cloud’

FORCES

Deployability(Elasticity)

(Conway’s Law)

DEPLOYABILITY

Single applicationSingle user journey

Single feature

DATABASE DEPLOYABILITYMinimize changes in ProductionReduce accidental complexity

Archive, distinguish, and split dataName things transparently

Source Business Intelligence from a data warehouseValue more highly the need for change

Avoid Production-only tooling and config

http://bit.ly/DatabaseDeployability

MICROSERVICES

Single applicationSingle user journey

Single feature

DATABASE DEPLOYABILITYEnd the siege mentality

BUILDING BLOCKS

BUILDING BLOCKS

Distributed transactionsScale (magnification)

Event streams

BETWEEN ORGANISATIONS

Asynchronous data reconciliationEventual consistency

BETWEEN MICROSERVICES

Asynchronous data reconciliationEventual consistency

Familiar patterns, different ‘magnification’

EVENT STREAMS

Event sourcingNoSQL, CQRS, Reactive

Pub-Sub messaging

EVENT STREAMSRDBMS database transaction log

Familiar patterns, different ‘magnification’

‘Fractal architecture’

FAMILIAR PATTERNS

Distributed transactionsEvent streams

Asynchronous data reconciliationEventual consistency

CHANGES

CHANGES

Team topologyResponsibilities

Conway’s Law

Mel Conway, 1968“organizations which design systems ... are constrained to produce designs which are copies of the communication structures of these organizations”

http://www.melconway.com/Home/Conways_Law.html

Ruth Malan, 2008“if the architecture of the system and the architecture of the organization are at odds, the architecture of the organization wins”

http://traceinthesand.com/blog/2008/02/13/conways-law/

Topologies and responsibilities

There is no single ‘right’ team topology, but there are several

‘bad’ topologies for any one organisation

Make the team topology & responsibilities match the

required software architecture

RECAPITULATION

Both Devs and DBAs know that:

Distributed transactions do not scale well

Event streams are the foundation of reliable systems

Shared concepts:

Distributed transactionsEvent streams

Asynchronous data reconciliationEventual consistency

Familiar concepts, different scale & context

New team topologies needed

http://theattractivearts.com/2013/11/life-after-love/

We did not really mention:

Microservices technologiesTeam topologies in depth

…

Continuous Delivery for databasesWe need finer-grained, more frequent and isolated deployments

Microservices allow these more fine-grained deployments

We need to support the microservice database model: a larger number of smaller databases with eventual consistency and asynchronous (back office) data reconciliation

This is what we've been doing at an organisational level anyhow (Master Data, inter -bank reconciliation) so we apply the same principles at a Business Service level

This needs team restructuring to be effective (Conway's Law)

What team topologies might work well?

Continuous Delivery for databases

Further details (article, slides):http://bit.ly/CDforDB

Further readingBuild Quality Inbuildqualityin.com

Continuous Delivery & DevOps experience reportsForewords by Dave Farley and Patrick Debois

70% of royalties go to Code Club (@CodeClub)

Contributors include several presenters at BBC:develop 2014:- Amy Phillips (Songkick)- Phil Wills (Guardian)

Discount for #bbcdevelop !https://leanpub.com/buildqualityin/c/BBCdevelop2014

Further readingRuth Malan (@ruthmalan) – Conway’s Law round-up: http://www.ruthmalan.com/Journal/2014/2014JournalMay.htm#Conways_Law

Allan Kelly on Conway’s Law: https://vimeo.com/channels/londoncd/85378217

Common database deployment blockers and Continuous Delivery headaches (Simple Talk) http://bit.ly/DatabaseDeployability

Team Topologies: http://bit.ly/DevOpsTopologies

Matt Hilbert on ‘Next Generation DBA’: http://bit.ly/NextGenDBA

Chris Richardson ‘Microservices: Decomposing Applications for Deployability and Scalability’ http://www.slideshare.net/chris.e.richardson/microservices-decomposing-applications-for-deployability-and-scalability-jax

ThanksBBC for inviting the talk

Audience for listening, asking questions

People at Red Gate Software for joining up some of the dots

Thank you!

http://skeltonthatcher.com/

enquiries@skeltonthatcher.com

@SkeltonThatcher

+44 (0)20 8242 4103