Conway's law revisited - Architectures for an effective IT

Conway’s law revisited Architectures for an effective IT

Uwe Friedrichsen, codecentric AG, 2012-2015

@ufried Uwe Friedrichsen | [email protected] | http://slideshare.net/ufried | http://ufried.tumblr.com

Why should we change?

Formal part of value creation Solution: machine

Dynamic part of value creation

Solution: man

sluggishness/low dynamic high dynamic high dynamic

The historical course of market dynamics and the recent rise of highly dynamic and complex markets

The dominance of high dynamics and complexity is neither good nor bad. It‘s a historical fact.

t 1970/80 today

Age of crafts manu- facturing

Age of tayloristic industry

Age of global markets

1850/1900

Spacious markets, little competition

Local markets, high customi-zation

Outperformers exercise market pressure over conventional companies

We call the graph shown here the “Taylor Bathtub”. The “bathtub” curve

Source: BetaCodex Network Associates, “Organize for complexity”, BetaCodex Network White Paper 12 & 13

Economic Darwinism

Economic Darwinism Everyone is affected by Economic Darwinism •  All sectors •  Growing globalization on all levels •  Internet business •  More competitors per customer •  Higher customer expectations •  Lower customer loyalty à In the long run only those will survive who meet the customer needs and demands best

Nice, but how does this relate to IT?

IT is the nervous system IT is vital •  All companies •  IT is not just supporter or „cost center “ … •  … but it is the central nervous system •  Even short IT outages considered critical •  No business change without IT •  No new products without IT à IT limits the maximum possible adaption rate of a company

IT is a key success factor for belonging to the survivors of the economic darwinism

What business needs from IT …

Responsiveness Reliability

Throughput Flexibility

How IT serves business …

What’s wrong with IT?

1960 1970 1980 1990 2000 2010 2020

Complicated

(Business functions)

Complex

(Business processes)

Highly complex

(Business nervous system)

Software crisis

Software engineering

PC

LAN

Internet Business Support

of IT

Selective

Holistic

Complicated

Complex “Moore’s law”

Mobile IoT

1960 1970 1980 1990 2000 2010 2020

Complicated

(Business functions)

Complex

(business processes)

Highly complex

(Business nervous system)

Software crisis

Software engineering

PC

LAN

Internet Business Support

of IT

Selective

Holistic

Complicated

Complex “Moore’s law”

Mobile IoT

We are here …

1960 1970 1980 1990 2000 2010 2020

Complicated

(Business functions)

Complex

(business processes)

Highly complex

(Business nervous system)

Software crisis

Software engineering

PC

LAN

Internet Business Support

of IT

Selective

Holistic

Complicated

Complex “Moore’s law”

Mobile IoT

… but we still base most of our decisions on that

We are here …

Formal part of value creation Solution: machine

Dynamic part of value creation

Solution: man

sluggishness/low dynamic high dynamic high dynamic

The historical course of market dynamics and the recent rise of highly dynamic and complex markets

The dominance of high dynamics and complexity is neither good nor bad. It‘s a historical fact.

t 1970/80 today

Age of crafts manu- facturing

Age of tayloristic industry

Age of global markets

1850/1900

Spacious markets, little competition

Local markets, high customi-zation

Outperformers exercise market pressure over conventional companies

We call the graph shown here the “Taylor Bathtub”. Remember the bathtub curve?

This adds an additional twist …

1960 1970 1980 1990 2000 2010 2020

Complicated

(Business functions)

Complex

(business processes)

Highly complex

(Business nervous system)

Software crisis

Software engineering

PC

LAN

Internet Business Support

of IT

Selective

Holistic

Complicated

Complex “Moore’s law”

Mobile IoT

… but we still base most of our decisions on that

We are here …

Business is very different today …

… than it was back then

We need to rethink IT!

Economic Darwinism

Business-related Change Drivers

IT

Technology-related Change Drivers

But there is more …

Lean Enterprise

Productshaping/optimization

Innovation

Measure & analyze Accelerating OODA loop

Quick customer feedback cycles

Economic Darwinism

Business-related Change Drivers

Lean Enterprise

IT

Technology-related Change Drivers

IT as a Product (“Digitization”)

Virtualization of products

IT-centric business models

Disruptive new business models

Economic Darwinism

Business-related Change Drivers

Lean Enterprise

IT as a Product

IT

Technology-related Change Drivers

Pay-per-Use

Business Case

Self-Service

Cloud

Elasticity

UnreliableCOTS Hardware

Provisioning Speed

Economic Darwinism

Business-related Change Drivers

Lean Enterprise

IT as a Product

Cloud

IT

Technology-related Change Drivers

Zero Downtime

Peer Multiplication

Mobile & IoT

Deep Process Integration

UnreliableCommunication Unpredictable

Load Patterns

Economic Darwinism

Business-related Change Drivers

Lean Enterprise

IT as a Product

Cloud

IoT

Mobile

IT

Technology-related Change Drivers

… and more

Big Data Analysis

Amplifiers

Social

Economic Darwinism

Business-related Change Drivers

Lean Enterprise

IT as a Product

Cloud

IoT

Mobile

IT Big Data Analytics Social

Technology-related Change Drivers

We really need to rethink IT!

Process People

Organization Governance

Agile Lean

DevOps Flow

Decentralization Autonomy

End2End Responsibility

Craftsmanship

Mastery Curiosity

Beyond Budgeting (BetaCodex)

Systemic Optimization

It would be fascinating to discuss them all and to see how they fit into each other …

… but right now we want to focus on architecture

So, what does that mean for architecture?

Architectural drivers •  Need for quick change and extension •  Replace over reuse •  Need for quick releases

•  Unpredictable load patterns •  Distributed, highly interconnected systems •  Extreme high service availability

•  Diverse front-ends and devices

•  Cost efficiency

Architectural requirements •  Easy to understand •  Easy to extend •  Easy to change •  Easy to replace •  Easy to deploy

•  Easy to scale •  Easy to recover

•  Easy to connect

•  Easy to afford

Architectural requirements •  Easy to understand à Understandability •  Easy to extend à Extensibility •  Easy to change à Changeability •  Easy to replace à Replaceability •  Easy to deploy à Deployability

•  Easy to scale à Scalability •  Easy to recover à Resilience

•  Easy to connect à Uniform interface

•  Easy to afford à Cost-efficiency (for development & operations)

Hey, what about Conway’s law?

Conway’s law: Organizations which design systems [...] are constrained to produce designs which are copies of the communication structures of these organizations

Conway’s law reversed: You won’t be able to successfully establish an efficient organization structure that is not supported by your system design (architecture)

Monolith

Example: Multiple teams working on a monolith usually end up in tightly coupled teams with excessive communication overhead

But what kind of organizational structure does our architecture need to support?

Process People

Organization Governance

Agile Lean

DevOps Flow

Decentralization Autonomy

End2End Responsibility

Craftsmanship

Mastery Curiosity

Beyond Budgeting (BetaCodex)

Systemic Optimization

Our architecture needs to support decentralized, autonomous* teams

* Autonomy in this context means to have the teams working as autonomous as possible, yet making sure they share* the same overall vision. This means a certain amount of communication between the teams is always needed but it* should be made sure that it happens as efficient as possible.

Architectural requirements

•  Easy to separate à Autonomy

•  Easy to understand à Understandability •  Easy to extend à Extensibility •  Easy to change à Changeability •  Easy to replace à Replaceability •  Easy to deploy à Deployability

•  Easy to scale à Scalability •  Easy to recover à Resilience

•  Easy to connect à Uniform interface

•  Easy to afford à Cost-efficiency (for development & operations)

What are the appropriate solutions?

Let’s check a few trending topics …

µServices •  Built for replacement (not reuse) •  Self-dependent, loosely coupled services •  Should be aligned with business capability •  Size should not exceed what one brain can grasp

µServices

Cost-

effic

ienc

y

Unifo

rm In

terfa

ce

Resil

ienc

e

Scal

abilit

y

Dep

loya

bilit

y

Repl

acea

bilit

y

Chan

geab

ility

Exte

nsib

ility

Unde

rsta

ndab

ility

Auto

nom

y

REST

•  Uniform access interface to resources •  Closely related to the HTTP protocol •  HATEOAS (Hypermedia as the engine of application state)

REST

Cost-

effic

ienc

y

Unifo

rm In

terfa

ce

Resil

ienc

e

Scal

abilit

y

Dep

loya

bilit

y

Repl

acea

bilit

y

Chan

geab

ility

Exte

nsib

ility

Unde

rsta

ndab

ility

Auto

nom

y

Event/Message-driven •  Asynchronous communication paradigm •  Technical decoupling of communication peers (isolation) •  Location transparency in conjunction with MOM •  Call-stack paradigm replaced by (complex) message networks

Event/Message-driven

Cost-

effic

ienc

y

Unifo

rm In

terfa

ce

Resil

ienc

e

Scal

abilit

y

Dep

loya

bilit

y

Repl

acea

bilit

y

Chan

geab

ility

Exte

nsib

ility

Unde

rsta

ndab

ility

Auto

nom

y

CQRS •  Command Query Responsibility Segregation •  Separate read and write interfaces including underlying models •  Separation can be extended up to the data store(s) •  Allows for optimized data representations and access logic

READWRITE

CQRS

Cost-

effic

ienc

y

Unifo

rm In

terfa

ce

Resil

ienc

e

Scal

abilit

y

Dep

loya

bilit

y

Repl

acea

bilit

y

Chan

geab

ility

Exte

nsib

ility

Unde

rsta

ndab

ility

READWRITE

Auto

nom

y

Reactive •  Message-driven – asynchronous and non-blocking •  Scalable – scaling out and embracing the network •  Resilient – isolation, loose coupling and hierarchical structure •  Responsive – latency control and graceful degradation of service

Reactive

Cost-

effic

ienc

y

Unifo

rm In

terfa

ce

Resil

ienc

e

Scal

abilit

y

Dep

loya

bilit

y

Repl

acea

bilit

y

Chan

geab

ility

Exte

nsib

ility

Unde

rsta

ndab

ility

Auto

nom

y

Functional Programming •  Alternative programming paradigm •  Functional languages (Erlang, Haskell, Clojure, …) •  Hybrid languages (Scala, …) •  Languages with functional extensions (Python, JavaScript, Java, …)

Functional Programming

Cost-

effic

ienc

y

Unifo

rm In

terfa

ce

Resil

ienc

e

Scal

abilit

y

Dep

loya

bilit

y

Repl

acea

bilit

y

Chan

geab

ility

Exte

nsib

ility

Unde

rsta

ndab

ility

Auto

nom

y

NoSQL •  Augments the data store solution space •  Different sweet spots than RDBMS •  Key-Value Store – Wide Column Store – Document Store •  Graph Database

NoSQL

Cost-

effic

ienc

y

Unifo

rm In

terfa

ce

Resil

ienc

e

Scal

abilit

y

Dep

loya

bilit

y

Repl

acea

bilit

y

Chan

geab

ility

Exte

nsib

ility

Unde

rsta

ndab

ility

Auto

nom

y

Continuous Delivery •  Automate the software delivery chain •  Build – Continuous Integration, … •  Test – Test Automation, … •  Deploy – Infrastructure as Code, …

Continuous Delivery

Cost-

effic

ienc

y

Unifo

rm In

terfa

ce

Resil

ienc

e

Scal

abilit

y

Dep

loya

bilit

y

Repl

acea

bilit

y

Chan

geab

ility

Exte

nsib

ility

Unde

rsta

ndab

ility

Auto

nom

y

Cloud provisioning model •  On-demand provisioning and de-provisioning •  Instant availability •  Self-service •  Pay-per-use

Cloud provisioning model

Cost-

effic

ienc

y

Unifo

rm In

terfa

ce

Resil

ienc

e

Scal

abilit

y

Dep

loya

bilit

y

Repl

acea

bilit

y

Chan

geab

ility

Exte

nsib

ility

Unde

rsta

ndab

ility

Auto

nom

y

Docker •  Build, ship, run on container-basis •  Process-level isolation •  Declarative communication path configuration •  Cambrian explosion of ecosystem at the moment

Docker

Cost-

effic

ienc

y

Unifo

rm In

terfa

ce

Resil

ienc

e

Scal

abilit

y

Dep

loya

bilit

y

Repl

acea

bilit

y

Chan

geab

ility

Exte

nsib

ility

Unde

rsta

ndab

ility

Auto

nom

y

… and there are many more

What can we learn from this?

Findings •  There is not a simple solution and no “one size fits all” •  Some of the topics evaluated bear a high potential •  Some of the topics evaluated are more of a side note •  A mutually reinforcing combination of concepts is needed

How could an architectural style look like?

µServices •  Conway’s law •  Built for replacement •  Aligned with business capabilities •  Bounded Context (Domain-Driven Design) •  Separate UI and service

Bounded Context

Bounded Context

Bounded Context

µS µS

µS µS

µS

µS µS

µS µS

µS

µS µS

µS µS

µS

UI

e.g., B2C-Portal

UI

e.g., embedded in Partner-Portal

UI

e.g., Mobile App

UI

e.g., Clerk Desktop

REST interfaces •  Use as API gateway for client access •  Encapsulate dynamics and complexity of service landscape •  Provide client-driven, coarse-grained service calls

behind a uniform API based on a proven protocol •  Should be provided on bounded context level •  Decouple speed of evolvement (services vs. API)

Bounded Context

µS

µS µS µS

µS

REST API Gateway

µS

Bounded Context

Other Client

User Interface

Bounded Context

Message-based API

also okay, but requires clear and stable, client-oriented

contract

Event-driven communication •  Use for inter-service communication •  Decoupling and isolation •  Vertical slicing of functionality •  Easier evolution of flows and processes •  Configuration-visualization-monitoring support required

µS

Request/Response : Horizontal slicing

Flow / Process

µS µS

µS µS µS

µS

Event-driven : Vertical slicing

µS µS

µS

µS µS

Flow / Process

Resilient (reactive) software design •  Resilience and responsiveness are mandatory •  Elastic design for scalability •  Start with isolation and latency control •  Separate control and data flow •  Many new challenges for developers

Event/data flow Event/data flow

Resource access

Error flow Control flow

µS

Isolation

Cloud/container provisioning model •  Basis for elasticity at runtime •  Basis for speed and flexibility at development time •  Private, hybrid or public •  Should be combined with container approaches (e.g., Docker) •  “Natural” infrastructure for µService architecture

Container Manager

µS

µS

µS

µS

µS

µS

µS

µS

µS

µS

µS

µS

µS

µS

µS

Container

Explicitly declared communication paths

µService

Production-readiness •  Configuration •  Discovery, Routing •  Choreography, Orchestration •  Observability

•  Monitoring, Measuring, Logging

Configuration •  Netflix Archaius

Orchestration •  Apache Aurora on Apache Mesos •  Marathon •  Kubernetes •  Fleet

Discovery •  Netflix Eureka •  Apache ZooKeeper •  Kubernetes •  Etcd

Routing •  Netflix Zuul & Ribbon •  Twitter Finagle

Monitoring •  Hystrix •  Twitter Zipkin (Distributed Tracing)

Measuring •  Dropwizard Metrics

Logging •  ELK •  Graylog2 •  Splunk

Automation •  Automate everything •  Build, test & deployment (Continuous Delivery) •  Resource provisioning (Cloud API) •  Restart, failover, error handling (Resilience) •  Starting and tearing down instances (Scalability)

5 + 2 style •  µServices

•  REST interfaces

•  Event-driven communication

•  Resilient (reactive) software design

•  Cloud/container provisioning model

•  Production-readiness

•  Automation

One more thing …

Side note

Enterprise architecture management •  Should not impact team autonomy

•  No interference with team responsibilities

•  Focus on shared vision •  Provide common vision for all teams

•  Makes communication efficient •  To follow common vision teams need to communicate •  Minimal set of joint technical collaboration standards

required for efficient communication

•  Not a separate team •  Use representatives from the teams •  Add a person in charge if required

à Very different from traditional EAM

Wrap-up

•  Business & technological driven change

•  IT as a whole must respond to the drivers

•  Architecture must support the drivers

•  Architecture must support the organization

•  The new architectures are different

•  New challenges for developers (& ops)

We need to rethink IT

Join the most disruptive and exciting change we have seen in IT for many years

@ufried Uwe Friedrichsen | [email protected] | http://slideshare.net/ufried | http://ufried.tumblr.com