Yves caseau@md day2011

Yves Caseau – Complexity, Modularity, Abstraction – November, 2011 1/31

Complexity, Modularity, Abstraction:

Designing Architecture-Oriented Services

Model-Driven Day

November 24th, 2011 (v0.0)

Yves Caseau

Bouygues Telecom – Académie des Technologies


Outline

1. Complexity

the 21st century challenge for enterprises and their information

system

2. Enterprise Architecture

Anticipation in a complex world is not forecasting,

but building a « situation potential »

3. SOA and sustainability

repeatable long-term performance requires discipline /

governance

4. Cloud-Ready Architecture

Distributed, scalable, massively parallel computing resource for

the whole information system


Companies are Facing a Complex World

A Complex World:

Hyper-competition, globalization, time is shrinking

The power has shifted to the consumers (F. Dupuy)

T. Friedman : « All that is easy has been done, what’s left is the hard

stuff »

Complicated problems require specialists,

Complex problems require everyone

Diversity of skills and viewpoints …

… organized into teams

Complex problems are solved “on the gemba”,

where they occur, one at a time

Abstractions hide too much, decomposition does not work !

“Reproducible conditions” … do not always exist (isolation is impossible)

Communication is hard (cf. IT when specifying is harder than coding)

Part

I:

Com

ple

xty


An Enterprise is a « Complex System »

Numerical complexity (number of things)

Multi-scale

Temporal complexity

Richness of interactions with environment

Symptom examples:

Costs (Information Systems evolution)

Example: non-linear evolution of project costs w.r.t. project size

Rate of errors & failures

Difficulty to « guarantee » robustness and fault-tolerance

Ross Ashby « regulation of a (complex) system requires a control system

that is as complex as the system itself »

Time-to-market

The first complexity-related issue

Time-to-integrate-a-new-component depends on the host size :

– Human complexity (organization)

– Modularity failure (unforseen impacts & interaction between components)

Law of unintended consequences – Feature Interaction Problem

Part

I:

Com

ple

xty


A systemic view of the enterprise & its IS

Decision System

Information System

IS

environnement

Operating

System

Precesses

Input flows

Structure

Finality

Evolution

Business Model (CEISAR)

Organisation (CEISAR)

Political Aspects

Objets

Changes

“Pragmatic” aspect

operation procedures

“pragma” aspect

Functional

Model Process

Model

Semantic aspect

Logic aspect

IS is a

complex

system itself

Software aspect

Hardware

Geographical

/ localization

Part

I:

Com

ple

xty

PRAXEM

E


Consequences of a « systemic vision »

Features / properties « emergence »

from « Performance by Design » …

.. to « Performance though simulation & prototypes »

Humility and Continuous improvement

Complexity Governance

Acknowledge the problem !

Attack it with method

Cf. CEISAR’s cube

(three dimensions)

Explicit policies are a key governance tools

SLA, service contacts, data ownership policies, …

“Enterprise Architecture” as a corporate practice

Stakeholders alignment

The “external environment” is critical to IS strategic planning

Complexité

Agilité

Synergie

Exécution

dans le

Monde Réel

Modèle

Eléments Spécifiques

Operations Transformations

Eléments Partageables

ou Réutilisables

Part

I:

Com

ple

xty


Measuring Complexity

Measuring the numerical complexity

Sizing the objects (TPMC, FP, Tb)

Complexity (« relationship intricacy ») Metrics

DSM (design structure matrix)

Cyclomatic complexity

Scalar Euclidian Complexity

The only scale-invariant metric that applies to an architecture map

Bottom line : metrics offer a small amount of help

Part

I:

Com

ple

xty

Gyx

yxG),(

)().(),(


Taming Information System Complexity (I)

Simple approach:

maps and architecture schemas (UML2 helps ).

Common sense approach:

Energetic standardization

Reducing heterogeneity reduces complexity (Printz).

Technological approach:

Infrastructure (middleware)

Introduction of physical de-coupling, sharing and intermediation

Part

I:

Com

ple

xty


Taming Information Systems Complexity (II)

Systematic approach (« Urbanisation »):

Enterprise Architecture, seen as a company-wide practice, since it helps to

align everyone towards a common target and a transformation path.

Architectural approach (hardest, at the structure level):

modularity (semantic de-coupling).

Strategic Approach (Service-Oriented Architecture):

SOA as a governance practice, reduces complexity:

share & reuse

Common understanding

Reduces the temporal complexity

Part

I:

Com

ple

xty

Cf. Parts

II & III


Part II

1. Complexity


system






governance

4. Cloud-ready architecture



Lessons from Bouygues Telecom

Back-office re-engineering 2001-2005


Enterprise Architecture

Architecture: why ? IS Re-engineering

Communicate a vision

Key transformation tool

Favor reuse

Reduce complexity and costs

Support standardization

Align stakeholders

Architects need to avoid

complex tools & formalisms

Sensitive to each

enterprise’s maturity level

Asynchronous / diachronous

Acts as a corporate memory

Visual change management

« Urbanisation »

Component-based

Process-oriented (business logic extraction)

Temporal de-coupling (asynchronous message)

Functional de-coupling (intermediation)

« Enterprise Architecture »

Coherence of three domains

Strategy : goals

operations: processes and data

Information Systems: applications and services

Similar topic / two views

EA is more concerned with the target

Re-engineering is a process

Part

II:

Arc

hit

ectu

re


Data Architecture

Data Model

Semantic reference: what does a customer, a bill, … mean ?

Conceptual model: how is a customer defined ?

Ontology: class hierarchy (UML)

Major business asset & key alignment tool

Data Architecture

Distribution

Formats (ex: XML)

Life cycle

Dynamic Management of Business Objects

Distribution / synchronization

Save / restore (disaster recovery)

Data flow management

E.g., network capacity planning

Part

II:

Arc

hit

ectu

re


Functional Architecture and Object-Oriented Design

Functional Architecture means to decompose:

Into functions and sub-functions, in a top-down manner

Descriptive normalization: (input, output, transformation, roles, …)

Functional Architecture is no longer self-contained(vs. 20 years ago)

A narrow focus on functional architecture leads to take business process

and data distribution issues into account too late.

When functional analysis is carried too deeply, one gets “”silos”

Functional top-down is poorly suited to the efficient use of off-the-shelf

software packages

Object-Oriented Design :

mixing functional & data model at the IS level

Functional Architecture feeds :

Business roadmaps & « level 0 » decomposition: descriptive analysis of

activities/jobs within the company

Service definition within SOA

Contributes to data architecture and business object model

Part

II:

Arc

hit

ectu

re


Service Architecture

Service = Function + Interface + Contract

Service Architecture

Design a structure (clean up the call graph)

Provide meaning (to simplify change management and reuse)

“local” SOA = service-based architecture

Often linked to technology, such as « Web Services »

A new look for an old best practice

The goal is the system (and its architecture), services are a mean

“global” SOA = build a service catalog, whose rich structure is “an architecture”

Independent from technologies (Tuxedo, procedures, …)

Reuse of previous software component reuse theory, applied to large components which are pieces of the IS

The goal is to design a reusable service catalog (the lasting asset), architecture (organization) is a mean (and varies across time)

Part

II:

Arc

hit

ectu

re


Process Architecture

Design a structure on top of temporal interactions:

Events

Chaining & dependencies => business logic

Goals => processes

Describe « fractal/recursive » structure for processes

Processes / sub-processes

Process families

Sharing resources: data, GUI, …

Roles (organizational alignment)

Business process mapping is the best alignment tool between

organization and information system

Process description -> services, functions, … (cf. previous slides)

Normalize / Standardize

Share / reuse / outsource

Best approach to integrate packaged software

Part

II:

Arc

hit

ectu

re


Designing a Target Architecture

Functions Processes Data

Lexicography Métiers

Objets (ontology)

Object Lifecycle

Macro-functions Macro-processes

(draft)

Macro-processes

Exchanges - Content

Events

Services

Processes

Update protocol

Data Archi. v1 Process Archi v1 Service Archi. v1

Level 0

Catalog

External

Reference

Référence

externe

Part

II:

Arc

hit

ectu

re

External

Reference


Designing a Modular Architecture

Goal: minimize impact dispersion for new services

“Definition”: modularity is the correlation

« Distance in the code » and frequency of interaction

« Distance in the code » and « co-evolution »

Good practices :

Layered architecture (define abstraction levels)

Process Architecture (define a composition grammar)

Sharing/reuse & modularity go hand-in-hand : sub-process

identification

Event-Oriented Architecture

Pub/sub is still a one of the best modular patterns

Model-Driven Architecture:

careful design of « future-proof » data model

Service Architecture reduces unmanaged interactions

Reification of functional architecture

Abstraction/ encapsulation

Part

II:

Arc

hit

ectu

re

Cf. Part III


Part III

1. Complexity


system






governance




Image blog


Strategic Issues : Tomorrow’s Information System

Complexity does not mean that some challenges are not clearly visible

Information Systems 2.0 – our customers …

want their voice heard: they need a feedback mechanism

collaborate with other customers or prospects

expect Information Systems to be always on, any time on any device

Information Systems need to be personalized & predictive

Social networks analytics, Bayesian learning, …

Semantic processing : making sense from customer & partner input

Time acceleration generates more data - need to master « big data »

analytical skills

Customers and end-users demand to be more autonomous

End-users have control over their own processing , write their own « apps »

IS must follow the customer wherever she goes, not the opposite !

Customer are « the architect of their own experience » → IS must become

interoperable platforms

Part

III:

Sust

ain

abilit

y &

SO

A


Sustainable Information Systems »

« develop today’s information services without preventing the ability

to develop those for tomorrow, through the overuse of resources, or

the production of un-manageable complexity».

Freely inspired from « sustainable » as defined by Brundtland Commission

(global) SOA is the only method for sustainable IS development

Not the only re-engineering method

(other approaches exist which reduce IS complexity),

but the best method to do so continuously, as a company-wide

practice (with all stakeholders), a long-term progressive &

measured effort, which generates its own reward

Clean-up : learn to remove/trip (classic from asset management)

Cf. Extreme programming (Agile Manifesto) :

Leveling the effort, continuous integration, favor simple designs

Continuous simplification, not a last-resort heroic attempt

Part

III:

Sust

ain

abilit

y &

SO

A


SOA & Governance :

Three steps:

Service Définition: build the business service catalog.

Starts like a functional analysis (from business processes) – but reusability and

composability is engineered during this step, trough the dialog between business

and IS.

Architecture de Service: Transform a list into a hierarchical and modular structure.

Classical difficulties and solutions (ex: refactoring) …

to avoid a « Web Services spaghetti».

Service Intégration : the « technical » step – often confused (SOA SOI).

Integration/middleware technology is quite mature nowadays

“Think globally, act locally” SOA starts with the periphery of the system

and ends with the core. Data architecture is critical from day one.

Beware of the “proof of concept” which is much harder to integrate than

to build

« Something you do, not something you buy » - David Linthicum

SOA Governance

its first goal is to establish the existence of shared artifacts (architecture

schemas, service catalogs, roadmaps) et everyone’s role (rights and

duties)

Part

III:

Sust

ain

abilit

y &

SO

A


For a company with expected growth g, the IT budget is defined as

the sum of the project budget P and the Operation budget O.

An application portfolio of size S is built, with life expectancy A

a given share (d) of the application portfolio is removed/destroyed

We suppose that the operation cost for an application that costs 1k€ is w k€

We also suppose that there is a productivity gain over the years of p%.

The usual two ratios that are of interest :

r = (P / O) = ration of build / run - usual measure R = (P / P + O)

n = Pn / P = % of the project budget spent on creating new applications

Sustainability Theorem : stable IT budget relative to company growth

n = [A (g + d) ] / (1 + (g+d) A) & r = [g + d + p +(1-d)/A] / [w (1 - d + A(g+d))]

Corollary

Easy to plug into an Excel spreadsheet

Example: w = 25%, d = 5%, p = 4%, A = 10,

we get R = 42% and n = 33%. (no surprise )

How to improve r ?

Clean-up old applications

Increase productivity for operations

Sustainability’s Equation Part

III:

Sust

ain

abilit

y &

SO

A


SOA as a discipline : Architecture-oriented Services

How get reusability, across the enterprise (sharing)

and across time (reuse) ?

Abstraction

Trade-off between too generic and too specific

Modularity

Relies on process and event architecture

Composability

Horizontal (Process) : Common (Pivotal) Object Model

Vertical (Functional) : Parametric Polymorphism

API Model Discipline

Manage versions !

API (meta) data model : worth some effort !

Each CIO becomes a software editor

This is more an art than a science

Part

III:

Sust

ain

abilit

y &

SO

A


MDA: where Abstraction and Modularity start

Data model is the key factor for modularity & flexibility

A few rules drawn from experience:

Reify links

Reify roles

Hierarchy products vs. complex ontologies

Group/individual substitution

Work on the meta-model

Think « object »

Ontology before description, encapsulation (hiding vs. transparency)

The data model defines the IS « situation potential »

Difficulty of strategic planning in a complex world

scenario practices: what if …

… we were … one of our competitor ?

… we outsourced this activity ?

… we provided this service to other companies (SaaS)

Part

III:

Sust

ain

abilit

y &

SO

A


Part VI

1. Complexity


system






gouvernance





Cloud Computing : Scalable Resources & Services

Two angles :

Cloud as a computing resource : a CIO issue

Cloud as a service platform : joint ownership

A « new » kind of resource

The heir of grid and « commodity servers »

Fast provisioning

A new kind of business model

CAPEX to OPEX (rent)

Full scalable (up and down)

Part

IV:

Clo

ud-r

eady A

rchit

ectu

re


The Innevibilaty of Cloud Computing

Better TCO

Commoditization

economy of scale (operations)

Utilization rate (from sharing)

PUE (Power Usage Effectiveness)

Parallel computing performance

Example from the movie industry

MapReduce & Hadoop

Distributed computing reliability

Massive redundancy …

State-of-the-art availability (99.9% to 99.99%)

Flexibility

Scalability

Pay-as-you-grow

Part

IV:

Clo

ud-r

eady A

rchit

ectu

re


Don’t Confuse Clouds with Smoke

Data privacy & security

A major concern for most citizens

Not all data centers are equal in front of hackers

Data architecture must distinguished hot/warm/cold

Brewer’s Theorem (Distributed computing is still hard )

Consistency (all views obtained through different process represent

the same information)

High-availability (relies on massive replication)

Fault-tolerance (disaster recovery)

Cloud, Caching and Peer-to-Peer Architecture

« Cloud projection over the edges »

Digital homes future architecture

Part

IV:

Clo

ud-r

eady A

rchit

ectu

re


SOA is not scale-free / speed of light is still a constraint

Latency is still a constraint

Tens of ms to access the cloud

Performance is not guaranteed (or price is no longer cheap)

Service « scale » (size) is crucial

SOA is not scale-free

A classic rule of « performance management » for SOA

Even more important with a cloud-architecture

RCA: Rich Client Architecture

Used for SaaS and MMORPG … for a reason

Applies to heavy transactional contexts

Part

IV:

Clo

ud-r

eady A

rchit

ectu

re


Cloud-ready Architecture

A cloud-ready architecture supports progressive migration

of both front-office & back-office services to the cloud

SaaS for front-office is easier : way to start …

Portal integration, then mash-up

Variation of a service-oriented

architecture

Requires to:

Leverage Web technology

“think platform” cf. “What would Google Do”

Build catalogs of

Web APIs

Part

IV:

Clo

ud-r

eady A

rchit

ectu

re

Annuaire UDDI

Applications interactives(ex: portail)

Processus

événements

Batchs

service Propriété clés:• Stateless•Gestion explicite du contexte•Contrat de service

Référentiels(données)

« Cloud »(Internet)

Applications Ressources

Infrastructure (ex: ESB asynchrone)

Infrastructure (ex: ESB synchrone)

service

Mash-ups


Conclusion

Models are key for:

agility

situation potential (seizing opportunities)

cost management

Innovation in the digital world happens in the source code

agile development (end user / designer / developer)

Fail often to succeed early (Eric Riess)

Lean IT: no delays, quality in, fast delivery, less code,

short intervals (small batches), customer participation,

VSM & « muda » removal

You should prepare to the advent of massively parallel computing

Cloud or multi-core

Even for traditional back-office functions

From VNG to DCG

Yves caseau@md day2011

Technology

21st century

yves caseau

sustainability

ready architecture

enterprise

ready architecture

data model

sustainability