Getting Some Respect - How to Measure and Communicate Your EA Success

Getting Some Respect:

How to Measure and

Communicate Your EA SuccessDavid Baker

Michael Janiszewski

Diamond Management and Technology Consultants

[email protected],

[email protected]

October 26, 2007

Page 1

Introductions

We both have extensive experience practicing Enterprise Architecture

as well as developing Enterprise Architecture organizations,

processes, and measurement techniques

David Baker

Partner and chief architect for

Diamond Management &

Technology Consultants

Mike Janiszewski

Manager for Diamond

Management & Technology

Consultants

Page 2

Introduction – Ice Breaker

Split up into groups and discuss your expectations for this workshop

Before you begin the discussion, assign a person who will take notes and share the highlights of the discussion with rest of the team

Upon completion of the assignment, share your group’s thoughts

Intro – 5 minutes

Exercise – 10 minutes

Debrief – 15 minutes

Page 3

Introduction – Ice Breaker

Take the next 10 minutes to discuss these three questions…

1. What do you hope to learn from this workshop?

2. Why are EA metrics important to your organization?

3. What emerging trends are shaping the future of EA measurement?

Page 4

Share your group’s discussion

Page 5

By the end of this course, you should be able to…

Explain the importance and classification of EA metrics

Include appropriate resources in your EA metrics program

Analyze EA measurements to improve your EA program

Integrate EA metrics into your organization’s IT lifecycle

Identify training and management issues associated with launching

an EA program

Identify organizational problems that may limit your EA metrics

program

Design your own EA metrics deliverables and reporting templates

Page 6

8:00 AM Introduction

8:30 AM What Problem Are We Solving?

9:00 AM Context for Measuring EA

9:30 AM Team Exercise

10:00AM Break

10:15 AM EA Metrics – What to Measure

11:00 AM Organizational Impacts

11:30 AM Lunch

12:00 PM Metrics Process – How to Use Metrics

12:30 PM Deliverables and Templates (Case Examples)

1:15 PM Team Exercise

1:45 PM Break

2:00 PM Metrics Frameworks

2:30 PM Using Tools

2:45 PM Wrap-up

Agenda

Page 7

What Problem Are We Solving?

We know EA is valuable, but our stakeholders often do not

Business

“I just can’t see the benefit of our EA spend. Do we really

need this EA thing around?”

IT Managers

“We know doing EA is a best practice, but we can’t really

quantify how good we are at it. You [the business] can’t

measure it that way”

Technical Staff

“Our EA is so conceptual, we don’t know if we’re really doing

what you want us to do with it”

Page 8


The reality is…

EA offers significant benefits

Architecture and IT are cost organizations

EA groups don’t have significant budgets

We need to demonstrate the EA value associated with our business

impact

If we can’t quantify our value, we shouldn’t be in business

Metrics are a key part of marketing the value of EA

Page 9


Discussion: How do you market your EA

What do you measure?

EA operations?

EA compliance?

Business value?

How do you communicate those measures?

Page 10


We can market EA by focusing on results

Is my IT portfolio aligned with

the business?

Is IT meeting expectations

efficiently?

What risks am I taking?

Business

wants to know

IT

Managers

want to know

Is my strategy being realized?

Is my EA delivering business

value?

Technical

Staff

wants to know

Am I making efficient use of EA assets?

How well am I aligning with our EA?

What things should I NOT be developing?

EA Metrics

Page 11


Why are EA metrics important?

Metrics quantify what we do in a useful way

Metrics allow stakeholders to make informed decisions

Manage change

Set targets and challenges

Manage risk

Celebrate what goes well

Understand and address what does not

The capability to measure, track, and evaluate drives continual

improvement

Page 12


EA metric capture and reporting is difficult for a variety of

reasons

Scope – The Enterprise

Dynamic - EA is a moving target

Skills - Lack of development within the discipline

Vague - Success and failure are sometimes difficult to define

Measurability - Actionable EA deliverables are not always

discrete/easily measurable

Priorities - Stakeholders value other disciplines to assess

performance (e.g., program management, development practices)

Page 13





10:00AM Break



11:30 AM Lunch




1:45 PM Break


2:30 PM Using Tools

2:45 PM Wrap-up

Agenda

Page 14

Context for Measuring EA

EA Operates within the context of the IT Lifecycle…

Business

Strategic

Planning

IT Strategic

Planning

Release Planning(Portfolio

Mgmt)

ProjectExecution

(SDLC)

Business

Operations

Use enterprise and business unit

direction and goals to drive IT plans

Develop projects that

support businesses’

annual and strategic

plans

Prioritize the allocation of

IT resources to achieve

business strategy, in

alignment with enterprise

architecture

Run the business

Portfolio 1

Blueprints

Portfolio 2

Blueprints

Portfolio 3

Blueprints

Enterprise

Blueprints

Filte

r

Project

Project

Project

Project

Project

Multi-Year Plan Budget Cycle Project Cycle Continuous

Page 15


…with EA involvement at every stage

Business

Strategic

Planning

IT Strategic

Planning


Mgmt)

ProjectExecution

(SDLC)

Business

Operations

Strategic Architecture Innovation

Blueprinting

Standards Definition

and Management

Reference Architecture

Application Portfolio

Management Application Portfolio

Rationalization

Inventory Refresh

Annual Budgeting Cycle Impact Analysis

Risk Analysis

QA of IT Financials

RFP Process Support Content

Vendor Support

Assessment and

Selection

EA Oversight Architecture Reviews

Architecture Issue

Management Issue Management

Corrective Action Plan

Escalation Management

Exception Management

Architecture Support Short Term Issue

Resolution

Long Term Engagement

Needs Assessment

Page 16


EA activities during strategic planning

Associated Processes

Participants

Things to measure

Innovation

Architecture Planning

Standards Definition and Management


Domain architects responsible for Architecture Planning

Enterprise architects responsible for other Strategy processes

Business owners and SMEs

Guidelines (tech standards, blueprints, reference architecture)

used in other Arch. functions, (e.g. Project Execution)

Objective Define Enterprise Architecture guidelines

Business

Strategic

Planning

IT Strategic

Planning

Page 17


EA activities during release planning

Business

Strategic

Planning

IT Strategic

Planning


Mgmt)


Participants

Things to measure

Application Inventory

Management

Inventory Refresh

Domain architects responsible for these portfolio activities

Enterprise architects provide oversight and enterprise view

Objective Review the application portfolio

Facilitate financial review of the application portfolio

Review content of RFPs, RFIs; assist in vendor evaluation

Impact Analysis

Risk Analysis

QA of IT Financials

Content

Vendor Support

Assessment and

Selection

Updated application

inventory

Inventory assessments

Dependencies

Tech, bus. & delivery

risks

Updated costs

RFP, RFI

RFP/proposal

assessment/scoring

Page 18


EA activities during project execution and operations

Business

Strategic

Planning

IT Strategic

Planning

Release

Planning(Portfolio

Mgmt)

ProjectExecution

(SDLC)

Business

Operations


Participants

Things to measure

Issue Management

and Gate Reviews

Vendor Assessment

Oversight

Domain architects responsible for these delivery activities

Enterprise architects provide oversight

CAPs, issues, exceptions, deferrals

generated during reviews

Objective Monitor adherence to standards, blueprints, reference architectures

Manage architecture-related issue resolution/escalation

Provide support for requests for architecture assistance

Issue Management


Escalation Mgmt

Exception Mgmt

Short Term Issue

Resolution

Long Term Support

Needs Assessment

Resolution of CAP

Project support

Page 19


Discussion: How many of these processes do you have in

place?

How mature is your EA practice?

Is there a relationship between the maturity of your EA practice and

your ability to measure it’s impact?

Page 20





10:00AM Break



11:30 AM Lunch




1:45 PM Break


2:30 PM Using Tools

2:45 PM Wrap-up

Agenda

Page 21

Team Exercise 1

Brainstorm EA metrics

Brainstorm EA metrics by stakeholders

Break into teams, teams will brainstorm metrics in different topic areas

Before you begin the discussion, assign a person who will take notes and share the highlights of the discussion with rest of the team

Upon completion of the assignment, debrief rest of the team about list of EA metrics required by various stakeholders

Intro – 5 minutes


Debrief – 15 minutes

Page 22

Team Exercise 1

Brainstorm EA metrics

Topic 2 - Brainstorm a list of

EA metrics necessary for IT

Managers

Business

wants to know

IT

Managers

want to know

Topic 1 - Brainstorm a list of EA

metrics necessary for Business

Technical

Staff

wants to know

Topic 3 - Brainstorm a list of EA

metrics required by Technical Staff

EA Metrics

Topics for discussion

Page 23





10:00AM Break



11:30 AM Lunch




1:45 PM Break


2:30 PM Using Tools

2:45 PM Wrap-up

Agenda

Page 24





10:00AM Break



11:30 AM Lunch




1:45 PM Break


2:30 PM Using Tools

2:45 PM Wrap-up

Agenda

Page 25

EA Metrics – What to Measure

EA not only takes place at different points in the Lifecycle, it also

takes place at different levels of detail

The Enterprise

Business Domain

(line of business)

Business Domain

(line of business)

Business Domain

(line of business)

Business Domain

(line of business)

Scope:

Enterprise

One (or more)

lines of

business

One (or more)

individual

projects

Page 26


It is helpful to group EA metrics into a small number of

categories

Value Creation Measures benefits delivered as result of architecture

processes

IT/Business

Alignment

Measures how aligned a technical project is with respect to

the underlying business strategic requirements

Risk Management Measures architectural risks inherent in project activities

(e.g. new platform risks, sunsetting, upgrades)

Operational

Effectiveness

Measures compliance accuracy with respect to enterprise

architecture

Measures effectiveness of architecture processes such as

governance

Page 27


Measuring EA is different from program management

Enterprise Architecture

Cross-LOB and cross-program

measures

On the path to the future state

Alignment focused

Blueprint earned value

Business operations

e.g. compliance with SLAs

Program Management

Cross-project and project

measures

On-time, on-budget

Execution focused

Project earned value

System operations

e.g. system uptime

Page 28


Supporting that difference requires a linked architecture…

Solution Architecture

Business Architecture

Technology Architecture

Desired Business Capabilities

Business

Strategy

Infrastructure Model

Interface

ModelInformation

Model

Application

Model

Data

Models

App

Models

Development

Models

Execution

Models

Operations

Models

Network

Models

Security

Models

Business

Operations

Blueprint &

RoadmapService Model

Page 29


…with sufficient detail to allow the relationships to be measured

Solution Architecture

Business Architecture

Business Services

Mission

Vision

Goals

Objectives

Infrastructure Model

InterfacesMaster

DataApplications

Organization

Stakeholders

Functions

Data / Application / Common Services

Business

Metrics

Linked business metrics

Supported business functions

Am I building the right things?

Reuse things that are of

interest to the enterprise

Page 30


Value creation metrics

Metric Definition Calculation

Blueprint Value

Measurement

Metric that provides quantitative view of

advancement towards a blueprint. Metric

is created by tracing back from initiative -

> capability -> objective -> corresponding

performance indicator (yearly target)

IT initiatives must have assigned business capabilities

Calculate project advancement based upon blueprint's earned value measurements

Architecture

Coverage

Comparison of the top 10 architecturally

complex projects versus the top 10 high

business priority projects

Also applies to Linkage metrics category

Estimated architecture complexity* for each initiative

Obtain top 10 architecturally complex projects

Obtain top 10 projects with the highest business priority

Count the number of projects that intersect

% of Innovation

Ideas

Converted into

Production

Captures the success rate of ideas

explored by architecture

# of ideas explored that ended up in production divided by the total # of ideas explored

* Discussed during “project typing” in “Metrics Process – How to Use Metrics”

Page 31


Value creation metrics (continued)

Metric Analysis

Blueprint Value

Measurement

The business should ensure that IT is making continual progress on delivery of the

defined business capabilities

IT should be able to explain any acceleration / deceleration in delivery

The business should be able to clearly see how infrastructure projects

(investments) support desired business capabilities

Architecture

Coverage

There is no right or wrong answer here, but it is valuable to show how many of the

business priorities depend upon solutions to architecturally complex problems. This

demonstration is valuable to the EA community.

# of Innovation

Ideas Converted

into Production

A high percentage is good, but does too high mean there’s not enough innovation

(no risk being taken)?

Page 32


Additional value creation metrics (continued)

% of Blueprints Defined

# of Shared Technology Services Identified

# of Shared Technology Services Delivered

# of Reference Implementations Developed

Savings from reuse of Standard Patterns

TCO Savings from Application Retirement

Page 33


IT/Business alignment metrics


Spend amount

per business

goal

Cost of all IT projects associated with

delivery of a business goal. Can be at the

enterprise or LOB level.

IT initiatives must have assigned

business goals

Sum the initiative costs for each goal

Number of

projects

involved with

each of the

business goals

Can be at the enterprise and LOB level

IT initiatives must have assigned

business goals

Count the projects under each initiative

Change

Agenda

Identifies the traceability between

business strategy, objectives and

strategic technology investments.

IT initiatives must be traceable back to

business capabilities and capabilities

traceable back to objectives and goals

Page 34


IT/Business alignment metrics (continued)

Metric Analysis

Spend amount

per business

goal

Look for cases where the spend is not commensurate with the business priority

High priority business goals with low spend (good thing)

Low priority business goals with high spend (bad thing)

Number of

projects

involved with

each of the

business goals

Helps identify high risk business goals

High number of projects may reflect complex implementation and therefore

high risk

Change Agenda

Can be used to conduct impact analysis of project delivery on advancing the

business strategy (and vice versa)

What is the business impact of NOT spending on an IT project?

What is the IT impact of changing a business strategy?

Page 35


Risk management metrics


Technology

Lifecycle Risk

Risks associated with project

architectural component maturity.

Measured in relation to product

technology lifecycle

Calculate percentage of lifecycle risks

attributed to each piece of the

technology lifecycle (e.g. emerging

technology, mature outside your

company, obsolete technology)

Architecture

risk count

An architecture risk is a technical

characteristic of a solution that has a

potential negative impact on that

solution’s chances of a success (e.g.

vendor risk, new technology risk, security

risk, supportability risk, . . . )

Accumulate risks as they are identified.

Classify both their likelihood (high,

medium, low) and severity (severe,

moderate, low)

Segment by architecture domain

Severe

Architecture

Risk Ratio

Measures the number of architecture

risks across technical domains

Calculate ratio of high probability and

severe risks to total architectural risks

per project.

Segment by architectural domain

Page 36


Risk management metrics (continued)

Metric Analysis

Technology

Lifecycle Risk

Provides insight into technology lifecycle root causes for projects and articulates

where the enterprise is with respect to technology early adopter, late adopter, etc.

This information can be used to ensure that projects use the right strategy for

implementing the EA vision (e.g. due to high importance of integration efforts, such

projects should use DB2 (mature) as opposed to IDMS (sunsetting))

Architecture

risk count

Can help identify common architecture elements that are at risk.

Can help identify weaknesses in architecture skills (architecture domain disciplines)

Severe

Architecture

Risk Ratio

Provides insight into the overall severity of architectural risks encountered by

projects. Segmentation by architectural domain allows a starting point to take

action to remedy at-risk projects (and at-risk reference architectures)

Page 37


Operational effectiveness metrics - compliance


Compliance Percentage of project using / reusing

standards, services, reference

architectures

% of projects using standard products

% reuse of standard services, patterns

EA Compliance

Efficiency

Measures the percentage of projects

with architecture signoff on first try

Calculate number of projects that

receive governance signoff on first try

divided by total number of reviewed

projects

EA Compliance

Ratio

Measures the percentage of projects

fully compliant with standards by

business domain and/or technology

architecture domain

Calculate number of reviewed projects

found to be fully compliant with EA

standards divided by total number of

reviewed projects

Exception/

Deferrals

frequency

Measures the frequency of

exceptions/deferrals in information,

application, technical and business

architecture as well as standards

Calculate the total number of

exceptions/deferrals per architecture

domain

Reference

Recommendation

Revision Count

Number of reference material revision

suggestions submitted by business

domain and/or technology architecture

domain

Calculate total number of suggestions

against a given reference material (e.g.

business domain blueprint)

Page 38


Operational effectiveness metrics - compliance (continued)

Metric Analysis

Compliance

A low percentage could indicate an incomplete set of guidelines, or poorly

communicated guidelines, or just the wrong set of guidelines

A high percentage shoud be correlated with better program management metrics

EA Compliance

Efficiency

Provides insight into accuracy of projects' delivery in compliance with your

company’s EA standards

EA Compliance

Ratio

Allows the business to understand the level of EA compliance within its project

portfolio. Provides confidence level in delivered architecture.

Exception/

Deferral

frequency

Too many exceptions and deferrals may indicate an unachievable architecture

vision or possibly an unachievable project schedule

Exceptions and deferrals are and important and valuable tool for IT governance,

yet deferrals should be tracked to ultimate remediation

Reference

Recommendation

Revision Count

Provides insight into use of reference materials. Also provides insight into

reference material needed by the enterprise.

Page 39


Additional operational effectiveness (compliance) metrics

Number of exceptions (by level)

Reference Recommendation Revision Approval Ratio

Exception/Deferrals frequency by Blueprints across domains

% of Projects Rejected

% of Applications that are in Obsolescence Stage

Page 40


Operational effectiveness metrics - governance


Oversight

Frequency

Measures the number of architecture

oversights conducted per week

Calculate the total number of oversights

Find the average over a week

Oversight

Efficiency

Measures the amount of time / level of

difficulty to complete the architecture

reviews for a project

Calculate the number of days and

sessions required to complete signoff

per project per gate (also average)

Issues,

exceptions,

deferrals,

corrective action

plans

Totals and averages for the various

dispositions of architecture issues

accumulated during reviews

Track, count, and average totals for

each

Provide totals per project, per gate

Trending A complete view of architecture health

includes the total counts as well as their

week-over-week trend

Week-to-week comparison of key

metrics (issues, CAPs, exceptions,

deferrals)

Indicate week-over-week trend

Page 41


Operational effectiveness metrics – governance (continued)

Metric Analysis

Oversight

Frequency Ensures governance processes are occurring according to targets

Oversight

Efficiency

Ensures that governance processes remain lightweight and do not impact project

delivery schedules

Issues,

exceptions,

deferrals,

corrective action

plans

Low numbers indicate maturity level of project compliance

Compare these historical numbers with problems during development and

operations to identify value of architecture compliance

Trending Trend combined with phase can warn of impending problems

Upward trend can signal the need to deploy senior architects

Page 42


Additional operational effectiveness (governance) metrics

Issue Closure Efficiency

Oversight Capacity Sufficiency

Page 43





10:00AM Break



11:30 AM Lunch




1:45 PM Break


2:30 PM Using Tools

2:45 PM Wrap-up

Agenda

Page 44

Organizational Impacts

Discussion: Architecture Organization

Do you identify different types of architects?

Or is everyone becoming an architect?

How are your architects organized?

Highly distributed?

Centralized?

Virtual?

Who develops standards and reference architectures?

If you do blueprints, who does them?

What role does architecture play in project delivery?

Page 45


Enterprise architecture is delivered via several different types of

architects

Type Role

Enterprise Architect

Oversee the blueprinting process to translate enterprise and business unit

strategies into the technology roadmap

Ensure interoperability across domains

Ensure compliance with architectural blueprints, roadmaps, and standards

Domain Architect

Translate business unit strategy into functional domain (such as Sales) blueprints

Provide deep business and technology knowledge

Guide project architects to ensure project compliance with enterprise and domain

architectural blueprints

Project Architect

Provide architectural expertise to lead the solution delivery projects

Responsible for overall system architecture and integration with other domains

and systems

Systems Engineer

“Internal” architecture

Requirements analysis, detailed software design

Process planning and control

Verification, validation

Software Developer Code, unit testing

Page 46


Architects are social, community oriented creatures, found in

groups

Infrastructure

Infra

architect

Solutions

Delivery

Project

Architect

Systems

Engineer

Software

Developers

Enterprise

Architecture

Business

Architect

Application

Architect

Information

Architect

Business Unit

C

Domain

Architect

Business Unit

B

Domain

Architect

Business Unit

A

Domain

Architect

Enterprise Architecture Virtual Community

CEO

Operations CIO

Diagram assumes a large

organization, same principles

apply to smaller organizations

but structure will be different

Page 47


The various architects engage at different levels to deliver

projects

Enterprise Architecture

Services Services Services

Sample Application Domain

Application Architecture

Infrastructure Architecture

Business Architects

Domain Manager

Systems Engineers

Domain Architects

Project Manager

Project Architects

Software Developers

Solutions Delivery

Services Services Services

Application Development Services

Quality Engineering Services

Program Management Office

Emerging Technologies

Business Strategy

IT Strategy

Business Environment

(regulation, compliance,

market factors)

Architecture

Community

(updates, exceptions)

Release Planning

Information Architecture Blueprints

(Enterprise &

Domain)

Page 48


EA requires the attention of a small group focused on

Architecture project management

The Enterprise Architecture Group should be a small, high powered

group of Enterprise Architects

One per architecture discipline

Domain architects may also be in this organization (or in the

business)

The Enterprise Architecture Group should also contain a small

number of project management resources

Oversees the operation of the virtual community and its

working groups

Manages and tracks oversight and issue management

This is the group that collects and reports the EA metrics

Page 49


Transformation to / creation of a centralized enterprise

architecture group requires:

Identify the true architects

Be sure to look in the business groups as well

Software engineers / programmers don’t always make the bets

architects

Establish career path for architects

Application -> project -> domain -> enterprise

Assess architect skills to identify gaps

Define standard architecture processes so that standard metrics can

be measured and reported across the enterprise

Innovation, blueprinting, portfolio assessment, governance,

issue management, etc

Page 50


Training and change management

Establish a small enterprise architecture group, initial focus on

establishing the virtual community and defining the architecture

processes (including metric definition)

Provide adequate training and learning opportunities to teach the

standard architecture processes to both architect and non-architect

resources

Implement these processes one domain at a time, gaining

experience as you go

Start with the operational metrics (governance and compliance)

Add risk management metrics as your processes mature

Then add the alignment and business value metrics

Page 51


Training metrics

Metric Definition Calculation% completion of

Architecture

Training

Curriculum

Measures progress towards creation

and delivery of architecture training

Establish a curriculum

Use curriculum project plan to estimate

completion

# of Skills Gaps

addressed

Identification of sub-par skills among

the architects and specific plans to

address each

Skills Assessments

Count of skills gaps

Plans to address each gap

% of Architects

trained

To understand the level of training

activity occurring in EA

# of architects trained divided by total

number of architects

Page 52


Training metrics (continued)

Metric Analysisx% completion of

Architecture

Training

Curriculum

A high percentage indicates that the Architecture training curriculum is complete

and available for use

# of Skills Gaps

addressed

Ensure any identified skills gap is being addressed either via training or on job

experience

% of Project

Architects trained Low is bad, high is good

Page 53





10:00AM Break



11:30 AM Lunch




1:45 PM Break


2:30 PM Using Tools

2:45 PM Wrap-up

Agenda

Page 54





10:00AM Break



11:30 AM Lunch




1:45 PM Break


2:30 PM Using Tools

2:45 PM Wrap-up

Agenda

Page 55

Metrics Process – How to Use Metrics

Establish the reporting process

Establish the Architecture program management office

Establish and communicate targets

Track trends

Integrate metrics with

Business & IT Strategy planning (feedback loop)

Release Planning

Project Execution and Operations

Page 56


Remember the context within which EA takes place

Business

Strategic

Planning

IT Strategic

Planning


Mgmt)

ProjectExecution

(SDLC)

Business

Operations

Strategic Architecture Innovation

Blueprinting

Standards Definition

and Management



Management Application Portfolio

Rationalization

Inventory Refresh

Annual Budgeting Cycle Impact Analysis

Risk Analysis

QA of IT Financials

RFP Process Support Content

Vendor Support

Assessment and

Selection

EA Oversight Architecture Reviews

Architecture Issue

Management Issue Management


Escalation Management

Exception Management

Architecture Support Short Term Issue

Resolution

Long Term Engagement

Needs Assessment

Page 57


Responsibility for collecting and using metrics varies by

lifecycle stage

Business

Strategic

Planning

IT Strategic

Planning


Mgmt)

ProjectExecution

(SDLC)

Business

Operations

Strategic Architecture Enterprise Architects

Domain Architects


Management Domain Architects

Annual Budgeting Cycle Domain Architects

RFP Process Support Domain Architects

EA Oversight Domain Architects

(primary)

Enterprise Architects

Architecture Issue

Management Domain Architects

Enterprise Architects

Architecture Support Domain Architects

Page 58


Follow a goal -> objective -> metric framework to establish

targets

This approach should be built into

the your business architecture

Generate domain specific

business metrics for

measuring value and

alignment

For each objective, set

progress goals by establishing

yearly targets

Approach can also be applied to

generating targets for the risk and

operational metric categories

Strategic Business Architecture

KE

Y D

RIV

ER

S &

GU

IDIN

G P

RIN

CIP

LE

S

MISSION

VISION

GOAL GOAL GOALGOAL

BUSINESS

METRICS

CAPABILITIES

OBJECTIVE

Page 59

Example capabilities form a financial management system

Maintain common processes centrally

across the Department to support

centralized administration and

standardization

Support ad hoc data access across all

LOBs. This will provide a simplified,

single source for report information in a

timely fashion

Generate performance and

accountability reports, within OMB

specified timeline

Capability for drill-down to transaction

level information

Improve financial performance

Improve operating efficiency of financial

management and procurement functions

Consistently comply with federal,

accounting, and system standards

Improve financial performance

Capability Linked Goal

Page 60


Using metrics in the Business & IT Strategy phases

Technology

Assessment• Review current architecture

and plans

• Document current technology environment and pain points

• Confirm enterprise architecture strategy and initiatives

Business Alignment• Confirm business strategy

and goals

• Refine understanding of

requirements driven by

recently defined business

strategy

Technology Future

State Definition• Define key technology

capabilities to enable the

business strategy

• Define future state IT vision

and models

Roadmap• Develop and prioritize

projects and required

investments

• Define implementation plan

and analyze economics

• Define risk mitigation strategy

Business Future State

Definition• Define key business

capabilities to enable the

business strategy

• Define future state business

vision and models

Generate future state

alignment & value

metrics

Apply future state

alignment, value and risk

metrics

Apply current state

alignment and value metrics

Apply current state risk &

operational metrics

Page 61


Using metrics in the release planning phase

Initiate Release

Plan

Budget & Risk Assessment Update Release

Plan

Transition

Initial

Release

Plan

Release

Schedule

Initiative

Risks

Initiative

Resource

Reqs

Initiative

Depend-

encies

Initiative’s

Budget

Updated

Release

Plan

Initiative-focused Domain-wide

Initiative/

Portfolio-focused

Initiative

Assessment

Solution Design

Project Initiation

Blueprint

Phase

Roadmap

Project

focused

Domain - wide

Sequenced

Initiatives

Project

Charter

Approved &

Funded

Apply risk

metrics

Apply compliance

metricsApply alignment

metrics

Page 62


Metrics can be used to determine a project’s complexity rating

Are new technologies going to be introduced into the

environment?

How mature is the new technology?

New

Technologies

Are the service levels going to be significantly changed (e.g.

number of users, performances, availability levels)?Service Level

Changes

Are there significant impacts to applications outside of the

business domains?

Are there significant impacts to shared technology services?

Do the impacts require new model of integration?

Does the project proposal match overall architecture

planning? Is an implementation out of sequence with

proposed roadmap?

Are applications to be sunset being significantly enhanced?

Is data redundancy being introduced?

Are there significant exceptions to existing technology

standards and reference architectures?

Are products to be sunset used?

Key Considerations

Cross-Domain

Impacts

Architecture

Planning

Compliance

Technology &

Information

Standards

AttributeLow

More than one

exception

No standard

exception

Significant

deviations

No standard

exception

Significant

impacts

No impact

Significant new

technologies

No new

technologies

Significant service

level changes

No service

level changes

Are we enabling new / modifying existing business

capabilities?

Are new user types being introduced?

Are there compliance issues?

Significant impact

to business

No impact to

business

Business

Impact

HighMedium

Page 63


Project typing drives the level of governance during the project

execution phase

Basic:

Operational metrics such as Issue,

Exception, Deferral Mgmt

Moderate:

Domain level business

value, blueprint alignment,

EA compliance

Maximum:

Enterprise & Domain

Architect Involvement

Category Characteristics

• Fewest # of Gate Reviews

• “Low” metrics indicators

• Driven by Project Architect with

oversight by Domain Architects

• Moderate # of Gate Reviews

• “Medium” metrics indicators

• Domain Architect involved,

Enterprise Architect involvement as

needed

• Highest # of Gate reviews

• “High” metrics indicators

• Enterprise and Domain Architects

involved

Level of EA Metric Reporting

Type 3

Projects

Type 2

Projects

Type 1

All Projects

Page 64





10:00AM Break



11:30 AM Lunch




1:45 PM Break


2:30 PM Using Tools

2:45 PM Wrap-up

Agenda

Page 65

Deliverables and Templates (Case Examples)

Case #1 – Healthcare Payer

The Company

~$20bn multi-line healthcare insurance company employing over 27,000 people with over

$19bn in revenue managing over 10 million medical members in all 50 states

The Challenge

Lack of integration across business units presents fragmented view of constituents

Second highest cost structure amongst direct competitors

Slow and expensive product development inhibits ability to anticipate or lead in the market

Corporate vision and strategy in place without an execution plan

The EA program

Phase I: Establish a program level architecture governance & design board to ensure the

3-Year program adheres to architecture goals & principles

Phase II: Establish an architecture governance process & metrics/measurement model for

the entire I.T. organization

Page 66


Sample Report: Value creation report

Purpose: To provide visibility into the alignment of the actual release state with original target

release goals

Description: The Value Creation Report illustrates the each domain’s alignment with functional

and technical business capabilities. This report takes into account the impact of

exceptions and deferrals on the intended release state of each domain and exposes

any gaps that have formed between target and actual end-states.

Frequency: Monthly/Quarterly

Audience: CTO, Business Owners

Empty circle - Not addressed in releaseHalf circle - Core functionality met, high exception score3/2 circle - Core functionality met, low exception scoreFull Circle - On target with goal

Page 67


Sample Report: Architecture health status report

Purpose: To provide a detailed illustration of the architectural health of each domain.

Description: The architecture health status report is a synthesis of key metric data provided by

the architecture dashboard and textual highlights of key issues and milestones.

Frequency: Bi-weekly

Audience: CTO, Program Directors

Page 68


Sample Report: EA governance operational effectiveness report

Purpose: To illustrate project compliance with blueprints and standards and to measure

reference material effectiveness

Description: The Enterprise Operational Effectiveness Report will measure project compliance

with domain architecture standards, enterprise blueprints, and EA standards. It will

also measure the usage and effectiveness of EAG reference material.

Frequency: Bi-weekly

Audience: CTO, EA Core Team, Domain Architects, Program Leads, project Architects

Page 69


Process example: Weekly project architecture health reporting

EA Manager

EA Admin

Domain admin

Project Lead

Enterprise

Architecture

PMO

Domain Lead

Project Team

Determine project

health status

Consolidate all

TSMs health status

information

Create health

report overview

Request Architecture

health status

overview

Submit

aggregate

health status

report

1Project Architect

Submit Weekly

project health

status report

Triggering

Event

End

ResultProcess GateOrg Role Package Process Break

Legend

Page 70


Case #2 – Pharmaceutical Company

The Company

A leading pharmaceutical company, with over 100,000 employees and $19 billion in reported net income for 2006

The Challenge

Having launched an aggressive enterprise-wide modernization program, the company was looking for a way to ensure the program delivered the promised capabilities and economic improvements

The EA program

Improve enterprise architecture service delivery through new organization and governance structures and strong alignment with business value

Provide the mechanisms to more effectively link architecture to concerns of the business as well as to the updated software Delivery Model

Create meaningful and objective measures which can be used to assess the success of architectural efforts in business terms

Page 71


Sample Report – Local Portfolio Architecture Health

Page 72


Sample Report – Business Unit Portfolio Architecture Health

Page 73


Sample Report – EA Operational Effectiveness Dashboard

Page 74


Sample Report – Business Alignment and Value Creation

Page 75





10:00AM Break



11:30 AM Lunch




1:45 PM Break


2:30 PM Using Tools

2:45 PM Wrap-up

Agenda

Page 76

Team Exercise 2

What will you do differently when you get back to the office?

Consider metrics that you can begin to implement in your organization

This is an individual exercise using the worksheets in your handouts

Identify one or two metrics in each category that would benefit your organization

Jot down some notes about what you might do to get a metrics project underway

The instructors will be around to help you think about what you can do

Intro – 10 minutes


Page 77

Team Exercise 2

EA Metrics Worksheet (1 of 3)

Oversight Frequency

Oversight Efficiency

Issues, Exceptions, Deferrals, CAPs

Trending

Issue Closure Efficiency

Oversight Capacity Sufficiency

__________________________________

Operational Effectiveness (Governance) What will do you do?_________________________________________________

_________________________________________________

_________________________________________________

_________________________________________________

_________________________________________________

_________________________________________________

Spend per Business Goal

Number of Projects per Business Goal

Change Agenda Metrics

__________________________________

__________________________________

IT/Business Alignment Metrics What will do you do?_________________________________________________

_________________________________________________

_________________________________________________

_________________________________________________

_________________________________________________

Page 78

Team Exercise 2


Compliance

EA Compliance Efficiency

EA Compliance Ratio

Exception/ Deferrals frequency

Reference Recommendation Revision Count

Number of exceptions (by level)

Reference Revision Approval Ratio

Exception/Deferrals frequency by Blueprints

% of Projects Rejected

% of Applications in Obsolescence Stage

__________________________________

Operational Effectiveness (Compliance) What will do you do?_________________________________________________

_________________________________________________

_________________________________________________

_________________________________________________

_________________________________________________

_________________________________________________

_________________________________________________

_________________________________________________

_________________________________________________

Technology Lifecycle Risk

Architecture risk count

Severe Architecture Risk Ratio

Risk Management Metrics What will do you do?

_________________________________________________

_________________________________________________

_________________________________________________

Page 79

Team Exercise 2


Blueprint value

Architecture Coverage

% of Innovation Idea converted

Shared Technology Services

% of Blueprints Defined

# of Shared Technology Services Identified

# of Shared Technology Services Delivered

# of Reference Implementations Developed

Savings from reuse of Standard Patterns

TCO Savings from Application Retirement

__________________________________

Value Creation Metrics What will do you do?_________________________________________________

_________________________________________________

_________________________________________________

_________________________________________________

_________________________________________________

_________________________________________________

_________________________________________________

_________________________________________________

_________________________________________________

Page 80





10:00AM Break



11:30 AM Lunch




1:45 PM Break


2:30 PM Using Tools

2:45 PM Wrap-up

Agenda

Page 81





10:00AM Break



11:30 AM Lunch




1:45 PM Break


2:30 PM Using Tools

2:45 PM Wrap-up

Agenda

Page 82

Metrics Frameworks

Federal Enterprise Architecture – Performance Reference Model

Source: “How to Use the Performance Reference Model”, Version 1

Page 83

Metrics Frameworks

The PRM structure is designed to clearly articulate the cause

and effect relationship between inputs, outputs and outcomes


Page 84

Metrics Frameworks

Key intersections of the PRM and other management processes

Budget Assess progress and inform subsequent budget decisions

GPRA Progress towards relevant PRM Measurement Indicators informs

evaluations to inform subsequent strategic plans

Program

Assessment

Rating Tool (PART)

Progress towards Measurement Indicators can facilitate cross-agency

evaluation of PART programs within a BRM Sub-function. Higher-

scoring programs can share best practices with lower-scoring

programs with similar missions.

IT Capital Planning

& Investment

Control (CPIC)

Progress towards Measurement Indicators inform updated Exhibit

300s (business cases) and Agency Post-implementation Reviews

Enterprise

Architecture

Progress toward indicators can determine changes needed in

Migration Plans


Page 85

Metrics Frameworks

Example: Using PRM measures to track business case (Exhibit

300) progress


Page 86

Metrics Frameworks

COBIT – The overall framework is a form of IT lifecycle

Source: “CobiT 4.0”, www.itasca.org

Page 87

Metrics Frameworks

COBIT – Interrelationship of components


Page 88

Metrics Frameworks

COBIT – IT Metrics and indicators


Page 89





10:00AM Break



11:30 AM Lunch




1:45 PM Break


2:30 PM Using Tools

2:45 PM Wrap-up

Agenda

Page 90


An integrated EA toolset strategy is critical to EA success

Enterprise Architecture Tools

Architecture Deliverables Support

Support tools such as repository tools help provide a centralized store that can help the enterprise gain access to,

learn and implement standards and blueprints. EA modeling tools can also help deliver information in a reusable

and consistent manner

EA Governance Support

Tools

Governance support tools

such as scheduling tools

allows the EA PMO to

operate and function

effectively and efficiently.

Such tools can optimize

performance of the EA PMO

Project Implementation

Tools

Project Implementation tools

enable the sharing of

important knowledge across

the enterprise leading to

fewer redundancies and

increased visibility and

clarity. Issue/ exception/

deferral tracking and

management, approval

tracking are some examples

Strategic Planning

Tools

Strategic planning tools such

as Metrics/Dashboard tools

enables visualization of ties

between the business,

vision, goals, and objectives.

This feedback will help

escalate visibility of projects

to upper management

isolating key issues and

common inadequacies

Page 91


Tools must operate across governance levels and easily scale to

address enterprise-wide number of projects

EA Governance

Support Tools

Project Implementation ToolsStrategic Planning

Tools

Architecture

Deliverables

Support

LEVEL 2

LEVEL 1

LEVEL 3

Decision

Log

Level 1 EA

Issue, Exceptions &

Deferral

Tracking and

Reporting DB

EA Modeling

Tools

EA Repository

Tools

Level 2 & 3 EA

Issue, Exceptions &

Deferral

Tracking and

Reporting DB

CAP

Mgmt

EA Governance

Dashboard

Status Reporting Scheduling

Tools

Document

Management

Tools

Page 92





10:00AM Break



11:30 AM Lunch




1:45 PM Break


2:30 PM Using Tools

2:45 PM Wrap-up

Agenda

Page 93

Wrap-up

Key success factors for an EA metrics program

Get your architects working in unison via common architecture

processes

Innovation, blueprinting, portfolio management, governance, issue

management

Engage your architects in all phases of the IT Lifecycle

Business & IT Strategy

Release Planning (Budgeting/Funding)

Project Execution

Operations

Use a virtual community to report EA progress

Business value, Alignment, Risk, Compliance/Governance

Establish an EA program management office to aggregate and

report EA progress and value

Page 94

Thank you and good luck with your metrics efforts!

Getting Some Respect - How to Measure and Communicate Your EA Success

Business

core functionality

measureoperational

measuring

innovation

diamond management

performance

architecture

measureadditional