CBS Process WG. Participants David Carney (chair) M. Al-Said (scribe) Tony Jordano Kyung Whan Lee Jeffrey Poulin David Klappholz Glenn Berg Jongmoon Baik.

CBS Process WGCBS Process WG

ParticipantsParticipants

David Carney David Carney (chair)(chair)

M. Al-Said (scribe)M. Al-Said (scribe)

Tony JordanoTony Jordano

Kyung Whan LeeKyung Whan Lee

Jeffrey PoulinJeffrey Poulin

David KlappholzDavid Klappholz

Glenn BergGlenn Berg

Jongmoon BaikJongmoon Baik

Rami RazoukRami RazoukGeorge HulingGeorge HulingTim SpinneyTim SpinneySteve CrossSteve CrossMike MooreMike MooreCo-chairs:Co-chairs:

Betsy ClarkBetsy ClarkDan Port Dan Port Tricia OberndorfTricia Oberndorf

Working ApproachWorking Approach

Homework: come in with 3 issuesHomework: come in with 3 issues Brainstormed, collapsed, clusteredBrainstormed, collapsed, clustered Prioritized by importance (practitioner Prioritized by importance (practitioner

& researcher) & difficulty& researcher) & difficulty Formulated process-oriented Formulated process-oriented

statementsstatements Discussed prospective CeBASE Top 10 Discussed prospective CeBASE Top 10

ListList

Issue 1: CBS Lifecycle Issue 1: CBS Lifecycle Models and Processes - 1Models and Processes - 1

Is there a new lifecycle Is there a new lifecycle modelmodel for for CBSs? CBSs?

Can one (or more) of the existing Can one (or more) of the existing lifecycle models be tailored to lifecycle models be tailored to CBSs?CBSs?

How do we define a CBS life-cycle How do we define a CBS life-cycle processprocess??

Issue 1: CBS Lifecycle Issue 1: CBS Lifecycle Models and Processes - 2Models and Processes - 2

Both of these questions in light of:Both of these questions in light of: A requirements-driven (COTS-supported) A requirements-driven (COTS-supported)

vs. COTS-driven paradigm vs. COTS-driven paradigm Strategies for handling different kinds of Strategies for handling different kinds of

market segmentsmarket segments CBS time to market pressuresCBS time to market pressures Level of service quality needsLevel of service quality needs ……..How do you identify How do you identify [-> metrics group][-> metrics group] and and

factor in multiple CBS cost drivers?factor in multiple CBS cost drivers?

Issue 2: Post-Deployment Issue 2: Post-Deployment ProcessProcess

In what ways does the CBS post-In what ways does the CBS post-deployment process differ from:deployment process differ from:

the CBS development process the CBS development process the custom post-deployment the custom post-deployment

process?process?

In what ways do the differences affect In what ways do the differences affect cost estimation for CBSs?cost estimation for CBSs?

Issue 3: Issue 3: Release PlanningRelease Planning

What are the process implications of What are the process implications of planning for system releases where COTS planning for system releases where COTS product:product:

releases are not synchronized with each otherreleases are not synchronized with each other releases are not synchronized with your systemreleases are not synchronized with your system end-of-life occurs during your system lifeend-of-life occurs during your system life

Planning includes cost estimation, Planning includes cost estimation, scheduling, determining system release scheduling, determining system release content, etc.content, etc.

Issue 4: Assessment, Issue 4: Assessment, Evaluation & Testing Evaluation & Testing ProcessProcess

How and when do you assess, evaluate, How and when do you assess, evaluate, and test:and test:

COTS productsCOTS products COTS-based systemsCOTS-based systems““When” includes timing (e.g., multiple points When” includes timing (e.g., multiple points

in the process) and system-independent in the process) and system-independent certification of COTS products. certification of COTS products.

““How” includes process, techniques, tools, How” includes process, techniques, tools, roles, and metrics.roles, and metrics.

Related to determination of CBS requirements.Related to determination of CBS requirements.

Issue 5: COTS Market and Issue 5: COTS Market and IdentificationIdentification

What are the processes forWhat are the processes for Acquiring and maintaining market knowledgeAcquiring and maintaining market knowledge Analyzing the forces in a market segmentAnalyzing the forces in a market segment Forecasting trends in both market segments and Forecasting trends in both market segments and

specific productsspecific products Obtaining & disseminating COTS product information Obtaining & disseminating COTS product information

and product-specific experiencesand product-specific experiences Establishing and maintaining vendor relationshipsEstablishing and maintaining vendor relationships

Both the active (e.g., influencing) and passive aspects Both the active (e.g., influencing) and passive aspects of these should be considered.of these should be considered.

Issue 6: Surprise!Issue 6: Surprise!

How does a CBS process How does a CBS process accommodate/address large discontinuities accommodate/address large discontinuities stemming from COTS product surprises? stemming from COTS product surprises? Surprises result from releases or patches:Surprises result from releases or patches:

that are not backward compatiblethat are not backward compatible that fail to performthat fail to perform whose fundamental properties changewhose fundamental properties change

Exacerbated by inadequate documentation.Exacerbated by inadequate documentation. [-> architecture issue concerning product [-> architecture issue concerning product

characterization]characterization]

Issue 7: Organizational Issue 7: Organizational Assessment CapabilityAssessment Capability

How does an organization assess its How does an organization assess its capability to be successful with CBSs?capability to be successful with CBSs?

How does an organization improve its CBS How does an organization improve its CBS capability? This includes such issues as:capability? This includes such issues as:

relationship between CMMI and CBS processesrelationship between CMMI and CBS processes metrics for CBS process maturitymetrics for CBS process maturity

Applies to capabilities of CBS acquirers, Applies to capabilities of CBS acquirers, developers, and maintainers. developers, and maintainers.

Other IssuesOther Issues Dealing with extra featuresDealing with extra features Managing customer expectations (post-Managing customer expectations (post-

selection)selection) World-class SE organizations becoming COTS World-class SE organizations becoming COTS

integratorsintegrators LicensingLicensing Systematic way of deriving system challengesSystematic way of deriving system challenges In-place transition of baselines for In-place transition of baselines for

uninterruptible systemsuninterruptible systems Metrics for degree of product & system riskMetrics for degree of product & system risk

COTS Process Issue Priorities –1aCOTS Process Issue Priorities –1aImportance to Practitioner

0123456789

0.0

0.5

1.0

1.5

2.0

2.5

3.0R

ele

ase

A,E

,&T

Po

st-D

ep

loy

Su

rpri

ses

CO

TS

Mkt

&Id

CO

TS

Su

pvs

. Drv

n

Ve

nd

or

De

ficie

nci

es

Ca

pa

bili

tyA

sse

ss

Co

st D

rive

rs

Importance to Practitioner Scores

COTS Process Issue Priorities –1bCOTS Process Issue Priorities –1b

COTS Process Issue Priorities –2aCOTS Process Issue Priorities –2aImportance to be Researched Tallies

02468

1012

Cost D

river

s

A,E,&

T

COTS Sup

vs. D

rvn

Post-D

eploy

Releas

e

Capab

ility A

sses

s

Surpr

ises

Vendo

r Def

icien

cies

COTS Mkt

& Id


0.0

0.5

1.0

1.5

2.0

2.5

3.0C

ost

Dri

vers

A,E

,&T

CO

TS

Su

pvs

. Drv

n

Po

st-D

ep

loy

Re

lea

se

Ca

pa

bili

tyA

sse

ss

Su

rpri

ses

Ve

nd

or

De

ficie

nci

es

CO

TS

Mkt

&Id

Importance to be Researched Scores

COTS Process Issue Priorities –3aCOTS Process Issue Priorities –3aDifficulty Tallies

012345678

Post-D

eploy

Releas

e

Cost D

river

s

Surpr

ises

COTS Sup

vs. D

rvn

Capab

ility A

sses

s

A,E,&

T

Vendo

r Def

icien

cies

COTS Mkt

& Id


0.0

0.5

1.0

1.5

2.0

2.5

3.0P

ost

-De

plo

y

Re

lea

se

Co

st D

rive

rs

Su

rpri

ses

CO

TS

Su

pvs

. Drv

n

Ca

pa

bili

tyA

sse

ss

A,E

,&T

Ve

nd

or

De

ficie

nci

es

CO

TS

Mkt

&Id

Difficulty Scores

Release, Surprise VotesRelease, Surprise Votes

Post-deployment, AE&T Post-deployment, AE&T VotesVotes

Assessment, COTS Id. Assessment, COTS Id. Votes Votes

COTS-supp/–driven VotesCOTS-supp/–driven Votes

Priorities (Diff. = Imp.)Priorities (Diff. = Imp.)

8

Priorities (Diff/Imp = 1)Priorities (Diff/Imp = 1)

8

Prospective CeBASE Top Prospective CeBASE Top 1010

General reactions:General reactions: Use of list as guidance Use of list as guidance Good (only?) as top 10 list of Good (only?) as top 10 list of

things CeBASE will researchthings CeBASE will research Use of unfamiliar and imprecise Use of unfamiliar and imprecise

terminologyterminology Is it valid to have hypotheses on Is it valid to have hypotheses on

this list?this list?

Specifics -1Specifics -1

1. Valid within a limited scope only.1. Valid within a limited scope only.

2. What else is new? So does everything 2. What else is new? So does everything else!else! Add: “…. Just like traditional” Add: “…. Just like traditional” Or “Cost & schedule overruns are as Or “Cost & schedule overruns are as

common with COTS as with custom common with COTS as with custom development and they (CBSs) can cost as development and they (CBSs) can cost as much.”much.”


3. What is “CBS type”?3. What is “CBS type”? Domain?Domain? COTS-driven vs COTS-supported?COTS-driven vs COTS-supported? COTS product type?COTS product type? ……..?..?

4. Should be studied, but we don’t 4. Should be studied, but we don’t believe the hypothesis as posed:believe the hypothesis as posed: How can a parametric model account for the How can a parametric model account for the

situation in which the selected products can’t be situation in which the selected products can’t be integrated?integrated?


5. False, this is one of many 5. False, this is one of many parameters that influence the parameters that influence the estimate.estimate. Prefer: ” Architectural mismatch will Prefer: ” Architectural mismatch will

affect CBS cost.”affect CBS cost.”

6. “ …just as with traditional systems; 6. “ …just as with traditional systems; exaggerated with COTS products.”exaggerated with COTS products.” Change to “CBS post-deployment costs Change to “CBS post-deployment costs

dominate …”dominate …”


7. Intuitively agree; defects will be in 7. Intuitively agree; defects will be in mismatches between products, not mismatches between products, not something visible by inspection of something visible by inspection of line of code.line of code.

A (ADDITION): A (ADDITION): Writing glue code Writing glue code requires greater skill than requires greater skill than traditional coding. Often need to traditional coding. Often need to discover lots of things.discover lots of things.


8. Agree with “frequently far worse than linear”, 8. Agree with “frequently far worse than linear”, question use of square; heavily dependent on question use of square; heavily dependent on other factors (e.g.; use of standards, quality of other factors (e.g.; use of standards, quality of products, degree of integration)products, degree of integration)

9.9. Question “twice”. Question “twice”. Replace with“Risk specific to CBSs must be Replace with“Risk specific to CBSs must be

managed; CBS risks (and their mitigations) are managed; CBS risks (and their mitigations) are different; risk doesn’t disappear.” different; risk doesn’t disappear.”

Current statement is not a useful hypothesis.Current statement is not a useful hypothesis.

10. Disagree with percentage; agree vaporware 10. Disagree with percentage; agree vaporware is one of is one of manymany factors. factors.

ConclusionsConclusions

Process is foundational for CBSs.Process is foundational for CBSs.

Risk management is still key - and Risk management is still key - and will be different for CBSs.will be different for CBSs.

It was cold, but we had fun anyway!It was cold, but we had fun anyway!

CBS Process WG. Participants David Carney (chair) M. Al-Said (scribe) Tony Jordano Kyung Whan Lee Jeffrey Poulin David Klappholz Glenn Berg Jongmoon Baik.

Documents