SCAMPI Class A v1.3

SCAMPI Sampling Rules1

Sampling the SCAMPI Sampling Rulesor

Trying to Explain the Unexplainable

Pat O’Toole, PACTwww.PACTCMMI.com

May, 2012

http://www.pactcmmi.com/


SCAMPI Class A v1.3 SCAMPI

–Standard CMMI Appraisal Method for Process Improvement

SCAMPI MDD– SCAMPI Method Definition Document

250+ page process document for SCAMPI A appraisals

Specific source of angst–SCAMPI MDD v1.3 Section 1.1.4–It’s only 7 pages, but do NOT drive or operate heavy

equipment after reading it!


New Terminology with MDD v1.3 Basic Units

Sampling Factors

Subgroups

Coverage Rules.



Sampling Factors

Subgroups

Coverage Rules.


Basic Unit For CMMI-DEV, “Basic Unit” is roughly equivalent

to “Project”–However, “project” doesn’t work as well for the

Services or Acquisition constellations–“Basic Unit” is much more generic–Therefore, for purposes of this presentation, I will use

“PROJECT!”

<of Work>



Sampling Factors

Subgroups

Coverage Rules.


Sampling Factors – Why? Project work may be performed in different ways

depending on various characteristics– Small projects may be conducted differently than large ones– Project work conducted by your USA-based team may be

done differently than your offshore development group– Scrum projects are run differently than Waterfall projects

Sampling factors ensure that the organizational scope provides a robust (and complete) view of the various ways work is performed by the organization.


Mandatory Candidate Sampling Factors The Sampling Factors that MUST be considered:

–Location (e.g., Minneapolis vs. Las Vegas)–Customer (e.g., DoD vs. commercial industry)–Size (e.g., small vs. medium vs. large vs. 4XL)–Organizational Structure (e.g., different divisions)–Type of Work (e.g., development vs. maintenance)

For each candidate, the question to be answered is:–Do different settings for this factor affect the way that

work is performed in the organization to be appraised? Yes – Relevant Sampling Factor No – Irrelevant Sampling Factor.


Other Potential Sampling Factors Other Relevant Sampling Factors may exist and, if

so, must also be considered–Life Cycle (e.g., Scrum projects vs. Waterfall projects)–Funding Source (e.g., customer vs. internal funding)–Criticality (e.g., nice-to-have vs. life critical)–Etc.

Your organization may have unique characteristics that significantly affect how work is performed–These would be included as Relevant Sampling Factors

in a SCAMPI A appraisal.


ExampleProject Location Customer Size Type of Work

1 MSP Internal Small Maintenance2 MSP Internal Medium Development3 MSP External Medium Development4 MSP External Medium Maintenance5 MSP External Small Maintenance6 MSP External Large Development7 MSP External Medium Development8 MSP External Medium Development9 MSP External Medium Development

10 MSP External Large Development11 MSP External Small Maintenance12 MSP Internal Medium Development13 MSP External Medium Maintenance14 MSP External Small Maintenance15 MSP External Medium Development



Sampling Factors

Subgroups

Coverage Rules.


Subgroup “Cluster” of projects that share common relevant

sampling factor values and exhibit similar process implementations–If Customer, Size, and Type of Work are relevant

sampling factors, how many POSSIBLE subgroups are there? Customer: Internal / External Size: Small / Medium / Large Type of Work: Development / Maintenance

2 values for <Customer> x3 values for <Size> x2 values for <Type of Work> = 12 possible subgroups.


How Many ACTUAL Subgroups?Project Location Customer Size Type of Work

1 MSP Internal Small Maintenance2 MSP Internal Medium Development3 MSP External Medium Development4 MSP External Medium Maintenance5 MSP External Small Maintenance6 MSP External Large Development7 MSP External Medium Development8 MSP External Medium Development9 MSP External Medium Development

10 MSP External Large Development11 MSP External Small Maintenance12 MSP Internal Medium Development13 MSP External Medium Maintenance14 MSP External Small Maintenance15 MSP External Medium Development

3-tuple #Internal , Smal l , Dev.

Internal , Medium, Dev.

Internal , Large, Dev.

Internal , Smal l , Maint.

Internal , Medium, Maint.

Internal , Large, Maint.

External , Smal l , Dev.

External , Medium, Dev.

External , Large, Dev.

External , Smal l , Maint.

External , Medium, Maint.

External , Large, Maint.


Internal , Medium, Dev.

Internal , Large, Dev.











Internal , Medium, Dev. 2Internal , Large, Dev.























Internal , Smal l , Maint. 1Internal , Medium, Maint.























External , Medium, Dev. 5External , Large, Dev. 2External , Smal l , Maint. 3External , Medium, Maint. 2External , Large, Maint.


Number of projects in the given subgroup

Number of projects in the given subgroup

How Many Projects per Subgroup?

Minimum # of projectsfrom a given subgroup

=

Number of subgroups X

Total number of projects


Back to Our Example…Subgroups #

InternalMedium

Development2

InternalSmal l

Maintenance1

ExternalMedium

Development5

ExternalLarge

Development2

ExternalSmal l

Maintenance3

ExternalMedium

Maintenance2

# of IMD = (6 x 2) / 15 = 0.8 => 1

# of ISM = (6 x 1) / 15 = 0.4 => 1

# of EMD = (6 x 5) / 15 = 2.0 => 2

# of ESM = (6 x 3) / 15 = 1.2 => 1

# of ELD = (6 x 2) / 15 = 0.8 => 1

# of EMM = (6 x 2) / 15 = 0.8 => 1.


What if…

# of IMD = (6 x 200)/1500 = 0.8 => 1

# of ISM = (6 x 100)/1500 = 0.4 => 1

# of EMD = (6 x 500)/1500 = 2.0 => 2

# of ESM = (6 x 300)/1500 = 1.2 => 1

# of ELD = (6 x 200)/1500 = 0.8 => 1

# of EMM = (6 x 200)/1500 = 0.8 => 1

Subgroups #InternalMedium

Development200

InternalSmal l

Maintenance100

ExternalMedium

Development500

ExternalLarge

Development200

ExternalSmal l

Maintenance300

ExternalMedium

Maintenance200



Sampling Factors

Subgroups

Coverage Rules.


Coverage Rules Three flavors:

–Coverage Rules for Process Areas–Coverage Rules for Basic Units (whoops! – Projects)–Coverage Rules for Support Functions.


Cover Rules for Process Areas PA Coverage Rule 1 (in for a penny, in for a pound)

–Evaluation of a given PA must cover all practices, both specific and generic

–There is a wee bit of wiggle room here as the rule includes the words, “…for which they supply data”

PA Coverage Rule 2 (no need to keep pounding)–If a PA is performed in a consistent way across

subgroups, evaluation of that PA may be consolidated–For example, if CM, DAR, PPQA, and RSKM are

performed in the same manner across all projects, then they need not be evaluated for all 6 subgroups.


Project Coverage Rules Project Coverage Rule 1

– For each subgroup, both artifacts and affirmations shall be provided for at least one project for every process area This sampled project shall provide data for all process areas

Project Coverage Rule 2– For at least 50% of the sampled projects in each subgroup,

both artifacts and affirmations shall be provided for at least one process area

Project Coverage Rule 3– For all sampled basic units in each subgroup either

artifacts or affirmations shall be provided for at least one process area.


ATLAS #19a Can you apply what you’ve learned?

–Well, heard?

Let’s find out…

Work individually or together to answer the questions in ATLAS #19a–There is no need to provide comments.


Cover Rules for Process Areas PA Coverage Rule 1

–Evaluation of a given PA must cover all practices, both specific and generic

–HINT: Start with the question, “Which projects contribute to its fulfillment?”


Project Coverage Rules Project Coverage Rule 1

– For each subgroup, both artifacts and affirmations shall be provided for at least one project for every process area This sampled project shall provide data for all process areas

Project Coverage Rule 2– For at least 50% of the sampled projects in each subgroup,

both artifacts and affirmations shall be provided for at least one process area

Project Coverage Rule 3– For all sampled basic units in each subgroup either

artifacts or affirmations shall be provided for at least one process area.


ATLAS #19a Results ATLAS #19a was distributed to all SCAMPI lead

appraisers and only SCAMPI lead appraisers

61 SCAMPI lead appraisers responded–That’s about a 12% response rate–But it’s still more than “one expert’s opinion”

Let’s see how YOU did compared with those responsible for REALLY understanding this stuff…


Question 1

1. Is Process Area Coverage Rule 1 fulfilled?<Evaluation of a given PA must cover all practices, both specific and generic.>

36% Yes / 64% No

Which project(s) contribute to its fulfillment:

P1? 82% Yes / 17% No

P2? 90% Yes / 10% No

P3? 74% Yes / 24% No

P4? 7% Yes / 93% No.


Question 2

2. Is Project (BU) Coverage Rule 1 fulfilled?“For each subgroup, both artifacts and affirmations shall be provided for at least one project for every process area”

34% Yes / 66% No


P1? 42% Yes / 56% No

P2? 14% Yes / 86% No

P3? 4% Yes / 96% No

P4? 4% Yes / 96% No.


Question 3

3. Is Project (BU) Coverage Rule 2 fulfilled?“For at least 50% of the sampled projects in each subgroup, both artifacts and affirmations shall be provided for at least one process area”

51% Yes / 49% No


P1? 54% Yes / 46% No

P2? 97% Yes / 3% No

P3? 22% Yes / 78% No

P4? 18% Yes / 82% No.


Question 4

4. Is Project (BU) Coverage Rule 3 fulfilled?“For all sampled basic units in each subgroup either artifacts or affirmations shall be provided for at least one process area.”

48% Yes / 52% No


P1? 94% Yes / 6% No

P2? 98% Yes / 2% No

P3? 93% Yes / 7% No

P4? 25% Yes / 75% No.


Optical Illusion – 2 Images in One


Modified Coverage Rules - #1 PA Coverage–1: Objective evidence provided for a PA must

address all practices that are part of that PA, both specific practices as well as all generic practices. The Basic Unit and Support Function coverage rules indicate the types of objective evidence that must be provided for each practice.

Project Coverage–1: For each subgroup, both artifacts and affirmations shall be provided for each practice for at least one project.

Project Coverage–2: For at least 50 percent of the sampled projects in each subgroup, both artifacts and affirmations shall be provided for each practice for at least one PA

Project Coverage–3: For all sampled projects in each subgroup either artifacts or affirmations shall be provided for each practice for at least one PA.


Modified Coverage Rules - #2 PA Coverage–1: Objective evidence provided for a PA must

address all practices that are part of that PA, both specific practices as well as all generic practices. [No change]

Project Coverage–1: For each subgroup, some combination of both artifacts and affirmations shall be provided for at least one project.

Project Coverage–2: For at least 50 percent of the sampled projects in each subgroup, some combination of both artifacts and affirmations shall be provided for at least one PA

Project Coverage–3: For all sampled projects in each subgroup either artifacts and/or affirmations shall be provided for at least one PA.


Conclusion The new sampling and coverage rules are intended to

provide a truly representative set of projects that reflect how work is performed in the organization–There are some issues, however:

They only solve the “proportional problem,” not the “how much is enough” problem

There are multiple ways to interpret the rules as written, arriving at significantly different results

Based on ATLAS #19a, the SEI has committed to clarify, or even re-write the coverage rules

With respect to brain function, covering this topic is equivalent to drinking 6 beers; please drive carefully!


Questions???

Pat O’TooleProcess Assessment, Consulting & Training (PACT)

[email protected]

If anyone is interested in receiving my monthly “Do’s and Don’ts” articles, either:

Give me a business card

Send me an email.

SCAMPI Class A v1.3

Documents