-
Evolving a MeasurementEvolving a MeasurementProgram for Systems
andProgram for Systems and
Software Engineering ProcessSoftware Engineering
ProcessImprovementImprovement
CMMI ConferenceCMMI ConferenceDenver, ColoradoDenver,
Colorado
20042004
-
Techniques for Meas Pgm - 2version CMMI Conf 2004© 2004 Kasse
Initiatives, LLC
WelcomeWelcome
Wilkommen
Bienvenido
WelKom
Bienvenue
BienvenutoVälkom
men
Tervetuloa Witamy
Huan Yín
ЌАΛΟΣΟΡΙΣΑΤ
Ε
����
-
Techniques for Meas Pgm - 3version CMMI Conf 2004© 2004 Kasse
Initiatives, LLC
MeasurementMeasurementIs It ReallyIs It
ReallyNecessary?Necessary?
-
Techniques for Meas Pgm - 4version CMMI Conf 2004© 2004 Kasse
Initiatives, LLC
At Project Start, DoAt Project Start, DoYou Know…?You Know…?
�Can it be done?�How long will it take?�How much will it
cost?�How many people will it take?�What is the risk?�What are the
tradeoffs?�How many errors will there be?
-
Techniques for Meas Pgm - 5version CMMI Conf 2004© 2004 Kasse
Initiatives, LLC
What Do You KnowWhat Do You KnowNow?Now?
�How much does your current developmentprocess cost?
�How much value does each piece of theprocess add?
�What would the impact be of deleting,modifying, adding a
procedure to the process?
�What activities contribute the most to the finalproduct
cost?
�Have you tried to improve the currentdevelopment process?�What
changes in cost/value resulted from that
improvement effort?
-
Techniques for Meas Pgm - 6version CMMI Conf 2004© 2004 Kasse
Initiatives, LLC
What Do You KnowWhat Do You KnowNow?Now? -- 22
�What processes represent the greatestpotential for return on
improvementinvestment?
�How would you quantify the value of theprocess improvement
investment?
�Do you really want to know where the moneyis going in your
software developmentprojects?
�What value do you think you are delivering toyour customers? Do
they agree?
�How much is the knowledge of your costs andthe value delivered
worth to you?
-
Techniques for Meas Pgm - 7version CMMI Conf 2004© 2004 Kasse
Initiatives, LLC
MeasurementMeasurementand Analysisand Analysis
-
Techniques for Meas Pgm - 8version CMMI Conf 2004© 2004 Kasse
Initiatives, LLC
Measurement andMeasurement andAnalysis OverviewAnalysis
Overview
�A measurement initiative involves thefollowing:�Specifying the
objectives of measurement and
analysis such that they are aligned with establishedinformation
needs and business objectives
�Defining the measures to be used, the datacollection process,
the storage mechanisms, theanalysis processes, the reporting
processes, andthe feedback processes
�Implementing the collection, storage, analysis, andpresentation
of the data
�Providing objective results that can be used inmaking business
judgments and taking appropriatecorrective actions
-
Techniques for Meas Pgm - 9version CMMI Conf 2004© 2004 Kasse
Initiatives, LLC
EstablishEstablishMeasurementMeasurementObjectivesObjectives
�Measurement objectives�Document the purposes for which
measurement
and analysis is done�Specify the kinds of actions that may be
taken
based on the results of the data analyses�Continually ask the
question – what value will this
measurement be to those people who will be askedto supply the
raw measurement data and who willreceive the analyzed results
�Involve the end users of the measurement andanalysis results in
setting measurement objectiveswhenever possible
�Maintain traceability of the proposed measurementobjectives to
the information needs and businessobjectives
-
Techniques for Meas Pgm - 10version CMMI Conf 2004© 2004 Kasse
Initiatives, LLC
Specify MeasuresSpecify Measures
�Identify candidate measures based ondocumented objectives and
refine intoprecisely defined, quantifiable measures
�Define how data can and will be derived fromother measures
�Examples of Commonly Used DerivedMeasures�Defect density�Peer
review coverage�Reliability measures
-
Techniques for Meas Pgm - 11version CMMI Conf 2004© 2004 Kasse
Initiatives, LLC
Specify Data CollectionSpecify Data Collectionand Storage
Proceduresand Storage Procedures
�Specify how to collect and store the data foreach required
measure�Make explicit specifications of how, where, and
when the data will be collected
�Develop procedures for ensuring that the datacollected is valid
data
�Ensure that the data is stored such that it is easilyaccessed,
retrieved, and restored as needed
-
Techniques for Meas Pgm - 12version CMMI Conf 2004© 2004 Kasse
Initiatives, LLC
Specify AnalysisSpecify AnalysisProceduresProcedures
�Define the analysis procedures in advance�Ensure that the
results that will be fed back
are understandable and easily interpretable�Collecting data for
the sake of showing an
assessor the data is worthless�Showing how it can be used to
manage and control
the project is what counts
-
Techniques for Meas Pgm - 13version CMMI Conf 2004© 2004 Kasse
Initiatives, LLC
CollectCollectMeasurement DataMeasurement Data
�Collect the measurement data as defined, atthe points in the
process that were agreed to,according to the time scale
established
�Generate data for derived measures
�Perform integrity checks as close to thesource of the data as
possible
-
Techniques for Meas Pgm - 14version CMMI Conf 2004© 2004 Kasse
Initiatives, LLC
Analyze theAnalyze theMeasurement DataMeasurement Data
�Conduct the initial analyses�Interpret the results and make
preliminary
conclusions from explicitly stated criteria�Conduct additional
measurement and
analyses passes as necessary to gainconfidence in the
results
�Review the initial results with all stakeholders�Prevents
misunderstandings and rework
�Improve measurement definitions, datacollection procedures,
analyses techniques asneeded to ensure meaningful results
thatsupport business objectives
-
Techniques for Meas Pgm - 15version CMMI Conf 2004© 2004 Kasse
Initiatives, LLC
Store theStore theMeasurement DataMeasurement Dataand Analyses
Resultsand Analyses Results
�The stored information should contain orreference the
information needed to:�Understand the measures�Assess them for
reasonableness and applicability
�The stored information should also:�Enable the timely and cost
effective future use of
the historical data and results�Provide sufficient context for
interpretation of the
data, measurement criteria, and analyses results
-
Techniques for Meas Pgm - 16version CMMI Conf 2004© 2004 Kasse
Initiatives, LLC
Communicate theCommunicate theMeasurement ResultsMeasurement
Results
�Keep the relevant stakeholders up-to-dateabout measurement
results on a timely basis
�Follow up with those who need to know theresults�Increases the
likelihood that the reports will be
used�Assist the relevant stakeholders in
understanding and interpreting themeasurement results
-
Techniques for Meas Pgm - 17version CMMI Conf 2004© 2004 Kasse
Initiatives, LLC
Measurement andMeasurement andAnalysis GroupAnalysis Group
�Consider creating a measurement group thatis responsible for
supporting the Measurementand Analysis activities of multiple
projects
-
Techniques for Meas Pgm - 18version CMMI Conf 2004© 2004 Kasse
Initiatives, LLC
Measurement andMeasurement andAnalysis ToolsAnalysis Tools
�Incorporate tools used in performingMeasurement and Analysis
activities such as:�Statistical packages�Database
packages�Spreadsheet programs�Graphical packages�Packages that
support data collection over
networks and the internet
-
Techniques for Meas Pgm - 19version CMMI Conf 2004© 2004 Kasse
Initiatives, LLC
Measurement andMeasurement andAnalysis TrainingAnalysis
Training
�Provide training to all people who will performor support the
Measurement and Analysisprocess�Data collection, analyses, and
reporting processes�Measurement tools�Goal-Question-Metric
Paradigm�How to establish measures
� how to determine efficiency and effectiveness�Quality factors
measures (e.g., maintainability,
expandability)�Basic and advanced statistical techniques
-
Techniques for Meas Pgm - 20version CMMI Conf 2004© 2004 Kasse
Initiatives, LLC
Basic MeasuresBasic Measures
-
Techniques for Meas Pgm - 21version CMMI Conf 2004© 2004 Kasse
Initiatives, LLC
Basic MeasuresBasic Measures
�Estimate Size and/or Complexity - a relativelevel of difficulty
or complexity should beassigned for each size attribute
�Examples of attributes to estimate for SystemsEngineering
include:�Number of logic gates�Number of interfaces
�Examples of size measurements for SoftwareEngineering
include:�Function Points�Lines of Code�Number of requirements
-
Techniques for Meas Pgm - 22version CMMI Conf 2004© 2004 Kasse
Initiatives, LLC
Basic MeasuresBasic Measures -- 22
�Determine effort and cost – use historicaldata or models
�Establish the project’s schedule based on thesize and
complexity estimations
�Include, or at least consider, infrastructureneeds such as
critical computer resources
�Identify risks associated with the cost,resources, schedule,
and technical aspects ofthe project
�Control data (various forms of documentation)required to
support a project in all of its areas.
-
Techniques for Meas Pgm - 23version CMMI Conf 2004© 2004 Kasse
Initiatives, LLC
Basic MeasuresBasic Measures –– 33
�Identify the knowledge and skills needed toperform the project
according to the estimates
�Select and implement methods for providingthe necessary
knowledge and skills�Training (Internal and
External)�Mentoring�Coaching�On-the-job application of learned
skills
�Monitor staffing needs – based on effortrequired and the
necessary knowledge andskills to achieve the defined tasks
-
Techniques for Meas Pgm - 24version CMMI Conf 2004© 2004 Kasse
Initiatives, LLC
Basic MeasuresBasic Measures -- 44
�Involve relevant stakeholders throughout theproduct
lifecycle
�Track technical performance (Completionof activities and
milestones against theschedule Example:�Product components
designed, constructed, unit
tested and integrated�Compare actual milestones completed
vs.
established commitments�Monitor commitments and critical
dependencies against those documented inthe project plan
-
Techniques for Meas Pgm - 25version CMMI Conf 2004© 2004 Kasse
Initiatives, LLC
Basic MeasuresBasic Measures -- 55
�Track quality – Problems/Defects(open/closed by
product/activity)�Problems and defects are direct contributors
to
the amount of rework that must be performed—asignificant cost
factor in development andmaintenance
-
Techniques for Meas Pgm - 26version CMMI Conf 2004© 2004 Kasse
Initiatives, LLC
Effectiveness ofEffectiveness ofProcessesProcesses
-
Techniques for Meas Pgm - 27version CMMI Conf 2004© 2004 Kasse
Initiatives, LLC
Effectiveness ofEffectiveness ofProcessesProcesses
�In addition to defining the processes that wewish to follow on
our project, we need toensure we are following them and we shouldbe
able to determine if the processes areworking for us the way we
expected them to�How well are the processes working?
-
Techniques for Meas Pgm - 28version CMMI Conf 2004© 2004 Kasse
Initiatives, LLC
RequirementsRequirementsManagementManagement ––Effectiveness
ExampleEffectiveness Example
�Number of change requests per monthcompared with the original
number ofrequirements for the project�Critical change
requests�Intermediate change requests�Nice to have change
requests
�Time spent on change requests up until a Y/Ndecision is given
from the Senior Contractgroup
�Number and size of critical change requeststhat arise after the
requirements phase hasbeen completed
-
Techniques for Meas Pgm - 29version CMMI Conf 2004© 2004 Kasse
Initiatives, LLC
Set of StandardSet of StandardOrganizationalOrganizational
ProcessesProcesses
-
Techniques for Meas Pgm - 30version CMMI Conf 2004© 2004 Kasse
Initiatives, LLC
Importance of anImportance of anOrganizational
ViewOrganizational Viewof Processesof Processes
�Builds a common vocabulary�Allows others to anticipate behavior
and be
more proactive in their interactions�Allows the organization to
measure a
controlled set of processes to gain economy ofscale
�Trends can be seen and predictability can beachieved
�Process performance baselines can bedeveloped to support
quantitativemanagement later
-
Techniques for Meas Pgm - 31version CMMI Conf 2004© 2004 Kasse
Initiatives, LLC
OrganizationalOrganizationalMeasurementMeasurementRepositoryRepository
�Develop an organization measurementrepository -
include:�Product and process measures that are related to
the organization’s set of standard processes
�The related information needed to understandand interpret the
measurement data and asses itfor reasonableness and
applicability
�Develop operational definitions for themeasures to specify the
point in the processwhere the data will be collected and for
theprocedures for collecting valid data
-
Techniques for Meas Pgm - 32version CMMI Conf 2004© 2004 Kasse
Initiatives, LLC
Slightly MoreSlightly MoreAdvanced MeasuresAdvanced Measures
-
Techniques for Meas Pgm - 33version CMMI Conf 2004© 2004 Kasse
Initiatives, LLC
Peer ReviewsPeer Reviews
-
Techniques for Meas Pgm - 34version CMMI Conf 2004© 2004 Kasse
Initiatives, LLC
Peer ReviewPeer ReviewMeasuresMeasures
�Optimum Checking Rate (e.g., Number ofpages to be checked per
hour)
�Logging Rate (e.g., Number of major anddefects logged per
hous)
�Number of Major and Minor Defects�Effectiveness - Number of
Major Defects
found in this stage compared to the totalnumber of defects found
so far
-
Techniques for Meas Pgm - 35version CMMI Conf 2004© 2004 Kasse
Initiatives, LLC
TestingTesting
-
Techniques for Meas Pgm - 36version CMMI Conf 2004© 2004 Kasse
Initiatives, LLC
Defects DiscoveredDefects DiscoveredDuring TestingDuring
Testing
�Effectiveness - Number of Major defects foundin a particular
testing phase or instantiation ofa testing phase compared to the
total numberof defects found during testing
�Number of defects projected to escape fromthe current testing
phase
-
Techniques for Meas Pgm - 37version CMMI Conf 2004© 2004 Kasse
Initiatives, LLC
Test CoverageTest Coverage
�Statement coverage measures whether eachexecutable statement is
encountered
�Block coverage is the same as statementcoverage except that the
unit of codemeasured is each sequence of
non-branchingstatements
�Decision coverage measures whether booleanexpressions tested in
control structures (suchas if-statements or
while-statements)evaluated to both true and false
�Condition coverage measures the true or falseoutcome of each
boolean sub-expression
-
Techniques for Meas Pgm - 38version CMMI Conf 2004© 2004 Kasse
Initiatives, LLC
Test CoverageTest Coverage -- 22
�Function Coverage�Measures whether you invoked each function
or
procedure�Call Coverage
�Measures whether you executed each function call�Based on the
idea that faults commonly occur in
interfaces between modules�Loop Coverage
�Measures whether you executed each loop bodyzero times, exactly
once, and more than once
�Relational Operator Coverage�Measures whether boundary
situations occur with
relational operators (=)
-
Techniques for Meas Pgm - 39version CMMI Conf 2004© 2004 Kasse
Initiatives, LLC
Risk ManagementRisk Management
-
Techniques for Meas Pgm - 40version CMMI Conf 2004© 2004 Kasse
Initiatives, LLCImpact
Prob
abili
ty
Time
Prioritizing RisksPrioritizing Risks
�Priorities are established by merging theinformation available
into a three dimensionaltable
-
Techniques for Meas Pgm - 41version CMMI Conf 2004© 2004 Kasse
Initiatives, LLC
PrioritizingPrioritizing -- 22
Catastrophic
Critical
Marginal
Negligible
Probable Possible Unlikely
Top
High
Serious
Medium
High
Serious
Medium
Low
Serious
Medium
Low
Bottom
Probability:
Impact:
-
Techniques for Meas Pgm - 42version CMMI Conf 2004© 2004 Kasse
Initiatives, LLC
PrioritizingPrioritizing -- 33
Immediate Short Medium
1
2
3
4
2
3
4
5
3
4
5
6
5 6 7
6 7 8
Top
High
Serious
Medium
Low
Bottom
Long
4
5
6
7
8
9
TimeFrame:
-
Techniques for Meas Pgm - 43version CMMI Conf 2004© 2004 Kasse
Initiatives, LLC
Quantitative ProjectQuantitative ProjectManagementManagement
-
Techniques for Meas Pgm - 44version CMMI Conf 2004© 2004 Kasse
Initiatives, LLC
QuantitativeQuantitativeManagementManagementConceptsConcepts
�Quantitative Management is tied to theorganization’s strategic
goals for productquality, service quality, and
processperformance
�When higher degrees of quality andperformance are demanded, the
organizationand projects must determine if they have theability to
improve the necessary processes tosatisfy the increased demands
-
Techniques for Meas Pgm - 45version CMMI Conf 2004© 2004 Kasse
Initiatives, LLC
QuantitativeQuantitativeManagementManagementConceptsConcepts --
22
�Achieving the necessary quality and processperformance
objectives requires stabilizing theprocesses that contribute most
to theachievement of the objectives
�Assuming the technical requirements can bemet, the next
decision is to determine if it iscost effective
�Reducing process variation is an importantaspect to
quantitative management
-
Techniques for Meas Pgm - 46version CMMI Conf 2004© 2004 Kasse
Initiatives, LLC
Establish the Organization’sEstablish the Organization’sProcess
PerformanceProcess PerformanceBaselineBaseline
�Process performance is a measure of theactual results achieved
by following a process
�Measure the organization’s processperformance for the
organization’s set ofstandard processes at various
levelsincluding:�Individual processes (e.g., test case
inspection
element)�Sequence of connected processes�Processes for the
complete lifecycle�Processes for developing individual work
products
-
Techniques for Meas Pgm - 47version CMMI Conf 2004© 2004 Kasse
Initiatives, LLC
Establish the Organization’sEstablish the Organization’sProcess
PerformanceProcess PerformanceBaselineBaseline -- 22
�There may be several process performancebaselines to
characterize performance forsubgroups of the organization –
Examplesinclude:�Product Line�Application domain�Complexity�Team
size�Work product size
-
Techniques for Meas Pgm - 48version CMMI Conf 2004© 2004 Kasse
Initiatives, LLC
Steps to Establish Project’sSteps to Establish Project’sQuality
and ProcessQuality and ProcessPerformance ObjectivesPerformance
Objectives
�Identify the quality and process performanceneeds and
priorities of the customer, endusers, and other relevant
stakeholders�Reliability�Maintainability and
Expandability�Usability�Development cycle time
�Determine how quality and processperformance will be
measured
-
Techniques for Meas Pgm - 49version CMMI Conf 2004© 2004 Kasse
Initiatives, LLC
Steps to Establish Project’sSteps to Establish Project’sQuality
and ProcessQuality and ProcessPerformance ObjectivesPerformance
Objectives -- 22
�Define and document measurable quality andprocess performance
objectives for the project
�Examples of Quality Objectives�Mean time between
failures�Critical resource utilization
�Examples of Process Performance Objectives�Percentage of
defects removed by type of
verification activity�Defect escape rates�Number and density of
defects (by severity) found
during the first year following product delivery�Rework time as
a percentage of total project life-
cycle time
-
Techniques for Meas Pgm - 50version CMMI Conf 2004© 2004 Kasse
Initiatives, LLC
Steps to Establish Project’sSteps to Establish Project’sQuality
and ProcessQuality and ProcessPerformance ObjectivesPerformance
Objectives -- 33
�Establish objectives for each life-cycle stageto track progress
and predict achievement ofproject’s objectives
�Establish criteria to identify whichsubprocesses are the main
contributors toachieving the identified quality and
processperformance objectives and for whichpredictable performance
is important
-
Techniques for Meas Pgm - 51version CMMI Conf 2004© 2004 Kasse
Initiatives, LLC
MeasuresMeasuresand Analyticand AnalyticTechniquesTechniques
-
Techniques for Meas Pgm - 52version CMMI Conf 2004© 2004 Kasse
Initiatives, LLC
Select Measures andSelect Measures andAnalytic
TechniquesAnalytic Techniques
�Identify the measures that are appropriate forstatistical
management
�Examples of subprocess control measuresinclude:�Ratios of
estimated to measured values of the
planning parameters�Coverage and efficiency of work product
inspections�Test coverage and efficiency�Reliability,
Maintainability, and Expandability�Percentage of the total defects
inserted or found in
the different stages of the lifecycle
-
Techniques for Meas Pgm - 53version CMMI Conf 2004© 2004 Kasse
Initiatives, LLC
�There are a number of quantitative tools consideredto be
applicable to statistical process or quality control:�Quantifying
and Predicting Process Performance
� Control Charts� Histograms
�Cause and Effect Relationships� Scatter diagrams� Run charts�
Cause-and-effect (fishbone) diagrams� Bar charts� Pareto charts�
Interrelationship Diagraph� Kiviat Diagram
Quantitative ToolsQuantitative Tools
-
Techniques for Meas Pgm - 54version CMMI Conf 2004© 2004 Kasse
Initiatives, LLC
Control ChartsControl Charts
�Control charts – techniques for quantifyingprocess
behavior�Classical control charts have a centerline and
control limits on both sides of the centerline�Both the
centerline and the limits represent
estimates that are calculated from a set ofobservations
collected while the process is running
�The centerline and control limits cannot beassigned arbitrarily
as they are intended to showwhat the process can actually do
Florac, W.A. & Carleton, A.D. Measuring the Software
ProcessAddison-Wesley, 1999
-
Techniques for Meas Pgm - 55version CMMI Conf 2004© 2004 Kasse
Initiatives, LLCNumerical data taken
in time sequence
UpperControlLimit(UCL)
LowerControlLimit(LCL)
Upper andLower
Control Limitsrepresent the
natural variationIn the process
METRIC:PROCESS CONTROL CHART TYPE:
A point above or below thecontrol lines suggests that the
measurement has a specialpreventable or removable cause
Plotted points are eitherindividual measurements or the
means of small groups ofmeasurements
The chart is used for continuousand time control of the
process
and prevention of causes
The chart is analyzed usingstandard Rules to define the
control status of the process
Datarelating to
the process
Center Line (CL)(Mean of data used toset up the chart)
Statistical Methods for Software QualityAdrian Burr – Mal Owen,
1996
-
Techniques for Meas Pgm - 56version CMMI Conf 2004© 2004 Kasse
Initiatives, LLC
HistogramsHistograms
�Histograms�Allows a project team to take measurement data
and display the distribution of the observed values�Show the
frequencies of events that have occurred
in ways that make it easy to compare distributionsand see
central tendencies
�Illustrates quickly the underlying distribution of thedata
�Helps indicate if there has been a change in theprocess
Florac, W.A. & Carleton, A.D. Measuring the Software
ProcessAddison-Wesley, 1999
-
Techniques for Meas Pgm - 57version CMMI Conf 2004© 2004 Kasse
Initiatives, LLC
HistogramsHistograms -- 22
�Histograms - continued�Provides useful information for
predicting future
performance of the process�Helps answer the question “Is the
process capable
of meeting my customers requirements?”�Histograms are created by
grouping the results into
“cells” and then counting the number in each cell
Florac, W.A. & Carleton, A.D. Measuring the Software
ProcessAddison-Wesley, 1999
-
Techniques for Meas Pgm - 58version CMMI Conf 2004© 2004 Kasse
Initiatives, LLC
Num
ber o
f Day
s
Product – Service Staff Hours
20
56
181614121086420
32 5452504846444240383634
HistogramsHistograms
-
Techniques for Meas Pgm - 59version CMMI Conf 2004© 2004 Kasse
Initiatives, LLC
Applying StatisticalApplying StatisticalMethods toMethods
toUnderstandUnderstand
VariationVariation
-
Techniques for Meas Pgm - 60version CMMI Conf 2004© 2004 Kasse
Initiatives, LLC
Understanding VariationUnderstanding Variation
�Understanding variation is achieved bycollecting and analyzing
process and productmeasures so that special causes of variationcan
be identified and addressed to achievepredictable performance
�All characteristics of processes and productsdisplay variation
when measured over time
�Variation may be due to�Special or “assignable” causes of
variation�Natural or common causes
-
Techniques for Meas Pgm - 61version CMMI Conf 2004© 2004 Kasse
Initiatives, LLC
SummarySummary
�Techniques for Establishing a MeasurementProgram
include:�Clearly define the need for a measurement
program�Establish a measurement initiative with objectives
that are aligned with established information needsand business
objectives
�Ensure basic measures are included for planning,tracking, and
taking corrective action as necessary
�Incorporate process effectiveness measures�Establish
organizational standard processes
-
Techniques for Meas Pgm - 62version CMMI Conf 2004© 2004 Kasse
Initiatives, LLC
SummarySummary -- 22
�Establish and utilize measures such as peer reviewmeasures,
testing measures, and risk managementmeasures
�Evolve into project management based on aquantitative
understanding of the organization’sand project’s defined
processes
-
Techniques for Meas Pgm - 63version CMMI Conf 2004© 2004 Kasse
Initiatives, LLC
Tim KasseTim KasseKasse Initiatives LLCKasse Initiatives LLC
PMB 293PMB 2931900 Preston Road, Suite 2671900 Preston Road,
Suite 267
Plano, Texas 75093Plano, Texas 75093U.S.A.U.S.A.
Tel: +1 972 987 7606Tel: +1 972 987 7606
E-mail: [email protected]
Web Site: www.kasseinitiatives.com
Evolving a Measurement Program f...WelcomeMeasurement�Is It
Really Necessa...At Project Start, Do You Know…?Can it be done?How
long will it take?How much will it cost?How many people will it
take?What is the risk?What are the tradeoffs?How many errors will
there be?
What Do You Know Now?How much does your current devel...How much
value does each piece o...What would the impact be of dele...What
activities contribute the m...Have you tried to improve the
cu...What changes in cost/value resul...
What Do You Know Now? - 2What processes represent the gre...How
would you quantify the value...Do you really want to know
where...What value do you think you are ...How much is the
knowledge of you...
Measurement �and AnalysisMeasurement and Analysis Overvie...A
measurement initiative involve...Specifying the objectives of
mea...Defining the measures to be used...Implementing the
collection, sto...Providing objective results that...
Establish Measurement ObjectivesMeasurement objectivesDocument
the purposes for which ...Specify the kinds of actions
tha...Continually ask the question – w...Involve the end users of
the mea...Maintain traceability of the pro...
Specify MeasuresIdentify candidate measures base...Define how
data can and will be ...Examples of Commonly Used Derive...Defect
densityPeer review coverageReliability measures
Specify Data Collection and Stor...Specify how to collect and
store...Make explicit specifications of ...Develop procedures for
ensuring ...Ensure that the data is stored s...
Specify Analysis ProceduresDefine the analysis procedures
i...Ensure that the results that wil...Collecting data for the sake
of ...Showing how it can be used to ma...
Collect �Measurement DataCollect the measurement data as
...Generate data for derived measur...Perform integrity checks as
clos...
Analyze the Measurement DataConduct the initial
analysesInterpret the results and make p...Conduct additional
measurement a...Review the initial results with ...Prevents
misunderstandings and r...Improve measurement definitions,...
Store the Measurement Data and A...The stored information should
co...Understand the measuresAssess them for reasonableness a...The
stored information should al...Enable the timely and cost
effec...Provide sufficient context for i...
Communicate the Measurement Resu...Keep the relevant
stakeholders u...Follow up with those who need to...Increases the
likelihood that th...Assist the relevant stakeholders...
Measurement and Analysis GroupConsider creating a measurement
...
Measurement and Analysis ToolsIncorporate tools used in
perfor...Statistical packagesDatabase packagesSpreadsheet
programsGraphical packagesPackages that support data colle...
Measurement and Analysis Trainin...Provide training to all
people w...Data collection, analyses, and r...Measurement
toolsGoal-Question-Metric ParadigmHow to establish measureshow to
determine efficiency and ...Quality factors measures (e.g.,
...Basic and advanced statistical t...
Basic MeasuresBasic MeasuresEstimate Size and/or Complexity
...Examples of attributes to estima...Number of logic gatesNumber
of interfacesExamples of size measurements fo...Function
PointsLines of CodeNumber of requirements
Basic Measures - 2Determine effort and cost – use ...Establish
the project’s schedule...Include, or at least consider,
i...Identify risks associated with t...Control data (various forms
of d...
Basic Measures – 3Identify the knowledge and skill...Select and
implement methods for...Training (Internal and
External)MentoringCoachingOn-the-job application of
learne...Monitor staffing needs – based o...
Basic Measures - 4Involve relevant stakeholders th...Track
technical performance (Com...Product components designed,
con...Compare actual milestones comple...Monitor commitments and
critical...
Basic Measures - 5Track quality – Problems/Defects...Problems
and defects are direct ...
Effectiveness of ProcessesEffectiveness of ProcessesIn addition
to defining the proc...How well are the processes worki...
Requirements Management – Effect...Number of change requests per
mo...Critical change requestsIntermediate change requestsNice to
have change requestsTime spent on change requests up...Number and
size of critical chan...
Set of Standard Organizational P...Importance of an
Organizational ...Builds a common vocabularyAllows others to
anticipate beha...Allows the organization to measu...Trends can be
seen and predictab...Process performance baselines ca...
Organizational Measurement Repos...Develop an organization
measurem...Product and process measures tha...The related
information needed t...Develop operational definitions ...
Slightly More Advanced MeasuresPeer ReviewsPeer
Review�MeasuresOptimum Checking Rate (e.g., Num...Logging Rate
(e.g., Number of ma...Number of Major and Minor
Defect...Effectiveness - Number of Major...
TestingDefects Discovered During Testin...Effectiveness - Number
of Major ...Number of defects projected to e...
Test CoverageStatement coverage measures whet...Block coverage
is the same as st...Decision coverage measures wheth...Condition
coverage measures the ...
Test Coverage - 2Function CoverageMeasures whether you invoked
eac...Call CoverageMeasures whether you executed ea...Based on the
idea that faults co...Loop CoverageMeasures whether you executed
ea...Relational Operator CoverageMeasures whether boundary
situat...
Risk ManagementPrioritizing RisksPriorities are established by
me...
Prioritizing - 2Prioritizing - 3Quantitative Project
ManagementQuantitative Management ConceptsQuantitative Management
is tied ...When higher degrees of quality a...
Quantitative Management Concepts...Achieving the necessary
quality ...Assuming the technical requireme...Reducing process
variation is an...
Establish the Organization’s Pro...Process performance is a
measure...Measure the organization’s proce...Individual processes
(e.g., test...Sequence of connected processesProcesses for the
complete lifec...Processes for developing individ...
Establish the Organization’s Pro...There may be several process
per...Product LineApplication domainComplexityTeam sizeWork product
size
Steps to Establish Project’s Qua...Identify the quality and
process...ReliabilityMaintainability and
Expandabilit...UsabilityDevelopment cycle timeDetermine how quality
and proces...
Steps to Establish Project’s Qua...Define and document
measurable q...Examples of Quality ObjectivesMean time between
failuresCritical resource utilizationExamples of Process
Performance ...Percentage of defects removed by...Defect escape
ratesNumber and density of defects (b...Rework time as a percentage
of t...
Steps to Establish Project’s Qua...Establish objectives for each
li...Establish criteria to identify w...
Measures �and Analytic Technique...Select Measures and Analytic
Tec...Identify the measures that are a...Examples of subprocess
control m...Ratios of estimated to measured ...Coverage and
efficiency of work ...Test coverage and efficiencyReliability,
Maintainability, an...Percentage of the total defects ...
Quantitative ToolsThere are a number of quantitati...Quantifying
and Predicting Proce...Control ChartsHistogramsCause and Effect
RelationshipsScatter diagramsRun chartsCause-and-effect (fishbone)
diag...Bar chartsPareto chartsInterrelationship DiagraphKiviat
Diagram
Control ChartsControl charts – techniques for ...Classical
control charts have a ...Both the centerline and the limi...The
centerline and control limit...
Slide 55HistogramsHistogramsAllows a project team to take
me...Show the frequencies of events t...Illustrates quickly the
underlyi...Helps indicate if there has been...
Histograms - 2Histograms - continuedProvides useful information
for ...Helps answer the question “Is th...Histograms are created by
groupi...
HistogramsApplying Statistical Methods to ...Understanding
VariationUnderstanding variation is achie...All characteristics of
processes...Variation may be due toSpecial or “assignable” causes
o...Natural or common causes
SummaryTechniques for Establishing a Me...Clearly define the
need for a me...Establish a measurement initiati...Ensure basic
measures are includ...Incorporate process effectivenes...Establish
organizational standar...
Summary - 2Establish and utilize measures s...Evolve into
project management b...
Tim Kasse�Kasse Initiatives LLC�...