THE INCREMENTAL COMMITMENT SPIRAL MODEL PROCESS PATTERNS FOR RAPID‐FIELDING PROJECTS by Supannika Koolmanojwong A Dissertation Presented to the FACULTY OF THE USC GRADUATE SCHOOL UNIVERSITY OF SOUTHERN CALIFORNIA In Partial Fulfillment of the Requirements for the Degree DOCTOR OF PHILOSOPHY (COMPUTER SCIENCE) December 2010 Copyright 2010 Supannika Koolmanojwong
153
Embed
THE INCREMENTAL COMMITMENT SPIRAL MODEL …csse.usc.edu/TECHRPTS/PhD_Dissertations/files/... · · 2010-12-21Chapter 3 : Research ... Faster Time to Market Projects in Software
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
THE INCREMENTAL COMMITMENT SPIRAL MODEL PROCESS PATTERNS
FOR RAPID‐FIELDING PROJECTS
by
Supannika Koolmanojwong
A Dissertation Presented to the FACULTY OF THE USC GRADUATE SCHOOL UNIVERSITY OF SOUTHERN CALIFORNIA
In Partial Fulfillment of the Requirements for the Degree DOCTOR OF PHILOSOPHY (COMPUTER SCIENCE)
December 2010
Copyright 2010 Supannika Koolmanojwong
ii
Acknowledgements
During my PhD journey, I have encountered numerous successes and many failures.
I am so fortunate to receive countless encouragements, opportunities, friendships, and
trusts from many people that I am indebted to.
First and foremost, I am grateful and thankful to my advisor, Prof. Barry Boehm, for
his tremendous supports and for believing in my PhD study feasibility evidence. I would like
to thank my dissertation committee members: Prof. Stan Settles, Prof. Nenad Medvidovic,
Dr. Robert Neches and Dr. Rick Selby for their valuable feedback and numerous constructive
critiques.
My journey would not have been fun without a good company. I would like to thank
for the warmest supports from Computer Science department and CSSE family: Pongtip
Aroonvatanaporn, Jo Ann Lane, Vu Nguyen, Qi Li, Aaron Chang, Thomas Tan, Julie Sanchez
and Monvorath Phongpaibul.
I would like to express my gratitude to Dr. Fred Hadaegh for help shaping up my
journey.
My dissertation would not have been possible without USC Software Engineering
course students, clients, teaching assistants, and graders. Thank you for the data and
participation in the study.
I also find myself lucky to have colleagues and friends from Assumption University,
Thailand, who tagged along and comforted me in various trips of my life.
The endless thanks go to my husband Sohrab Mobasser, and his family for
continuously giving me a hug, a hand, hope, happiness, humor, and huge amount of love that
iii
I never thought I could get from anyone. I also have to thank Tooleh for her unconditional
love. She is the best buddy and the best audience I have ever had.
My journey could not be started and could not be completed without my devoted
mom and my brothers. Your love and your caring give me strengths to overcome all
obstacles.
Lastly, I am indebted to my father who inspired me to start my journey. I know you
are proud of me. I am honored to be your daughter and I miss you very single day.
iv
Table of Contents
Acknowledgements ii
List of Figures vi
List of Tables ix
Abbreviations xi
Abstract xiii
Chapter 1 : Introduction 1 1.1. Motivation 1 1.2. Research Questions 3 1.3. Research Contribution 4 1.4. Organization of Dissertation 4
Chapter 2 : Background and Related Work 6 2.1. The Incremental Commitment Spiral Model (ICSM) 6 2.2. Definitions of Non‐Developmental Items and Net‐Centric Services 10 2.3. Related Software Development Processes for Rapid‐Fielding Projects 11 2.3.1. ICSM, Lean Principles, and Agile Principles 11 2.3.2. CBA Process Decision Framework 13 2.3.3. COTS Interoperability Framework 15 2.4. Process Patterns of Software Development Project 16 2.5. USC Software Engineering Course 19 2.6. Electronic Process Guide Generator Tools 19 2.6.1. Spearmint 20 2.6.2. Little‐JIL 20 2.6.3. IBM Rational Method Composer 21
Chapter 3 : Research Methodology 23 3.1. Research Timeline 23 3.2. Hypotheses 26 3.3. Threats to Validity 27
Chapter 4 : Process Patterns in Rapid‐Fielding Projects 29 4.1. Different Opportunities or Risk Patterns in Rapid‐Fielding Project 29 4.2. Differences between NDI and Services 30 4.3. Rapid‐Fielding Project Characteristics 32 4.4. NDI/NCS Process Decision Framework 33 4.5. Work Breakdown Structure in Rapid‐Fielding Process 47 4.6. Validation Results 52
v
4.6.1. Client Satisfaction 52 4.6.2. Faster Time to Market Projects in Software Engineering Class 54 4.6.3. Conclusion 55
Chapter 5 : Decision Criteria of Process Deployment 56 5.1. Process Decision Driver 56 5.2. Discussion with Field Experts 64 5.3. Process Adoption 65 5.3.1. Incidents of Process Selection and Direction Changes 66 5.3.2. Incidents of Wrong Process Selection 69 5.3.3. Conclusion 73
Chapter 6 : The ICSM Electronic Process Guide 75 6.1. Representation of Process Elements and Their Relationship 76 6.2. Process Representation 80 6.3. Discussion 83 6.3.1. Comparison Software Process Modeling Tools 83 6.3.2. ICSM EPG Analysis 87 6.3.3. Effort Comparison between EPG and non‐EPG 89 6.3.4. Qualitative Feedback from students’ survey results 90 6.3.5. Quantitative Survey Results on ICSM EPG 91 6.3.6. Quantitative Survey Results regarding PDF‐guideline vs EPG 92 6.3.7. Conclusions 93
Chapter 7 : Conclusion and Future Work 95 7.1. General Conclusions 95 7.2. Summary of Contributions 95 7.3. Future Work 96
References 97
Appendices Appendix A : Characteristics of each ICSM Process Pattern 101 Appendix B : Key Activities for Each Process Pattern of the ICSM 103 Appendix C : CSCI 577 projects 104 Appendix D : Projects that deliver in one‐semester 108 Appendix E : Student General Information Questionnaire 109 Appendix F : Effort Category 113 Appendix G : ICSM EPG: Roles, Activities, Work products, and Delivery Process 114 Appendix H : Example of Decision driver 119 Appendix I : Client Feedback Form – CSCI 577a 125 Appendix J : Client Feedback Form – CSCI577b 127 Appendix K : Qualitative Interview Form 129 Appendix L : Results of ICSM EPG Survey 131 Appendix M : Software Process Guidelines Survey 132 Appendix N : Results of Software Process Guidelines Survey 138
vi
List of Figures
Figure 1: Net Centric Services Usage in USC Software Engineering Class 2
Figure 2: Statistics about Mashup created and listed at ProgrammableWeb.com 3
Figure 3: Overview of the Incremental Commitment Spiral Model 8
Figure 4: Phased View of the Generic Incremental Commitment Spiral Model Process 9
Figure 5: CBA Process Decision Framework 15
Figure 6: COTS Interoperability Framework 16
Figure 7: An example page of a Spearmint‐based process guide 20
Figure 8: Tree‐like structure diagram of Little‐JIL 21
Figure 9: An example of an IBM RMC page 22
Figure 10: Dissertation Development Approach 24
Figure 11: Dissertation Timeline 24
Figure 12: Different Opportunities and Risk Patterns yield Different Processes 30
Figure 13: NDI Process Decision Framework 34
Figure 14: Identify OC&Ps and explore alternatives 35
Figure 15: Assess NDI/Services Candidates 40
Figure 16: Check Interoperability 43
Figure 17: Tailor a single NDI/ Service 43
Figure 18: Tailor multiple NDI/Services 44
Figure 19: Develop Glue Code 46
Figure 20: Delivery Process in Exploration Phase 47
Figure 21: Delivery Process in Valuation phase 49
vii
Figure 22: Delivery Process in Foundations phase 50
Figure 23: Development phase ‐ Construction increment 50
Figure 24: Delivery Process in Development phase ‐ Trnaition increment 51
Figure 25: Translating Rating Scales to Process Template 62
Figure 26: An Example of Using Decision Drivers to map with an NDI‐Intensive Project 63
Figure 27: Example of Process Pattern Selection 64
Figure 28: Result of right process pattern selection 66
Figure 29: Results of incorrect process selection due to unclear project scope 67
Figure 30: Result of incorrect process selection due to minor changes 68
Figure 31: Results of Process Re‐Selection due to available NCS 68
Figure 32: Result of infeasible project 69
Figure 33: The Incremental Commitment Spiral Model Electronic Process Guide 75
Figure 34: View of Operational Concept Development Practice 78
Figure 35 Defining Relationship between Process Elements 79
Figure 36: Relationship between roles, tasks, and work product 79
Figure 37: Overview of the ICSM EPG 80
Figure 38: Work Breakdown Structure for Architected Agile Process 82
Figure 39: Activity Diagram of an Explore Current System Activity 82
Figure 40: Student Background Information Survey ‐ Page 1 109
Figure 41: Student Background Information Survey ‐ Page 2 110
Figure 42: Student Background Information Survey ‐ Page 3 111
Figure 43: Student Background Information Survey ‐ Page 4 112
Figure 44: Welcome Page of ICSM EPG 114
Figure 45: List of Roles in ICSM EPG 115
viii
Figure 46: List of Practices in ICSM EPG 115
Figure 47: Practice Page in ICSM EPG 116
Figure 48: A task page in ICSM EPG 117
Figure 49: A role and responsibilities page in ICSM EPG 117
Figure 50: A delivery process page in ICSM EPG 118
Figure 51: list of work products in ICSM EPG 118
Figure 52: Architected Agile Process Pattern Template 119
Figure 53: Use Single NDI Process Pattern Template 120
Figure 54: NDI‐Intensive Process Pattern Template 120
Figure 55: Services‐Intensive Process Pattern Template 121
Figure 56: An Architected Agile team with Architected Agile Decision Pattern 121
Figure 57: An Architected Agile team with Use Single NDI Decision Pattern 122
Figure 58: An Architected Agile team with NDI‐Intensive Decision Pattern 122
Figure 59: An Architected Agile team with Services‐Intensive Decision Pattern 122
Figure 60: An NDI‐intensive team with NDI‐Intensive Decision Pattern 123
Figure 61: An NDI‐intensive team with Architected Agile Decision Pattern 123
Figure 62: An NDI‐intensive team with Use Single NDI Decision Pattern 124
Figure 63: An NDI‐intensive team with Services‐Intensive Decision Pattern 124
ix
List of Tables
Table 1: Number of NDI/ NCS used in Software Engineering Class 2
Table 2: Comparison of the Core Concepts of ICSM, Lean, and Agile Principles 12
Table 3: Twelve ICSM Process Patterns 17
Table 4: Software Engineering Processes Used in Software Enmgineering class 25
Table 5: Number of teams in each process pattern 25
Table 6: Differences between NDI and Net‐Centric Services 31
Table 7: Differences between NDI and NCS 32
Table 8: Characteristics of the Risk‐Driven Process Patterns of the ICSM 33
Table 9: Results of Client Satisfaction Document from 2005 ‐ 2009 52
Table 10: Analysis results of Points lost on Client satisfaction 53
Table 11: Comparison of Point Lost from Client Satisfaction 53
Table 12: Percentage of teams that deliver Faster Time‐To‐Market projects 54
Table 13: Process Decision Drivers 56
Table 14: Analysis on teams with incorrect process patterns 72
Table 15: Summary of team performance on inappropriate process selection 73
Table 16: Comparison of effort between paper‐based guidelines and the ICM EPG 90
Table 17: Results of ICSM EPG Survey 92
Table 18: Results of PDF‐guidelines Survey 93
Table 19: Comparison between the EPG and PDF‐Guidelines 93
Table 20: Characteristics of each ICSM Process Pattern 101
Table 21: Key Activities for each process pattern 103
x
Table 22: List of Projects and their processes in Fall 2005 104
Table 23: List of Projects and their processes in Fall 2006 105
Table 24: List of Projects and their processes in Fall 2007 106
Table 25: List of Projects and their processes in Fall 2008 107
Table 26: List of Projects and their processes in Fall 2009 107
Table 27: List of one‐semester projects 108
Table 28: Effort Categories in Effort Reporting System 113
Table 29: Results of ICSM EPG Survey 131
Table 30: Results of Software Process Guidelines Survey 138
xi
Abbreviations
CBA COTS‐Based Application
CBD COTS‐Based Development
COCOMO II Constructive Cost Model
COCOTS Constructive Commercial Off‐the‐Shelf Cost Model
COTS Commercial‐Off‐The‐Shelf
DART Distributed Assessment of Risks Tool
EPG Electronic Process Guide
FED Feasibility Evidence Description
GUI Graphical User Interface
ICSM Incremental Commitment Spiral Model
IIV&V Integrated, Independent Verification and Validation
LeanMBASE Lean Model‐Based (System) Architecting and Software Engineering
LCP Life Cycle Plan
LSI Large‐Scale Integration
MS Master of Science
NCS Net‐Centric Services
NDI Non‐Developmental Item
OCD Operational Concept Description
OC&P Objective, Constraint, and Priority
PDL Process Definition Language
PML Process Modeling Tool
xii
QMP Quality Management Plan
RMC Rational Method Composer
RUP Rational Unified Process
SEI Software Engineering Institute
SID Supporting Information Document
SOA Service‐Oriented Architecture
SSAD System and Software Architecture Description
SSRD System and Software Requirements Description
USC University of Southern California
UML Unified Modeling Language
V&V Verification and Validation
xiii
Abstract
To provide better services to customers and not to be left behind in a competitive business
environment, a wide variety of ready‐to‐use software and technologies are available for
one to grab and go and build up software systems at a very fast pace. Rapid fielding plays a
major role in developing software systems to provide a quick response to the organization.
This research investigates the appropriateness of current software development processes
and develops new software development process guidelines, focusing on four process
patterns: Use single NDI, NDIintensive, Servicesintensive, and Architected Agile.
Currently, there is no single software development process model that is applicable to all
four process patterns, but the Incremental Commitment Spiral Model (ICSM) can help a new
project converge on a process that fits the project process scenario. The output of this
research has been implemented as an Electronic Process Guide for USC Software
Engineering students to use as a guideline to develop real‐client Software Engineering
course projects. An empirical study has been conducted to assess the suitability of the
newly developed process as compared to results data from previous course projects.
Subject to sources of variability in the nature of the projects, the assessment confirmed that
the process selection guidelines led project teams to choose the most appropriate process
pattern, and that the performance of the project teams choosing inappropriate processes
produced less satisfactory results.
1
Chapter 1 : Introduction
1.1. Motivation
The growing diversity of software systems (requirements‐driven, NDI‐driven,
services‐driven, learning‐driven, qualities‐driven, systems of systems) has made it clear
that there are no one‐size‐fits‐all processes for the full range of software systems. Some
process models are being developed that provide specific evidence‐based and risk‐based
decision points. One of the most thoroughly elaborated of these models is the Incremental
Commitment Spiral Model (ICSM). The ICSM with its risk‐driven nature and its process
decision table can help new projects converge on a process that fits their process drivers
and circumstances. To select and follow the appropriate process pattern should help the
projects conclude more quickly and efficiently. A set of decision criteria and the ICSM
decision points have been defined on a general‐experience basis. But quantitative evidence
has been lacking on the ability of the criteria to produce a viable process decision early in
the life cycle.
The USC real‐client MS‐level team projects provide a significant number of projects
that are suitable to four of the ICSM process patterns: Architected Agile, Use Single NDI,
NDIintensive, and Servicesintensive. The first process pattern, Architected Agile
exemplifies a scalable balance between plan‐driven and agile‐driven approaches [Boehm
2004]. The second process pattern, NDIintensive is an alternative for software developers
to reduce development time and cost while increasing software quality and productivity via
software reuse [Basili 2001; Li 2006]. The third process pattern, Servicesintensive is very
similar to NDI‐intensive but instead of selecting certain NDI products, the development
2
team considers to deploy available web services. Lastly, Use single NDI provides an option
of a ready‐to‐use product either as a complete project solution or as a partial development
solution.
Table 1: Number of NDI/ NCS used in Software Engineering Class
For the small e‐services projects developed in the software engineering project
course, four of the 12 process patterns of the ICSM predominate:
Architected Agile ‐ For a less than 80 agile‐ready‐people team and a fairly mature
technology project, agile methods can be scaled up using an Architected Agile approach,
emphasizing early investment in a change‐prescient architecture and all success‐critical‐
stakeholders team building [Boehm 2004]. The Valuation and Foundations phases can be
brief. A scrum of Scrums approach can be used in the Development phase.
Use Single NDI – When an appropriate NDI (COTS, open source, reuse library,
customer‐furnished package) solution is available, it is an option to either use the NDI or
develop, perhaps, a better version by oneself, or outsource such a development, which
generally incurs more expense and takes longer to begin capitalizing on its benefits. On the
other hand, an NDI may come with high volatility, complexity, or incompatibility. Major
effort will then be spent on appraising the NDI.
NDIIntensive – An NDI‐Intensive system is a system in which 30% of end‐user
functionality is provided by NDI [Yang 2006]. A great deal of attention goes into appraising
the functionality and interoperability of NDI, effort spent on NDI tailoring and Integration,
and NDI upgrade synchronization and evolution [Li 2006; Morisio 2000].
ServicesIntensive – Net Centric Services support community service organizations
in their online information processing services such as donation, communication or their
special interest group activities such as discussion boards, file sharing, and cloud
computing. Similar to NDI‐Intensive, the focus is on appraising the functionality of the
available services and tailoring to meet needs.
19
2.5. USC Software Engineering Course
In the keystone two‐semester team project graduate software engineering course
sequence CS577ab [USC CSCI577] at USC, students learn through experience how to use
good software engineering practices to develop software systems from the Exploration
Phase to the Operation Phase, all within a 24‐week schedule. Six on‐campus and two off‐
campus students team up to develop real‐client software system products. Based on the
nature of the course projects, all teams follow the ICSM Exploration Phase guidelines to
determine their most appropriate process pattern and could switch to other process
pattern as appropriate. The teams adopt the selected process pattern and follow its
guidelines until the end of the end of the project lifecycle. Most of the clients are
neighborhood non‐profit organizations, small businesses or USC departments. Examples of
the projects are an accounting system, an art gallery web portal, a theatre script online
database, and an EBay search bot. Because of the semester break between fall and spring
semester, for the Software Engineering class, a short Rebaselined Foundations phase is
added to accommodate the possible changes.
2.6. Electronic Process Guide Generator Tools
Software processes are complex and difficult for process users to understand and
follow. To capture the software process, process authors can either use the formal
representation called process definition language (PDL) such as Little‐JIL or use process
modeling tool such as Spearmint or IBM Rational Method Composer to specify the process
and develop electronic process guide (EPG). Spearmint, Little‐JIL, and IBM Rational Method
Composer are selected as the candidates to model the LeanMBASE or ICSM, and convert the
processes and represent the process content in the electronic version.
20
2.6.1. Spearmint
Spearmint is an integrated environment for modeling, analyzing, and measuring process
[Becker 1999]. It provides four different views of a process model, which are product flow
view, properties view, decomposition view, and textual view. The product flow view
provides graphical representations illustrating the relationship between artifacts, activities,
roles, and tools, while the properties view represents the detail of a process model element
such as agent/role, activity, artifact and tool. An example of an electronic process guide
generated from Spearmint is shown at Figure 7.
Figure 7: An example page of a Spearmintbased process guide
2.6.2. LittleJIL
Little‐JIL is a graphical agent coordination language developed by LASER (Laboratory for
Advanced Software Engineering Research) of University of Massachusetts, Amherst
21
(UMASS). To help process engineer and process performer to better understand the system
and its activities, Little‐JIL is used to capture system’s process and describe them in clear
graphical view (Figure 8). Four principles of Little‐JIL are simplicity, expressiveness,
precision, and flexibility [Cass 2000]. Visual‐JIL is an eclipse plug‐in that the LASER team
developed by using the Little‐JIL language. Visual‐JIL converts the Little‐JIL program, which
is XML format files, into tree‐like graphical diagram that represents steps, sub steps,
responsible agents, produced artifacts, pre/post conditions, cardinality, dependency,
exception, project resources, etc.
Figure 8: Treelike structure diagram of LittleJIL
2.6.3. IBM Rational Method Composer
The IBM Rational Method Composer (RMC), as shown in Figure 9, is a process
management platform that integrates best practices of the Rational Unified Process (RUP)
22
framework and provides process content library, delivery processes, and capability
patterns allowing for process engineers to author, configure, view, and publish your
software development process [IBM RMC 2008]. There are two main purposes of the RMC
[Huamer 2005]. Firstly, the RMC acts a content management system that store, maintain
and publish your knowledge base of process contents to development practitioners.
Secondly, its purpose is to provide a tool for process engineers to select, tailor, and
assemble process contents that fit to their specific development projects. Since IBM RMC
library stores the process content separately from the process, the process engineer can
create a new process by configuring the pre‐defined content in the method content area. As
a result, process engineers can create different processes for different types of project using
the predefined content in the method content library.
Figure 9: An example of an IBM RMC page
23
Chapter 3 : Research Methodology
This chapter discusses the scope of the research and how the research is conducted.
Section 3.1 elaborated on how the ICSM is applied to the research and the overview of
research schedule and research scope. Scope of data collection and analysis based on
hypotheses is discussed in section 3.2. Lastly, section 3.3 discusses threats to validity.
3.1. Research Timeline
The dissertation development approach and overview timeline of this research is
shown in Figure 10 and Figure 11. This research is developed by using the concept and
structure of the Incremental Commitment Spiral Model (ICSM). As in the Exploration phase,
major activities include literature review, current process guideline analysis, and
alternatives exploration. In the Valuation phase, early 2007, the ICSM was found to be the
most appropriate process model to follow. Additionally, there is a high demand to improve
the COTS‐Based Development (CBD) process guideline. There is also a need for the
guidelines for services‐intensive process and a tool that can help the development team
select the most appropriate process pattern and to publish the interactive and online
version of the process guidelines content. In the Foundations phase, the IBM Rational
Method Composer is selected as a tool to develop the electronic process guide (EPG) and the
structure of process guidelines is developed. In the Development phase, early 2008, the
first version of the Architected Agile process guidelines was developed and deployed in the
Operation phase in the form of ICSM EPG. On the other hand, by using the Iterative
development cycle, while the Architected Agile is deployed, the CBD process guidelines are
24
architected and transformed to fit with the ICSM concept and adopt the same structure that
the Architected Agile pattern is using, in order to solve the problem of high context
switching and high learning curve when a team switches between the Architected Agile
pattern and CBD pattern. Another iterative development cycle occurred when the process
decision driver was developed and later deployed in August 2009.
Figure 10: Dissertation Development Approach
Figure 11: Dissertation Timeline
25
Process and process guidelines that are used in software engineering classes are
evolving. As shown in Table 4, in fall 2008, the Electronic Process Guide replaced the PDF‐
generated process guidelines, and at the same time, the MBASE process was replaced with
the ICSM. But for the process guidelines, in 2008, the LeanMBASE guidelines were replaced
with the ICSM‐Architected Agile, while the COTS‐based development teams or Services‐
based development teams followed the CBD process guidelines. Based on the feedback and
problems found in CBD guidelines, the four process patterns for rapid‐fielding projects
were developed and introduced to the Software Engineering class in 2009.
Table 4: Software Engineering Processes Used in Software Enmgineering class
Year Process followed Process guidelines Process guidelines media2005 MBASE LeanMBASE, CBD Word Processing/ PDF 2006 MBASE LeanMBASE, CBD Word Processing/ PDF 2007 MBASE LeanMBASE, CBD Word Processing/ PDF 2008 ICSM Architected Agile, CBD Electronic Process Guide2009 ICSM 4 Process Patterns Electronic Process Guide
Each year, the development teams are informed about the possible process patterns
and characteristics of each process pattern, and then the teams select the development
process to follow. Table 5 reports on number of teams selecting to adopt which process
pattern.
Table 5: Number of teams in each process pattern
Year LeanMBASE CBA Total 2005 17 3 20 2006 17 2 19 2007 19 0 19 2008 11 2 13 Architected Agile Use Single NDI NDI
Intensive Service Intensive
Total
2009 9 0 2 2 13
26
3.2. Hypotheses
The data are collected and analyzed to answer the following hypotheses:
Hypothesis 1: The rapid‐fielding software development teams following the
Incremental Commitment Spiral process model would outperform others using
traditional processes.
To test hypothesis 1, compare between the teams that follow the Incremental Commitment
Spiral Model (2008 – 2009) and the teams that follow traditional processes; in this case it is
LeanMBASE process (2005 – 2007). The following aspects are compared and analyzed:
client satisfaction and percentage of faster time‐to‐market teams.
Hypothesis 2: The rapid‐fielding software development teams using the
Incremental Commitment Spiral process model – Electronic Process Guide (ICSM
EPG) would outperform others using traditional processes guidelines.
Focusing on how the ICSM EPG supports the development team, the following data is
analyzed to compare the difference in personnel effort between the teams that used EPG
and the teams that used word processing/pdf‐generated process guidelines. Additionally, to
analyze the usability and benefits of the EPG, quantitative survey and qualitative interview
are conducted.
Hypothesis 3: The rapid‐fielding software development teams using the
Incremental Commitment Spiral process model – Process Decision Drivers would
outperform others who do not use it.
27
Incidents of the process selection and re‐selection are analyzed to prove that the process
decision drivers assist in process selection. Moreover, a retrospective analysis is performed
to compare the teams that selected the process pattern with no support from the process
decision drivers. Details and rationale of incident changes are also provided.
3.3. Threats to Validity
This section discusses possible validity threats and ways in which the threats can be
reduced.
‐ Inconsistent Effort Reporting: It is possible that there may be inaccurate efforts
reported, so 2 forms of effort reporting will be used, classroom effort reporting system
and a post‐experiment questionnaire.
‐ Inconsistent Client Satisfaction Ratings: In this experiment, team performance can be
measured from client satisfaction or grade received from graders. Some clients are
finicky or exacting than others. Also, some graders are tougher than others. However,
with the close observation, if there is an outlier or non‐corresponding results, the clients
and the graders will be asked for clarification feedback with the follow up qualitative
interview.
‐ Nonrepresentativeness of subjects: Based on the history of the software engineering
class, the participants, with average of not more than 2 years of industrial experience
and an average 12‐hour, non‐collocated work week, are not representative of a software
engineer in the industry. However, the clients and off‐campus students are full‐time
working professionals.
28
‐ Learning curve: The possibility of an imbalanced team and the threat of learning curve
in using these process models could be overcome by providing tutorials and discussion
sessions in order to build the foundations for all participants.
‐ Nonrepresentativeness of projects: Although the target projects need to conform to
fixed semester schedules, this experiment is to some degree representative of fixed‐
schedule industry projects. The projects are small e‐services applications, but will be
developed for real clients with diverse domains, and will use the same COTS or services
that are used in industry projects. Moreover, the process guidelines and decision drivers
will be cross checked with experts in the field for enterprise‐level compatibility.
29
Chapter 4 : Process Patterns in RapidFielding Projects
4.1. Different Opportunities or Risk Patterns in RapidFielding Project
One can consider ready‐made software such as NDI or NCS as an opportunity or a risk
in software development, different opportunities and risk patterns for each project yields
different software processes. As shown in Figure 12, when there is no significant application
NDI that could contribute their functionalities to the final capabilities of the development
project, the project proceeds with default activities in each phase. On the other hand,
consider the second example, when in the Exploration phase, the developers spent a good
amount of effort to explore and find a perfect NDI that satisfied all the win conditions. This
perfect NDI provides an opportunity for the development team to skip or spend little to no
time in the Valuation and Foundations phases since all functionalities and architecture are
provided by the selected NDI, hence the team does not need to spend time in prototyping or
defining the architecture. The team could start populating data or tailoring the NDI as
appropriate in the Development phase and perform early deployment in the operation
phase. In the third and the fourth examples, in the Exploration phase, the development team
found one or several possible combinations of NDIs/NCSs and its interoperability. To
mitigate the risk and to take the NDI/NCS‐driven opportunity; therefore, the team should
spend more effort and time evaluating the NDIs/NCSs and prioritizing their win‐conditions
or their constraints. On the other hand, since NDIs/NCSs provide a majority of the end user
features, the team could spend less time in Foundations and Development‐related efforts
30
and the team could start operation phase as early as possible. Processes in the NDI and NCS
cases might look similar, but the differences are in the details as mentioned in Table 6.
Figure 12: Different Opportunities and Risk Patterns yield Different Processes
4.2. Differences between NDI and Services
Although the Services‐Intensive case is similar to the NDI‐intensive case, they are
different enough to follow different processes. The development team should use different
criteria to consider the possible NDIs and NCSs. Risks from using NDI are different from
risks from NCS. For example, NCS products are always platform and language independent,
but the users have no control over the changes in the next version. On the other hand, most
NDI products are platform dependent, and the users are fully entitled to the version that
they own. Table 6 and Table 7 report summary of the differences between NDI and NCS.
This also provides another reason why the CBD guidelines an imperfect fit with a Services‐
Based development process.
31
Table 6: Differences between NDI and NetCentric Services
Category NonDevelopmental Item [ includes open source, customer
furnished software]
NetCentric Services
Payment • Non‐commercial items usually have no monetary cost
• Reliability, Availability, Cost, Available Support, Speed, Predicted longevity of the service provider, release cycle, Bandwidth
• Recurring costs to use of the service and future functionality offered
• Standards compatibility; Feature‐ data controllability
32
Table 6: Continued
Category NonDevelopmental Item [ includes open source, customer
furnished software]
NetCentric Services
Support Services
• Vendor support for integration, training and tailoring/modification sometimes available for a fee
• Help topics or FAQs would likely not be updated after installation
• Upgrades/Patches and data migration support
• Sometimes can be customized for specific user
• Upgrade through purchasing new releases, self‐install
• Support for tailoring/modification, training generally not available
• Help topics would generally be frequently updated; self‐learning
• Usually not customized for specific user • Patching on service provider’s side; mostly does not require installation on client side
Data • Data often stored locally. Backups generally the responsibility of the user
• Data access is generally fast • Possible variety of proprietary formats • May be inflexible for change but more secure
• Platform‐dependent data format • Can process data offline
• Data stored on service host’s servers. Backups by the provider. Introduces privacy and data‐retention
• Data access could be slower since it is internet based
• Common XML using web standard protocols
• Data from different web services can be used by a single client program
• Process data online
Table 7: Differences between NDI and NCS
Characteristics NDI NCS Platform Independent Yes / No Yes Required Internet Access Yes / No Yes Common Standard No Yes Option of rejecting next release Yes No Change / upgrade control Client /Server’s site Server’s site End user has the latest version Yes / No Yes Database Ownership Yes Yes/No
4.3. RapidFielding Project Characteristics
Given a project description, one may not know whether to follow a general waterfall or
agile development process or if it needs to adopt different development processes. Table 8
summarizes general characteristics for each rapid‐fielding process pattern. As shown in the
table, NDI‐intensive process pattern that requires NDI‐driven architecture and NDI‐
33
experienced personnel to develop the system. On the other hand, the only condition for Use
Single NDI is the complete solution provided by the selected NDI.
Table 8: Characteristics of the RiskDriven Process Patterns of the ICSM
Prototyper, Quality Focal Point, Requirements Engineer, System Architect,
and UML Modeler.
77
• Task – a piece of work assigned to a role to achieve a certain purpose. A task is
composed of steps, which explain detailed instruction on how to perform a task. A
task is associated with a primary performer(s) and a secondary performer(s), which
are selected from the pre‐defined roles. A task is also associated with an input work
product(s) and an output work product(s). Additional concepts or guidelines could
be attached to a task to provide clarification or supplementary information.
• Work Product – an output produced from a task. Usually an output will be in the
form of an artifact or a deliverable. The relationship of a work product to other roles
identifies the primary performer, the secondary performer and the modifier. The
work product can be used to identify an input of a task and an output of a task. A
work product can be associated with supplementary elements, such as example,
template, concept, or guideline.
• Supplementary Element – There are many kinds of supplementary elements such
as a checklist, concept, example, guideline, report, roadmap, template, term
definition, tool mentor and whitepaper. The objective of a supplementary element is
to provide additional information to a task, a work product or to other
supplementary elements.
• Practice – a practice, as shown in Figure 34, is used to define a group of related
tasks, work products and supplementary elements. Eight practices are
o Feasibility Evidence Description Practice focuses on how to provide the
evidence that the development project is feasible.
o Life Cycle Planning Practice focuses on answering the most common
questions about a project or activity: why?, whereas?, what?, when?, who?,
where?, how?, and how much?
78
o Operational Concept Development Practice focuses on capturing the
visions shared among all the success‐critical stakeholders for the project
being undertaken.
o Prototyping and Implementation Practice focuses on developing,
tailoring, integrating and transitioning the system.
o Quality Management Practice focuses on balancing the mutual satisfaction
along the lines negotiable by the stakeholders in their win‐win negotiations.
o System and Software Architecture Development Practice focuses on
capturing the design of the proposed system at the technical level of detail.
o System and Software Requirements Development Practice focuses on
negotiating, developing and assessing the requirements
o Testing Practice focuses on identification of test plans, test cases, test
procedures, and test results.
Figure 34: View of Operational Concept Development Practice
79
RMC provides an easy way to assign relationships between each process element.
The left panel of Figure 35 shows the process content library for feasibility evidence
development practice, which mainly comprises tasks, work products, and guidance. The
bottom tabs of the right panel allow a process engineer to describe overall information of
each task, identify steps of the task, assign the primary and secondary roles responsible for
the task, specify the input and output work products, and identify the related guidance, as in
this case, for the “analyze_business_case” task. Additionally, a process engineer can
illustrate the relationships between a particular role and his/her responsible tasks and
work products, which can be output in the form of an interactive graphical representation
as shown in Figure 36.
Figure 35 Defining Relationship between Process Elements
Figure 36: Relationship between roles, tasks, and work product
80
6.2. Process Representation
After all the process elements and their relationships have been defined, a process
engineer can configure the process framework by defining the time lines, such as phases,
iterations, and milestones, and constructing the work breakdown structure, team allocation,
and work product usage by assigning activities and the associated tasks in the appropriate
time lines. Figure 37 shows that there are 6 phases and 5 milestones for software
development in the CSCI577ab software engineering class. With the interactive interface,
when the user double‐clicks on a particular phase, the EPG displays a flow of activities
represented in an activity diagram. Furthermore, once the user clicks on each activity, the
EPG shows a list of related tasks for that particular activity. Find more pictures of ICSM EPG
in Appendix C.
Figure 37: Overview of the ICSM EPG
81
Finally, the view or structure, of the published process can be customized in order to
fit one’s development project or type of project. Firstly, the list of navigation menus can be
created based on the views and pages that would facilitate the page navigation of users or
development practitioners. Secondly, the process elements can be grouped together to
specify elements that are related such as templates or example documents. Lastly, the
grouping of the process elements allows for various views to be created for users, such as
role‐oriented or task‐oriented views. This enables the users to view the process from
different perspectives, while allowing them to easily navigate to their targeted process
elements.
The ICSM EPG also represents the dynamic state of the software development process
by illustrating the flow of activities of the process. An example of Process Representation
can be found in Appendix C.
• Delivery Process identifies possible types of processes to be followed. Since each
process pattern will have different course of actions, the process author could tailor
the delivery process of each process pattern by pick and choose the appropriate
description, flow of activity, work breakdown structure, team allocation, and work
product usage. There are 4 process patterns in the ICSM EPG
o Architected Agile Process
o Use Single NDI Process
o NDIIntensive Process
o Servicesintensive Process
82
Figure 38 illustrates how to configure the defined activities and tasks for each phase
and Figure 39 shows how to create the relationship between each activity or each task in
activity diagram representation.
Figure 38: Work Breakdown Structure for Architected Agile Process
Figure 39: Activity Diagram of an Explore Current System Activity
83
• Work Breakdown Structure represents the dynamic perspective of software
process by defining the software development phases and identifying a group of
tasks or activities for each phase. Moreover, by using UML activity diagram notation,
the process author can define the swim lane or scope of responsibilities for each
role and define the synchronization milestone, branching conditions, and
concurrent activities.
6.3. Discussion
6.3.1. Comparison Software Process Modeling Tools
Because of the complexity of the software process, various process definition languages
(PDL), process modeling tools (PML), workflow representation language, and Electronic
Process Guide generator tools have been created to represent and characterize software
processes and their related activities, roles, artifacts, and tools. [Fugetta 2000]. Main
purposes of software process modeling tools are to support and facilitate in process
understanding, process design, training and education, process simulation and
optimization, and process support. To model the LeanMBSE and ICSM, the EPG generator
tools have an advantage over PDLs since PDLs have limitations in representing non‐
sequential processes. PDLs require known pre‐condition, post‐condition, and sequence of
tasks. On the contrary, LeanMBASE and ICSM are risk‐driven process and the sequence of
tasks depends on the project risks.
IBM Rational Method Composer, Spearmint, and Little‐JIL are evaluated in order to
find the most appropriate tool to represent the process guidelines and to be experimented
in the software engineering class.
84
Spearmint vs. IBM Rational Method Composer
To compare between two EPG generator tools, Spearmint and IBM RMC, the criteria of
process modeling tools defined by [Humphrey 1989] are used as a foundation. The analysis
results are as followed:
• Representation of Process Elements – IBM RMC provides more representations for
guidance such as checklist, example, guidelines, template, and tool mentor.
Additionally, IBM RMC allows the process author to define skills for each role and
detailed steps for each task.
• Representation of Relationship Between the Process Elements – Spearmint
describes process elements textually, which IBM RMC is using graphical and textual
representation.
• Representation of a Process – Spearmint generates the behavior and functional
diagram for the process, while IBM RMC provides the process representation via
work breakdown structure (WBS) and activity diagram, which can be represented
hierarchically to show the tasks and sub tasks detail.
• Support Reusable of Content in the Process – IBM RMC separates method content
and process content, hence each process element are reusable. Also the method
content can be updated without affecting the process content.
• Dynamic Process Configuration – Both Spearmint and IBM RMC are not able to tailor
a process guideline in real‐time.
• Integration to the other Software Engineering Tools – IBM RMC are able to integrate
with various IBM tools and other tools such as Microsoft Project or Concurrent
Versions System (CVS).
85
• Comparative Usage feedback – Both Spearmint and IBM RMC are easy to use, but the
process author found that Spearmint is difficult to tailor or update the content.
LittleJIL vs. IBM Rational Method Composer
Little_JIL and IBM RMC are used to prototype the process guidelines for ICSM. This
section addresses the commonalities and differences of EPFC and Little‐JIL framework.
• Process lifecycle ‐ Both IBM RMC and Little‐JIL have basic ability to model the
software lifecycle as concurrent and iterative development lifecycle. However, there
is another challenge in order to model complete ICSM. Different risks can create
different ICSM processes. Hence, the process model needs ability to customize a
process guideline in real‐time based on the project risks. We call this ability as
“dynamic process configuration”. Currently, neither IBM RMC nor Little‐JIL provides
this capability.
• Process elements and their relationships ‐ IBM RMC provides more representations
for guidance such as checklist, example, guidelines, template, and tool mentor.
• Communication process ‐ Unlike the sophisticated and luxurious process elements
of IBM RMC, with the simple yet detailed tree‐like structure diagram, Little‐JIL can
precisely represent the software processes. With expandable step/sub‐step options,
the process performer can select to see the overview or detail information of the
project. With various kinds of notation for various kinds of step relationship and
little‐JIL formalism, we can effectively communicate the software process with very
precise information
• Facilitate process reuse ‐ Opposite to IBM RMC, for Little‐JIL, in order to represent
the preciseness and elaborateness of software process, it turns the process model to
86
be very project‐specific. Hence, this makes reuse of the model less frequent, but
enables the reuse to be checked for consistency
• Support process evolution ‐ When the project evolves and expands, Little‐JIL
supports the expandability in some extent. One can specify as many sub‐steps of
sub‐steps as needed. But if the project evolution causes the process structural
transformation, it will often cause tedious works to change the whole structure. So,
the process model can be expandable in vertical way but limited in horizontal way.
Because of the project‐specific preciseness, it is more difficult to tailor the process to
fit with projects in different scenario, but again more consistency‐checkable.
• Facilitate process management – IBM RMC and Little‐JIL support this criterion in
different ways. IBM RMC provides basic information to help the project manager
develop the project planning. Little‐JIL is stronger in supporting analysis of
consistency, completeness, and correctness, but has fewer constructs for assessing
required resources.
• Usability study ‐ To test the usability, preliminary experiment is conducted with
target users. The first group is graduate students who took software engineering
class, used paper‐based guidelines, familiar with software engineering process
especially LeanMBASE, and have brief knowledge about ICM. The second group is
first year graduate students who plan to take software engineering class, have never
used paper‐based guidelines, and have brief knowledge about software engineering
process and ICM. After briefing about ICM, we are observing both groups using ICM
process guidelines both in IBM RMC and Little‐JIL. During the study, project roles is
assigned to participants and asked them to find out their scope of responsibilities
from both tools. We found that, comparing with Little‐JIL, it is easier for the
87
participants to find role‐based tasks from ICSM‐EPG. After observation, post‐
interview is conducted. The open‐ended questions were asked during the
interview. The results show that the first group prefers ICSM‐EPG because it
provides a lot of information and link to external tools and document templates.
The second group prefers Little‐JIL because it is simple and easy to understand.
Both groups concur that ICSM‐EPG is very complex, difficult to get the overview
picture, and easy to get lost. On the other hand, both groups mentioned that Little‐
JIL does not represent responsibility in role‐based fashion, and it is very difficult to
find tasks that one has to do.
6.3.2. ICSM EPG Analysis
Fall 2008 is the first semester that Software Engineering students at USC have used the
EPG to guide their software development projects. In this section, I will provide analyses on
the data of students’ efforts spent on their projects and the results comparing the use of the
EPG and the use of the paper‐based guidelines. The EPG is evaluated based on process
model objectives defined by Humphrey and Kellner [Humphrey 1989] and good
characteristics of process modeling tools defined by Fuggetta [Fuggetta 2000].
Based on Humphrey and Kellner [Humphrey 1989]’s four objectives of a good software
modeling process tool, we found that
• In enabling effective communication regarding the process, RMC not only
provides basic process elements representation, such as roles, tasks, and artifacts or
additional representations such as checklist, tool mentor, and template, but it also
allows one to create special types of process elements with customized notations to
fit with one’s development projects. This benefit empowers the process engineers
88
with more alternatives to provide guidance and various types of supporting
materials.
• In facilitating process reuse, RMC clearly separates the method contents and
process contents, so the process engineer can tailor the method content without
affecting the process content, and vice versa. For example, when a process engineer
creates a new delivery process or an activity diagram, the current tasks and roles
can be reused as needed.
• In supporting process evolution, RMC fully supports process tailoring and
configurations, the EPG can surely be extended to support process evolution.
• In facilitating process management, RMC is fully interoperable with project
management tools such as the IBM Rational Team Concert [IBM RTC] and IBM
Rational Portfolio Manager [IBM RPM]. Although the current version of the EPG
does not provide project management facilities, it will be our challenge in the future
to pick the right tool to integrate with the software engineering class, as well as to fit
the nature of the development projects.
Secondly, to enable effective communication among various stakeholders, it is
important to use nomenclatures that will be widely comprehensible. RMC provides a rich
set of notations, such as process element icons and graphical representations in the delivery
process descriptions and activity diagrams. Based on the criteria of good notations
proposed by Humphrey [Humphrey 1989], we evaluated notations provided in RMC and
found that
• In preciseness and conciseness, RMC provides a set of graphical representations
that sharply conveys the meaning of each process element.
89
• In convenience of use, RMC provides sets of functionalities allowing the process
engineers to easily pick and choose the notations and elements in the course of
modeling a process.
• In being commonly understood, RMC uses pictorial icons that represent standard
real‐world objects. The users will be able to interpret the icons in universal
meanings
• In being suitable for representing a broad range of software functions, even though
the current sets of notations cover a wide range of software activities, RMC allows
the addition of extra custom process element icons to fit with any additional
requirements a process engineer might see fit.
Lastly, Heidrich proposed in [Heidrich et.al. 2006] that people‐oriented process
information should describe role‐specific responsibilities, activities, and abilities. RMC
represents this information by using work products, tasks, and properties respectively. The
groupings of these elements allow the process engineers to illustrate various perspectives
of the process to the readers, enabling the ability to represent the big picture of the process,
as well as specific views.
6.3.3. Effort Comparison between EPG and nonEPG
To determine the effectiveness of using the ICSM EPG, it is compared with the use of
traditional paper‐based versions of the guidelines. The data used to perform the
comparisons were the effort, or the number of hours, spent by students in performing
specific project‐related activities. These data were taken from the USC’s CSCI577a Software
Engineering course of fall 2007 to 2008 semesters, where the fall 2007 semester utilized the
paper‐based version and the fall 2008 semester utilized the electronic version. To
90
compensate for the difference in class sizes, the average of the class effort resulting in the
average effort spent by each person on their projects is considered.
Table 16: Comparison of effort between paper‐based guidelines and the ICM EPG
Guidelines \ Average Effort (hours) Learning Documentation Total EffortPaper‐based process guidelines 15.2 69.2 209.57 ICM EPG 11.9 46.6 175.8
As shown in Table 16 this information allows us to see the fractions of effort
students spent in learning the process and documentation with respect to the total effort
they put into the project. In the fall 2007 semester, the average student put in a total of 15.2
hours toward learning the process and 69.2 hours toward to documenting the artifacts,
while in the fall 2008 semester, the average student spent a total of 11.9 hours on the
project with hours to learn and 46.6 hours to document. It is very clear that with ICSM EPG,
it supports in saving learning and documentation effort for process users.
6.3.4. Qualitative Feedback from students’ survey results
A survey about the EPG usage is conducted in the software engineering class, and
the students provided their feedback on various dimensions. Here are the qualitative
feedback results
• Regarding the effectiveness in communication of the process, the ICSM EPG greatly
improves their understanding about the software development process and is also
highly supportive in discussing the process among team members and
communicating the process to the project clients.
• Regarding the role‐oriented process information, the ICSM EPG provides role‐
specific essential information, such as responsibilities, activities, and abilities, which
91
crucially help the student teams in task allocation, project planning, and project
management.
• Regarding the process modeling and support, the ICSM EPG highly assists the
students’ teams to plan and execute the process.
Moreover, the ICSM EPG mentors the students on the list of artifacts to be created,
maintained, and delivered. The majority of the students nominated the interactive graphical
representation as the most useful feature, while the improper search function was the least
useful function. Role‐specific information, delivery process, templates, and example
documents were the features that were most helpful in aiding them in learning and
understanding the development process.
6.3.5. Quantitative Survey Results on ICSM EPG
Table 17 provides a summary result of the quantitative survey from the students
who use the ICSM EPG. Based on a score of 5, the process users agree that the ICSM EPG
supports them in all categories. In the first category, the process users concur that the ICSM
EPG improves their understanding about the software development process, enables the
communication about software development process among team members, and enables
the communication about the software development process with the client. In the second
category, the process users highly agree that the ICSM EPG provides role‐specific
information about the software development process, helps one to understand
responsibilities of each role with respect to the process, assign tasks to each role if they
were the project manager, select the role that one wants, and explain activities in which
each has to participate. In the process modeling and support category, the process users
agree that the ICSM EPG provides essential information for one to plan and execute one’s
92
software development process, provides information that supports them in process
adaptation to fit with project status, and provides information about the software
development process and its deliverables in each phase. In the last category, the process
users agree that the ICSM EPG helps one to eliminate inconsistencies in the process
specification, easy to user, easy to understand and has a high clarity of navigation.
Moreover, more than 75% of process users prefer the electronic version of the process
guidelines. On the other hand, 25% of process users would prefer to have both electronic
version and paper‐based version to read off‐line. Detailed results of the quantitative survey
can be found in Appendix L.
Table 17: Results of ICSM EPG Survey
Category Result (Scale of 5) I. Effective communication regarding the process 3.86 II. Support People‐oriented process information 4.14 III. Support Process modeling 3.97 IV. Support Process Improvement 3.76
6.3.6. Quantitative Survey Results regarding PDFguideline vs EPG
Another survey is conducted to compare the experience on using PDF‐guidelines
and the EPG. The target groups are the students who took Software Engineering course
during fall 2005 – 2007 where the LeanMBASE process guidelines was available only in PDF
format. The questions are asked to evaluate the PDF‐guidelines usage experience. Table 18
summarizes the results of the survey in each category. The PDF‐guidelines moderately
supports the process users in learning about the process, provides people‐oriented process
information, and supports process modeling, but fairly support process improvement.
93
Table 18: Results of PDFguidelines Survey
Category Result (Scale of 5) I. Effective communication regarding the process 3.44 II. Support People‐oriented process information 3.30 III. Support Process modeling 3.18 IV. Support Process Improvement 2.69
Comparing the evaluation result between Table 17 for ICSM EPG evaluation and
Table 18 PDF‐guideline evaluation, it is clearly shown that the process users prefer the
ICSM EPG in every category.
Furthermore, the target groups are introduced about the ICSM EPG and are asked to
compare between the ICSM EPG and PDF‐guidelines in each category. To test whether the
ICSM EPG outperform the PDF guidelines, the target groups are asked whether they agree
that the EPG is superior to the PDF‐guidelines in each category. Table 19 shows that the
target groups strongly agree that the EPG is better than the PDF‐guidelines in every aspect.
Detailed information and questions can be found in Appendix M.
Table 19: Comparison between the EPG and PDFGuidelines
I. The EPG is more effective in process communication 4.11II. The EPG provides better support in people‐oriented process information 4.40III. The EPG provides better support in process modeling 4.19 IV. The EPG provides better support in process Improvement 4.29
6.3.7. Conclusions
Both quantitative and qualitative feedback has confirmed that the electronic process
guide supports the process users in process learning, team communicating, project
managing, task scheduling, and process improving. It also saves the effort spent in learning
94
about a new process. Hence, it is clearly shown that the ICSM EPG teams outperformed the
teams that used PDF‐process guidelines.
95
Chapter 7 : Conclusion and Future Work
7.1. General Conclusions
As technology evolves, various ready‐made software components can be used to speed
up the software development, especially for rapid‐fielding projects, where time‐to‐market is
the key factor to gain advantage over the competition. Current software development
process models and process guidelines are either outdated or not supported by the updated
technology, especially for Services‐intensive projects. Based on the Incremental
Commitment Spiral Model, there are 4 process patterns that dominate our Software
Engineering class rapid‐fielding projects, which are Architected Agile, Use Single NDI, NDI‐
Intensive, and Services‐intensive process patterns. The experiments have been done to
confirm that the Incremental Commitment Spiral Model and its process guidelines support
the team in developing rapid‐fielding software project. However, it is important to note that
the results are currently only representative of small web services applications.
7.2. Summary of Contributions
In summary, this dissertation describes how the Incremental Commitment Spiral
Model can be applied to four process patterns of the rapid‐fielding projects. To provide the
process guidelines, the related software development process guidelines, or standards, are
investigated and analyzed. The details of four ICSM rapid‐fielding process guidelines are
developed based on various good practices and the process content is published by using
the IBM Rational Method Composer. Additionally, the process decision drivers, which help
the development team in selecting the appropriate process pattern, are developed. The data
96
are collected from 5 years of Software Engineering courses (2005‐2009) and are analyzed
to validate the hypotheses.
7.3. Future Work
Several suggested future research directions include:
• Experiment on Agile Development – although Agile Development would be a good
process pattern choice for rapid‐fielding projects, it is very difficult to run an
experiment for agile development in the classroom environment. Additionally, agile
development requires agile‐ready personnel, which is lack in most of the software
engineering students in this class. Hence, it would be great to develop process
guidelines, including process decision drivers to cover agile development.
• Replicate Experiment in Industry Environment – although the projects
developed in the software engineering class environment are comparable to
projects developed in the industry environment, to proof that the process guidelines
are applicable to all rapid‐fielding projects especially large‐scale rapid‐fielding
project, the experiment should be repeated in the industry environment.
• Extend COCOTS to support servicesbased estimation – COCOTS can be used to
provide cost estimation for NDI‐intensive projects. However, it would be beneficial
to extend COCOTS to support services‐intensive project cost estimation.
97
References
[Agile 2009] Agile, Principles behind the agile manifesto, http://agilemanifesto.org/principles.html. accessed on 8/4/2009.
[Basili 2001] Basili, V., Boehm, B., "COTS‐Based Systems Top 10 List," Computer, Volume 34, Number 5, May, 2001, pp. 91‐93
[Becker 1999] Becker, U., Hamann, D., and , “Support for the Process Engineer: The Spearmint Approach to Software Process Definition and Process Guidance”. Matthias Jarke, Andreas Oberweis(Eds.): Advanced Information Systems Engineering, Proceedings of the 11th International Conference CAiSE'99, Lecture Notes in Computer Science, Vol. 1626, pp. 119‐133. Springer, 1999
[Bhuta 2007] Bhuta, J., “A Framework for Intelligent Assessment and Resolution of Commercial‐Off‐The‐Shelf Product Incompatibilities” PhD Dissertation, Department of Computer Science, University of Southern California, August 2007
[Boehm 1988] Boehm B, "A Spiral Model of Software Development and Enhancement", IEEE Computer, 21(5):61‐72, May 1988
[Boehm 1996] Boehm B, Anchoring the Software Process, IEEE Software, v.13 n.4, p.73‐82, July 1996
[Boehm 2004] Boehm, B. and Turner, R. 2004. Balancing agility and discipline: a guide for the perplexed. Addison‐Wesley, Boston.
[Boehm 2007a] Boehm, B. and Lane, J., “Using the Incremental Commitment Model to Integrate Systems Acquisition, Systems Engineering, and Software Engineering,” “Cross Talk, October 2007, pp.4‐9
[Boehm 2007b] Boehm, B., “Agility and quality.” IBM Agile Conference, May 15, 2007; ICSE Work‐shop on Software Quality, May 21, 2007.
[Boehm 2008] Boehm, B. and Bhuta, J., "Balancing Opportunities and Risks in Component‐Based Software Development," IEEE Software, November‐December 2008, Volume 15, Issue 6, pp. 56‐63
98
[Boehm 2009a] Boehm, B. and Lane, J., "Guide for Using the Incremental Commitment Model (ICSM) for Systems Engineering of DoD Projects" USC CSSE Tech Report 2009‐500
[Boehm 2009b] Boehm, B., Lane, J. and Koolmanojwong, S. "A Risk‐Driven Process Decision Table to Guide System Development Rigor," Proceedings of the 19 th International Conference on Software Engineering, Singapore, July, 2009.
[Cass 2000] Cass, A.G., Lerner, B.S., McCall, E.K., et al., Little‐JIL/Juliette: A Process Definition Language and Interpreter. Int. Conf. on Software Engineering, Ireland, (2000) 754‐758.
[CMMI‐COTS] CMMI for COTS‐based Systems, http://www.sei.cmu.edu/publications/documents/03.reports/03tr022.html
[CMMI‐Services] CMMI for Services Version1.2, ftp://ftp.sei.cmu.edu/pub/documents/09.reports/09tr001.doc
[CMMI 2009] CMMI for Services flyer, www.sei.cmu.edu/cmmi/models/CMMI‐for‐Services‐one‐pager‐20090320.doc
[CrossTalk] CrossTalk, The Journal of Defense Software Engineering, http://www.stsc.hill.af.mil/index.html, accessed 08/24/09
[Dymond 2010] Dymond R. , The Lean Agile Executive Blog, A Scrum project that failed. http://www.innovel.net/?p=62, accessed on 10/14/2010
[FAR] Federal Acquisition Regulation – Term definitions https://www.acquisition.gov/far/html/Subpart%202_1.html
[Fuggetta 2000] Fuggetta, A., Software process: a roadmap, Proceedings of the Conference on The Future of Software Engineering, p.25‐34, June 04‐11, 2000
[Google Map] Google Map, http://maps.google.com/
[Haumer 2005] Haumer, P. IBM Rational Method Composer: Part 1: Key concepts, December 2005, http://www.ibm.com/developerworks/rational/library/dec05/haumer/
[Heidrich 2006] Heidrich, J., Münch, J., Riddle, W.E., Rombach, D.: People‐oriented Capture, Display, and Use of Process Information. In: Acuña, S.T., Sánchez‐Segura, M.I. (eds.) New Trends in Software Process Modeling. Series on Software Engineering and Knowledge Engineering, vol. 18, pp. 121–179. World Scientific Publishing Company, Singapore
[Humphrey 1989] Humphrey, W. and Kellner, M. “Software Process Modeling: Principles of Entity Process Models.” Proceedings of the 11th International Conference on Software Engineering, PA, USA, 1989. pp. 331 – 342.
99
[IBM RMC] IBM Rational Method Composer http://www‐01.ibm.com/software/awdtools/rmc/
[IBM RPM] IBM Rational Portfolio Manager http://www‐01.ibm.com/software/awdtools/portfolio/
[IBM RTC] IBM Rational Team Concert http://www‐01.ibm.com/software/awdtools/rtc/
[ICSM EPG] Instructional ICSM‐Software Electronic Process Guide, http://greenbay.usc.edu/IICSMSw/index.htm
[Koolmanojwong 2007] Koolmanojwong, S., et.al, "Comparative Experiences with Software Process Modeling Tools for the Incremental Commitment Model" USC CSSE Technical Report 2007‐824
[Koolmanojwong 2008] Koolmanojwong, S., et.al, "Incremental Commitment Model Process Guidelines for Software Engineering Class", USC CSSE Technical Report 2008‐832
[Koolmanojwong 2009] Koolmanojwong, S. and Boehm, B., "Using Software Project Courses to Integrate Education and Research: An Experience Report," Proceedings of the 2009 22nd Conference on Software Engineering Education and Training ‐ Volume 00, CSEET, pp. 26‐33
[Koolmanojwong 2010] Koolmanojwong, S. and Boehm, B., "The Incremental Commitment Model Process Patterns for Rapid‐Fielding Projects," ICSP 2010, Paderborn, Germany
[Lenth 2001] Lenth, R. V., “Some practical guidelines for effective sample size determination”, American Statistician, 2001, 55, 187‐193.
[Li 2006] Li, J., et al., “An empirical study of variations in COTS‐Based Software Development Processes in Norwegian IT industry,” Journal of Empirical Software Engineering vol. 11, no.3, 2006, pp 433‐461
[Morisio 2000] Morisio, M., C. B. Seaman , A. T. Parra , V. R. Basili , S. E. Kraft , S. E. Condon, Investigating and improving a COTS‐based software development, Proceedings of the 22nd international conference on Software engineering, p.32‐41, June 04‐11, 2000, Limerick, Ireland
[Oppenheim 2010] Oppenheim B. W., Murman E.M., and Secor D.A., “Lean Enablers for System Engineering”, doi: 10.1002/sys.20161
[Pew 2007] Pew, R. W., and Mavor, A. S. , “Human‐System Integration in the System Development Process: A New Look”. 2007, National Academy Press.
[Poppendieck 2003] Poppendieck, M and Poppendieck, T. (2003); Lean software development, an agile toolkit. Addison Wesley.
100
[Phongpaibul et.al 2007] Phongpaibul, M., Koolmanojwong, S., Lam, A., and Boehm, B. "Comparative Experiences with Electronic Process Guide Generator Tools," ICSP 2007, pp. 61‐72
[ProgrammableWeb] ProgrammableWeb http://www.programmableweb.com/mashups accessed on 11/9/2009
[Rising 2000] Rising, L., & Janoff, N. The Scrum Software Development Process for Small Teams. IEEE Software, July/August 2000
[Scaffidi 2009] Scaffidi, C.,” Topes: Enabling End‐User Programmers to Validate and Reformat Data”, PhD Dissertation, Technical Report CMU‐ISR‐09‐105, Institute for Software Research (ISR), Carnegie Mellon University, May 2009.
[Smith 2004] Smith, J. “An Alternative to Technology Readiness Levels for Non‐Developmental Item (NDI) Software”, CMU‐SEI Technical Report, April 2004
[USC CSCI577] USC – CSCI577 Software Engineering Class I Website, http://greenbay.usc.edu/csci577/fall2008/site/index.html accessed on 08/24/08
[VModel 2009] V‐Model Lifecycle Process Model http://www.v‐modell.iabg.de/kurzb/vm/k_vm_e.doc accessed on 10/9/2009
[Yang 2006] Yang, Y., "Composable Risk‐Driven Processes for Developing Software Systems from Commercial‐Off‐The‐Shelf (COTS) Products," PhD Dissertation, Department of Computer Science, University of Southern California, December 2006
[Yang 2007] Yang Y. and Boehm B., COTS‐Based Development Process guidelines, http://greenbay.usc.edu/csci577/spring2007/site/guidelines/CBA‐AssessmentIntensive.pdf Accessed 08/24/07
101
Appendix A : Characteristics of each ICSM Process Pattern
Table 20: Characteristics of each ICSM Process Pattern
Special Case Example
Size,
Complexity
Change
Rate
(%/M
onth)
Criticality
NDI
Support
Organizati
onal and
Personnel
Capability
Time/Buil
d;
Time/Incre
ment
1. Use NDI Small accounting
Complete
2. Agile E‐services Low 1‐30 Low‐Med Good; in place
Agile‐ready Med‐high
<= 1 day;
2‐6 weeks
3. Architected Agile
Business data processing
Med 1‐10 Med‐High Good; most in place
Agile‐ready Med‐high
2‐4 weeks;
2‐6 months
4. Formal Methods
Security kernel; Safety‐critical LSI chip
Low 0.3 Extra High None Strong formal methods experience
1‐5 days;
1‐4 weeks
5. HW with embedded SW component
Multi‐sensor control device
Low 0.3‐1 Med‐Very High
Good; in place
Experienced; med‐high
SW: 1‐5 days;
Market‐driven
6. Indivisible IOC Complete vehicle platform
Med‐High
0.3‐1 High‐ Very High
Some in place Experienced; med‐high
SW: 2‐6 weeks;
Platform: 6‐18 months
7. NDI‐ intensive Supply chain management
Med‐High
0.3‐3 Med‐Very High
NDI‐driven architecture
NDI‐ experienced; med‐high
SW: 1‐4 weeks;
Systems: 6‐18 months
8. Hybrid agile/ plan‐driven system
C4ISR system Med‐Very High
Mixed parts; 1‐10
Mixed parts; Med‐Very High
Mixed parts Mixed parts 1‐2 months;
9‐18 months
9. Multi‐owner system of systems
Net‐centric military operations
Very High
Mixed parts; 1‐10
Very High Many NDIs; some in place
Related experience, med‐high
2‐4 months;
18‐24 months
10. Family of systems
Medical device product line
Med‐Very High
1‐3 Med‐Very High
Some in place Related experience, med‐high
1‐2 months;
9‐18 months
102
Table 20: Continued
11. Brownfield Incremental legacy phaseout
High‐Very High
0.3‐3 Med‐High NDI as legacy replacement
Legacy re‐engineering
2‐6 weeks/ refactor ;
2‐6 months
12a. Net‐ Centric Services— Community Support
Community Services or Special Interest Group
Low‐Med
0.3‐3 Low‐Med Tailorable service elements
NDI‐ experienced
<= 1 day;
6‐12 months
12b. Net‐Centric Services—Quick Response Decision Support
Table 22: List of Projects and their processes in Fall 2005
2005 # Project Name Process Followed
P1 Open Source Discussion Board and Research Platform ‐ I
LeanMBASE
P2 Physics Education Research (PER) LeanMBASE P3 Pasadena High School Computer Network Study COTS‐Based DevelopmentP4 Virtual Football Trainer System COTS‐Based DevelopmentP5 Data Mining PubMed Results LeanMBASE P6 USC Football Recruiting Database LeanMBASE P7 USC Equipment Inventory Database LeanMBASE P8 Data Mining of Digital Library Usage Data LeanMBASE
P10 Code Generator – Template based LeanMBASE P11 dotProject and Mantis integration LeanMBASE
P12 Mule as a highly scalable distributed framework for data migration
COTS‐Based Development
P13 Develop a Web Based XML Editing Tool LeanMBASE P14 CodeCount™ Product Line with XML and C++ ‐ IP15 COSYSMO Extension to COINCOMO LeanMBASE P16 Intelligent, 'diff'ing CodeCount Superstructure LeanMBASE P19 EBay Notification System LeanMBASE P21 Rule‐based Editor LeanMBASE P22 Open Source Discussion Board and Research
Platform ‐ II LeanMBASE
P23 CodeCount™ Product Line with XML and C++‐ II LeanMBASE
105
Table 23: List of Projects and their processes in Fall 2006
2006 # Project Name Process Followed P1 California Science Center Newsletter System LeanMBASE P2 California Science Center Event RSVP System LeanMBASE
P3 California Science Center Volunteer Tracking System
LeanMBASE
P4 Credit Card Theft Monitoring Project LeanMBASE
P6 USC Diploma Order/ Tracking Database System
LeanMBASE
P7 USC Civic and Community Relations (CCR) web application
LeanMBASE
P8 Student's academic progress web application LeanMBASE P9 Personal Care Technology Help Line LeanMBASE P10 Video Uploading and Conversion System LeanMBASE P11 New Economics for Woman (NEW) LeanMBASE P12 Eclipse COCOMO II LeanMBASE P13 Web Portal for USC Electronic Resources LeanMBASE P14 Early Medieval East Asian Tombs LeanMBASE P15 LANI Database Management System COTS‐Based DevelopmentP16 USC CONIPMO LeanMBASE P17 UAV Sensor Planning LeanMBASE P18 Electronic Data Discovery COTS‐Based DevelopmentP19 An Eclipse Plug‐in for Use Case Authoring LeanMBASE P20 Online Requirements Negotiation Support
System LeanMBASE
P21 African Millennium Foundation LeanMBASE
106
Table 24: List of Projects and their processes in Fall 2007
2007 # Project Name Process Followed P1 Box Office Database System LeanMBASE P2 Online Peer Review System LeanMBASE P3 Thai CDC Software Tool LeanMBASE P4 USC COINCOMO LeanMBASE P5 Crystal Stairs Dashboard LeanMBASE P6 Family & Homeless Well‐Being Project LeanMBASE P7 BTI Appraisal Projects LeanMBASE P8 LAMAS Customer Service Application LeanMBASE P9 Web‐based service for TPC Foundation LeanMBASE P10 REEO Database LeanMBASE P11 BID review System LeanMBASE P12 THSA Website Project LeanMBASE P13 EEO Section Database LeanMBASE P14 Conference Room Reservation System LeanMBASE P15 Proctor and Test Site Tracking System LeanMBASE P16 E‐Mentoring program LeanMBASE P17 Social Networking Tools for Librarians LeanMBASE P18 EZSIM II LeanMBASE P19 Los Angeles County Generation Web
Initiative LeanMBASE
P20 Development of hunter‐gatherer database LeanMBASE
107
Table 25: List of Projects and their processes in Fall 2008
2008 # Project Name Process Followed P1 Master Pattern Architected Agile P2 Housing Application Tracking System Architected Agile P3 Revamping Proyecto Pastoral Architected Agile P4 UNO Web Tool Architected Agile P5 Hunter‐gatherer interactive research database Architected Agile P6 The IGM On‐Line Art Gallery** COTS‐Based Development P7 EZBay Architected Agile P8 AAA Petal Pushers Remote R&D Architected Agile P9 Information Organization System** COTS‐Based Development P10 The Roots of Inspiration web site** Architected Agile P11 Web‐based Service for TPC Foundation COTS‐Based Development P12 Online Peer Review System for Writing Program Architected Agile P13 Acme Research Engine for USC‐WB Archives Architected Agile
P14 The Virtual Assistant Living and Education Program
Architected Agile
P15 Data Base for III, Inc. COTS‐Based Development P16 Theatre Script Online Database Architected Agile
** denotes the disregarded teams due to unusual circumstance of the projects, which could the outlier in data collection
Table 26: List of Projects and their processes in Fall 2009
2009 # Project Name Process Followed P1 Online DB support for CSCI 511 Architected Agile P2 SHIELDS for Family Architected Agile P3 Theater Stage Manager Program Architected Agile P4 Growing Great Online NDI‐Intensive P5 SPC Website Automation Enhancement NDI‐Intensive P6 VALE Information Management System Services‐Intensive P7 LANI D‐Base Architected Agile P8 Freehelplist.org Architected Agile P9 Early Medieval East Asian Timeline Architected Agile P10 BHCC Website Development Architected Agile P11 AI Client Case Management Database Architected Agile P12 AI Website Development Services‐Intensive P13 Healthcare The Rightway** NDI‐Intensive P14 AROHE Web Development Architected Agile
** denotes the disregarded teams due to unusual circumstance of the projects, which could the outlier in data collection
108
Appendix D : Projects that deliver in onesemester
Table 27: List of onesemester projects
Year Projects Group I – Prepare to transition within 12 weeks or recommend an NDI/NCS as a final deliverable 2005 Pasadena High School Computer Network Study2005 Virtual Football Trainer System2005 Mule as a highly scalable distributed framework for data migration2006 LANI Database Management System2006 Electronic Data Discovery2007 Thai CDC Software Tool2007 REEO Database2008 The IGM On‐Line Art Gallery2008 Information Organization System2008 Data Base for III, Inc.2009 Healthcare The Rightway Group II – Enter Operation Phase within 12 weeks2007 Box Office Database System2007 Family & Homeless Well‐Being Project2007 Social Networking Tools for Librarians2008 The Roots of Inspiration web site2008 Web‐based Service for TPC Foundation2009 SPC Website Automation Enhancement2009 VALE Information Management System2009 AI Website Development
109
Appendix E : Student General Information Questionnaire
Figure 40: Student Background Information Survey Page 1
110
Figure 41: Student Background Information Survey Page 2
111
Figure 42: Student Background Information Survey Page 3
112
Figure 43: Student Background Information Survey Page 4
113
Appendix F : Effort Category
Table 28: Effort Categories in Effort Reporting System
Operational Concept Development Implementation Analyze Current System Explore and evaluate alternatives Identify Shared Vision Analyze and prioritize capabilities to prototype Establish New Operational Concept Acquire NDI/ServicesIdentify System Transformation Prepare development / operational environment Identify Organizational and Operational Transformation Develop prototype Identify Objectives, Contraints and Priorities Develop componentAssess Operational Concept Assess Prototype / componentDocumenting of OCD Tailor NDI/ ServicesSystem and Software Requirements Development Integrate ComponentsSet up WinWin negotiation context Transition the systemNegotiate (during meeting) Develop Transition PlanNegotiation using WikiWinWin tool (after meeting) Develop Support PlanIdentify win conditions Provide TrainingIdentify issues Develop User ManualNegotiate options Documenting of PrototypingDevelop Requirements Definition Testing Assess requirements definition Identify Test CasesDocumenting of SSRD Identify Test PlanDocumenting of WinWin Negotiation Report Identify Test ProceduresSystem and Software Architecture Development Record Test ResultsAnalyze the Proposed System Identify Regression Test PackageDefine Technology‐Independent Architecture Documenting Test‐related documents Define Technology‐Dependent Architecture Perform custom component testSpecify Architecture Styles, Patterns and Frameworks Perform NDI/service component test Assess System Architecture perforn Integration testDocumenting of SSAD perform acceptance test
Life Cycle Planning perform regression test Identify Milestones and Products perform unit testIdentify Responsibilities and Skills Project Administration (currently not in ICSM EPG) Identify Life Cycle Management Approach Create and maintain project website Estimate Project Effort and Schedule using COCOMO II Interact with ClientsEstimate Project Effort and Schedule using COCOTS Interact between team membersAssess Life Cycle Content Learn about Incremental Commitment Model Detail Project Plan Learn about Application domainRecord Project Progress Prepare transition siteRecord Project Individual Effort Manage Code configurationIdentify Development Iteration Attend ARBAssess Development Iteration Planning and controlPerform Core Capabilities Drive‐Through Control Project PerformanceDocumenting of LCP Quality Management Documenting Iteration‐related cocuments Gather DefinitionsTrack problem and report closure Construct Traceability MatrixFeasibility Evidence Description Identify Configuration Management Strategy Analyze Business Case Identify Quality Management Strategy Identify, assess, and manage risks Verify and Validate Work Products Provide Architecture Feasibility Evidence Assess Quality Management Strategy Provide Process Feasibility Evidence Documenting of QMPAssess Feasibility Evidence Documenting of SIDAssess/Evaluate NDI/Services Candidate Documenting of review‐related document Check components Interoperability Documenting of FED
114
Appendix G : ICSM EPG: Roles, Activities, Work products, and Delivery
Process
Figure 44: Welcome Page of ICSM EPG
115
Figure 45: List of Roles in ICSM EPG
Figure 46: List of Practices in ICSM EPG
116
Figure 47: Practice Page in ICSM EPG
117
Figure 48: A task page in ICSM EPG
Figure 49: A role and responsibilities page in ICSM EPG
118
Figure 50: A delivery process page in ICSM EPG
Figure 51: list of work products in ICSM EPG
119
Appendix H : Example of Decision driver
Figure 52: Architected Agile Process Pattern Template
120
Figure 53: Use Single NDI Process Pattern Template
Figure 54: NDIIntensive Process Pattern Template
121
Figure 55: ServicesIntensive Process Pattern Template
Figure 56: An Architected Agile team with Architected Agile Decision Pattern
122
Figure 57: An Architected Agile team with Use Single NDI Decision Pattern
Figure 58: An Architected Agile team with NDIIntensive Decision Pattern
Figure 59: An Architected Agile team with ServicesIntensive Decision Pattern
123
Figure 60: An NDIintensive team with NDIIntensive Decision Pattern
Figure 61: An NDIintensive team with Architected Agile Decision Pattern
124
Figure 62: An NDIintensive team with Use Single NDI Decision Pattern
Figure 63: An NDIintensive team with ServicesIntensive Decision Pattern
125
Appendix I : Client Feedback Form – CSCI 577a
Project Name: __________________________________________ Team # _______ Please provide a ranking where indicated. Use a 1 5 scale where 1 is low and 5 is high. Where comments are requested, please include any descriptive statements you wish to add. FOR THE FIRST TWO ITEMS, consult your team's Development Commitment Package documentation by clicking on the project name in the table shown on the course webpage http://greenbay.usc.edu/csci577/fall2008/site/projects/index.html 1. Operational Concept Description (especially Sections: 2. Shared Vision; 3. System Transformation). How well did the team capture your ideas of what the new system should do? Ranking: (1‐5) Comments: 2. Team helpfulness Did the team suggest new idea about your project? Did the team support you in any project complexity? Ranking : (1‐5) Comments: FOR THE NEXT ITEMS, base your responses on your interactions with the project team and their product. 3. Team Responsiveness: Was the team responsive to you and to your requirements? Did the team answer any questions you might have had? How successful was the requirements negotiation between you and your team? Ranking : (1‐5) Comment: 4. Project Results: How satisfied are you with the prototype? How well does the proposed system meet the need identified by your project? Ranking: (1‐5) Comment: 5. Project Future: Do you think this project should be carried forward for development in CS577b? Has the team adequately identified and managed the potential risks and complications associated with the project? Do you foresee difficulties in transitioning this project into the development and implementation phase? Ranking : (1‐5) Comment: 6. Team Communication: How effective was the team in communicating with you? Do you feel you had enough meetings with them? Did the team make effective use of your time and expertise? What means did you use to communicate? Ranking : (1‐5) Comment: 7. Tools: Regarding software tools students used in the class 7a. Did the team share WinWin negotiation results with you? If so, how? Comment 7b. Did the team mention or describe any other software engineering tools? Comment: 8. Your Learning: Did you gain a better understanding of software engineering and information technology by participating in this project? Please provide specifics where possible. Ranking : (1‐5) Comment: 9. Suggestions about the course processes from your (a client's) perspective. Comment:
126
10. Overall Value: Did you feel your participation was worthwhile? Would you participate again? Did you find participation valuable as either a teaching or research activity? Ranking : (1‐5) Comment: If you have any other comments that you would like to offer, please feel free. We may get back to you in the future on these for clarification.
127
Appendix J : Client Feedback Form – CSCI577b
577B Client Evaluation Team # _______________________ Project ______________________ The items listed below will ask for either a ranking or a yes/no response. For rankings, use a 1‐5 scale where 1 is low and 5 is high. For yes/no items, p[lease indicate your choice. Additional comments will be very helpful for evaluation and planning purposes. Please use the questions in parenthesis as starting points for your comments. Documentation 1. User Manual: Rank your level of satisfaction with the User Manual. (Is it well written? Will it reasonably answer both user and administrator questions? Did your team ask you to evaluate the User Manual? Did they incorporate your comments?) Ranking: (1 to 5) Comments: 2. System Documents: Rank your level of satisfaction with the As Built system documents. Ranking: (1 to 5) Comments: Team Interaction 3. Team Responsiveness: How responsive was the team to your needs as a client? Did they address your project objectives and concerns? Ranking: (1 to 5) Comments: 4. Team Communication: How satisfied were you with the team’s communication with you? (Did they tell you what you needed to know when you needed to know it? Did they explain their work/the issues in terms you could understand?) Ranking: (1 to 5) Comments: System Preparation and Testing 5. How effective were the students at installing the software and providing any necessary set‐up and configuration support to enable you to get started? (Had they anticipated and prepared for any set‐up issues or problems?) Were the students responsive and effective in handling any configuration adjustments that might have been needed once you began using the system? Ranking: (1 to 5) Comments: 6. Did the students help to adequately prepare you to handle ongoing support and maintenance of the system? Yes/No: Comments: 7. Training: Did the team provide appropriate training on how to use the system? (How was this done?) Yes/No: Comments: 8. Training Quality: (Answer only if you received training for the system) – How adequate was your system training? Ranking: (1 to 5) Comments: 9A. Software Testing: Did the team ask you to test the final product?
128
Yes/No: 9B. Software Testing: If yes, did you test all of the system’s features? Yes/No: 9C. Software Testing: How much time did you spend doing system testing? 10. Software Test Results: (Answer only if you participated in software testing) – Please rate how close the functions you used came to meeting your expectations. Did they work as you expected? Ranking: (1 to 5) Comments: Implementation 11. Is the system up and running in full release? or in a beta release? Is any additional programming required to achieve sufficient basic functionality for public release? Yes/No: Comments: 12. Hardware & Software: Do you need additional hardware and/or software to implement the system? Yes/No: Comments: 13. Other implementation issues: Are there any other issues which impact whether or not the system can be implemented? Yes/No: Comments: Overall Value 14. Rate how successful the product is, as delivered, at achieving your objectives and desired functionality. Has the team made the right choices and trade‐offs? Ranking: (1 to 5) Comments: 15. Rate the value / anticipated benefits of the product you’ve received. Is it a worth the time you’ve spent working with the team over the past 2 semesters? Will the product provide sufficient utility? Does it have long term potential or applicability beyond the need outlined in your original proposal? Ranking: (1 to 5) Comments: Summary 16. Your Learning: Did you learn anything new this semester? Yes/No: Comments: 17. Teaching: Did you feel you made a contribution to the education of graduate CSCI students at USC? Yes/No: Comments: 18. Research: Did you feel you made a research contribution by participating in this project? Yes/No: Comments: 19. Recommendation for Others: Would you recommend participation in future projects to your colleagues? (Are there specific projects, either your own or projects of your colleagues, which you would recommend for future courses?) Comments: 20. Feel free to provide any other comments or suggestions for future course improvement.
129
Appendix K : Qualitative Interview Form
Part 1: Information about COTS/ Services in your project Section 1: General info about your project 1. Name, team: 2. What is your client’s computer technology level
[ ] Low [ ] Medium [ ] High 3. What is your client’s role in this project (check all that apply)
[ ] End user [ ] Maintainer/ Administrator [ ] Policy Planner [ ] Acquirer [ ] Other, 4. What are the COTS/ Services you finally selected? 5. What are the functionalities of your selected COTS/Service? 6. What were on your choices? 7. What are the cost(s) of the selected C/S? 8. Does your selected C/S satisfy all the project goals? 9. If not, what are the part(s) that is left to develop? 10. Who introduced these C/Ss?
[ ] Client [ ] Development Team [ ] Other, ___________________________ 11. If client was not the one who introduced the C/S, what was his/her first reaction?
[ ] Oppose [ ] Hesitate [ ] Support [ ] Other, ___________________________________ 12. When did you introduce your C/S?
[ ] Exploration Phase [ ] Valuation Phase [ ] Foundations Phase 13. When did you finalize your C/S?
[ ] Exploration Phase [ ] Valuation Phase [ ] Foundations Phase 14. What were the process / steps in finalizing the C/S selection? Section 2: Criteria in selecting COTS/ Services 1. What kind of information you used to select the COTS/ Services? [ ] Core capabilities in OCD [ ] Business workflow in OCD [ ] Requirements in SSRD [ ] Use case in SSAD [ ] Current system infrastructure [ ] Proposed system infrastructure [ ] Prototype [ ] Business Case Analysis Anything else? 2. Where did you get information about these COTS/ Services? Section 3: Activities in each phase Section 3.1 Exploration Phase 1. Was C/S part of the project proposal? a. If yes, what did your client have any specific C/S in mind at the beginning? b. If no, did you plan to develop everything from scratch at the beginning, why? 2. When you read the project proposal, did you know that there is/ are possible C/Ss available for this project? 3. Any problem regarding C/S in this phase? Section 3.2 Valuation Phase 1. In this phase, did you switch to C/S team or still follow the traditional development guidelines? 2. Did you prioritize or reprioritize your criteria? 3. Did you build any prototype? 4. If your C/S is selected, did you tailor the C/S? 5. If your C/S is selected, did you write any glue code for the C/S?
130
Section 3.3 Foundations Phase 1. In this phase, did you switch to C/S team or still follow the traditional development guidelines? 2. Did you reprioritize your criteria? 3. Did you drop any criteria? 4. Did you build any prototype? 5. If your C/S is selected, did you tailor the C/S? 6. If your C/S is selected, did you write any glue code for the C/S? Section 4: Hypothetical Situations Part 2: EPG for COTS/Services 1. Are there any activities/tasks that you perform but they are not listed/ explained in CBA Guidelines? 2. Is there any role(s) that should be added? 3. Is there any work product(s) that should be added? 4. Is there any guideline(s) that should be added?
131
Appendix L : Results of ICSM EPG Survey
Table 29: Results of ICSM EPG Survey
I. Effective communication regarding the process AVG
1 The ICM EPG improves my understanding about software development process. 4.25
2 The ICM EPG enables my communication about software development process among team members. 3.84
3 The ICM EPG enables my communication about software development process with the client. 3.47
II. Peopleoriented process information
4 The ICM EPG provides essential information about software development process based on your role. 4.25
5 The ICM EPG helps me to understand responsibilities of each role with respect to the process. 4.28
6 The ICM EPG would help me in assigning tasks to each role, if I were the project manager 4.18
7 The ICM EPG helps me to select the role that I want. 3.77
8 The ICM EPG explains about activities in which each role has to participate. 4.21
III. Process modeling and support
9 The ICM EPG provides essential information for me to plan and execute my software development process. 3.92
10 The ICM EPG allows me to learn about software development process incrementally. 4.10
11 The ICM EPG provides information that supports me in process adaptation to fit with your project status. 3.53
12 The ICM EPG provides a framework for analyzing and estimating patterns of resource allocation and consumption in each phase of the software development life cycle.
3.56
13 The ICM EPG provides complete information about software development process in the Exploration phase 4.20
14 The ICM EPG provides complete information about software development process in the Valuation phase 4.16
15 The ICM EPG provides complete information about software development process in the Foundations phase 4.13
16 Information provided in the ICM EPG is consistent. 4.13
17 The ICM EPG provides information about activities that have to be accomplished to achieve process objectives.
4.26
18 The ICM EPG provides information about artifacts to be created and maintained. 4.35
19 The ICM EPG provides an outline for what artifacts to produce for delivery to client. 4.00
20 The ICM EPG provides information about tools to be used in the process. 3.26
IV. Process Improvement
21 The ICM EPG helps me to eliminate inconsistencies in the process specification. 3.82
22 The ICM EPG provides information about quality model (i.e., provide ideal artifacts) 3.80
23 The ICM EPG suggests the steps to be accomplished in order to improve the quality of a software process 3.62
24 The ICM EPG is easy to use. 3.91
25 The notations/graphic representations used in the ICM EPG are easy to understand. 3.84
26 It is easy to find specific information in the ICM EPG. 3.65
27 The ICM EPG has a high clarity of navigation. 3.65
132
Appendix M : Software Process Guidelines Survey
Software Process Guidelines Survey Objectives: 1. To gather feedback regarding the use of PDF‐version of MBASE/LeanMBASE guidelines 2. To compare the experience of PDF‐version of guidelines usage and the ICSM EPG usage MBASE Guidelines http://sunset.usc.edu/classes/cs577a_2004/guidelines/MBASE_Guidelines_v2.4.1.pdf LeanMBASE Guidelines: http://greenbay.usc.edu/csci577/fall2005/site/guidelines/LeanMBASE_Guidelines_V1.4.pdf Part I: Usage of PDFversion of MBASE/LeanMBASE process guidelines Regarding the pdf‐version of MBASE/LeanMBASE guidelines (PDF‐MBASE), please answer the following questions: Effective communication regarding the process
(low/ bad) (high/good) 1 2 3 4 5
The PDF‐MBASE improves my understanding about software development process.
The PDF‐MBASE enables my communication about software development process among team members.
The PDF‐MBASE enables my communication about software development process with the client.
Comments: Peopleoriented process information (low/ bad) (high/good)
1 2 3 4 5
The PDF‐MBASE provides essential information about software development process based on your role.
The PDF‐MBASE helps me to understand responsibilities of each role with respect to the process.
The PDF‐MBASE would help me in assigning tasks to each role, if I were the project manager
The PDF‐MBASE helps me to select the role that I want. The PDF‐MBASE explains about activities in which each role has to participate.
Comments: Process modeling and support (low/ bad) (high/good)
133
1 2 3 4 5
The PDF‐MBASE provides essential information for me to plan and execute my software development process.
The PDF‐MBASE allows me to learn about software development process incrementally.
The PDF‐MBASE provides information that supports me in process adaptation to fit with your project status.
The PDF‐MBASE provides a framework for analyzing and estimating patterns of resource allocation and consumption in each phase of the software development life cycle.
Comments:
134
(low/ bad) (high/good)
1 2 3 4 5
The PDF‐MBASE provides complete information about software development process in the Inception Phase
The PDF‐MBASE provides complete information about software development process in the Elaboration phase
The PDF‐MBASE provides complete information about software development process in the Construction phase
Information provided in the PDF‐MBASE is consistent. The PDF‐MBASE provides information about activities that have to be accomplished to achieve process objectives.
The PDF‐MBASE provides information about artifacts to be created and maintained.
The PDF‐MBASE provides an outline for what artifacts to produce for delivery to client.
The PDF‐MBASE provides information about tools to be used in the process.
Comments: Process Improvement (low/ bad) (high/good)
1 2 3 4 5
The PDF‐MBASE helps me to eliminate inconsistencies in the process specification.
The PDF‐MBASE provides information about quality model (i.e., provide ideal artifacts)
The PDF‐MBASE suggests the steps to be accomplished in order to improve the quality of a software process
The PDF‐MBASE is easy to use. The notations/graphic representations used in the PDF‐MBASE are easy to understand.
It is easy to find specific information in the MBASE/LeanMBASE guideline.
The PDF‐MBASE has a high clarity of navigation. Comments:
135
Part II – Usage of the Incremental Commitment Spiral Model Electronic Process Guide (ICSMEPG) ICSM‐EPG: http://greenbay.usc.edu/IICMSw/index.htm ICSM EPG is a web‐based tool that is created to replace the pdf‐version of MBASE/LeanMBASE process guidelines. Please take a look at the link provided and answer the following questions Effective communication regarding the process (Do not agree) (Agree)
1 2 3 4 5
The ICSM EPG is better than the PDF‐MBASE in improving my understanding about software development process.
The ICSM EPG is better than the PDF‐MBASE in enabling my communication about software development process among team members.
The ICSM EPG is better than the PDF‐MBASE in enabling my communication about software development process with the client.
Comments: Peopleoriented process information (low/ bad) (high/good)
1 2 3 4 5
The ICSM EPG is better than the PDF‐MBASE in providing essential information about software development process based on your role.
The ICSM EPG is better than the PDF‐MBASE in helping me to understand responsibilities of each role with respect to the process.
The ICSM EPG is better than the PDF‐MBASE in assigning tasks to each role, if I were the project manager
The ICSM EPG is better than the PDF‐MBASE in selecting the role that I want.
The ICSM EPG is better than the PDF‐MBASE in explaining about activities in which each role has to participate.
Comments:
136
Process modeling and support (low/ bad) (high/good)
1 2 3 4 5
The ICSM EPG is better than the PDF‐MBASE in providing essential information for me to plan and execute my software development process.
The ICSM EPG is better than the PDF‐MBASE in allowing me to learn about software development process incrementally.
The ICSM EPG is better than the PDF‐MBASE in providing information that supports me in process adaptation to fit with your project status.
The ICSM EPG is better than the PDF‐MBASE in providing a framework for analyzing and estimating patterns of resource allocation and consumption in each phase of the software development life cycle.
Comments:
137
(low/ bad) (high/good)
1 2 3 4 5
The ICSM EPG is better than the PDF‐MBASE in providing complete information about software development process in the Inception Phase
The ICSM EPG is better than the PDF‐MBASE in providing complete information about software development process in the Elaboration phase
The ICSM EPG is better than the PDF‐MBASE in providing complete information about software development process in the Construction phase
The ICSM EPG is better than the PDF‐MBASE in providing more consistent Information
The ICSM EPG is better than the PDF‐MBASE in providing information about activities that have to be accomplished to achieve process objectives.
The ICSM EPG is better than the PDF‐MBASE in providing information about artifacts to be created and maintained.
The ICSM EPG is better than the PDF‐MBASE in providing an outline for what artifacts to produce for delivery to client.
The ICSM EPG is better than the PDF‐MBASE in providing information about tools to be used in the process.
Comments: Process Improvement (low/ bad) (high/good)
1 2 3 4 5
The ICSM EPG is better than the PDF‐MBASE in helping me to eliminate inconsistencies in the process specification.
The ICSM EPG is better than the PDF‐MBASE in providing information about quality model (i.e., provide ideal artifacts)
The ICSM EPG is better than the PDF‐MBASE in suggesting the steps to be accomplished in order to improve the quality of a software process
The ICSM EPG is easier to use compare to the PDF‐MBASE The notations/graphic representations used in the ICSM EPG are easier to understand compare to the PDF‐MBASE
It is easier to find specific information in the ICSM EPG The ICSM EPG has a higher clarity of navigation compare to the ICSM EPG
Comments: ICSM EPG PDF‐
MBASE As a process guideline, do you prefer ICSM EPG or PDF‐MBASE?
138
Appendix N : Results of Software Process Guidelines Survey
Table 30: Results of Software Process Guidelines Survey
I. Effective communication regarding the process 1. The PDF‐MBASE improves my understanding about software development process. 3.667
2. The PDF‐MBASE enables my communication about software development process among team members. 3.5
3. The PDF‐MBASE enables my communication about software development process with the client. 3.167
II. Peopleoriented process information 4. The PDF‐MBASE provides essential information about software development process based on your role. 3.667
5. The PDF‐MBASE helps me to understand responsibilities of each role with respect to the process. 3.5
6. The PDF‐MBASE would help me in assigning tasks to each role, if I were the project manager 3
7. The PDF‐MBASE helps me to select the role that I want. 38. The PDF‐MBASE explains about activities in which each role has to participate. 3.333
III. Process modeling and support 9. The PDF‐MBASE provides essential information for me to plan and execute my software development process. 3.333
10. The PDF‐MBASE allows me to learn about software development process incrementally. 2.667
11. The PDF‐MBASE provides information that supports me in process adaptation to fit with your project status. 2.333
12. The PDF‐MBASE provides a framework for analyzing and estimating patterns of resource allocation and consumption in each phase of the software development life cycle.
2.5
13. The PDF‐MBASE provides complete information about software development process in the Inception Phase 3.333
14. The PDF‐MBASE provides complete information about software development process in the Elaboration phase 3.333
15. The PDF‐MBASE provides complete information about software development process in the Construction phase 3.333
16. Information provided in the PDF‐MBASE is consistent. 317. The PDF‐MBASE provides information about activities that have to be accomplished to achieve process objectives. 3.333
18. The PDF‐MBASE provides information about artifacts to be created and maintained. 4
19. The PDF‐MBASE provides an outline for what artifacts to produce for delivery to client. 3.833
20. The PDF‐MBASE provides information about tools to be used in the process. 3.167
139
IV. Process Improvement 21. The PDF‐MBASE helps me to eliminate inconsistencies in the process specification. 3
22. The PDF‐MBASE provides information about quality model (i.e., provide ideal artifacts) 3.5
23. The PDF‐MBASE suggests the steps to be accomplished in order to improve the quality of a software process 2.833
24. The PDF‐MBASE is easy to use. 225. The notations/graphic representations used in the PDF‐MBASE are easy to understand. 3
26. It is easy to find specific information in the MBASE/LeanMBASE guideline. 2
27. The PDF‐MBASE has a high clarity of navigation. 2.5 Part II – Usage of the Incremental Commitment Spiral Model Electronic Process Guide (ICSMEPG) V. Effective communication regarding the process 28. The ICSM EPG is better than the PDF‐MBASE in improving my understanding about software development process. 4.167
29. The ICSM EPG is better than the PDF‐MBASE in enabling my communication about software development process among team members.
4.167
30. The ICSM EPG is better than the PDF‐MBASE in enabling my communication about software development process with the client. 4
VI. Peopleoriented process information 31. The ICSM EPG is better than the PDF‐MBASE in providing essential information about software development process based on your role. 4.667
32. The ICSM EPG is better than the PDF‐MBASE in helping me to understand responsibilities of each role with respect to the process. 4.5
33. The ICSM EPG is better than the PDF‐MBASE in assigning tasks to each role, if I were the project manager 4.5
34. The ICSM EPG is better than the PDF‐MBASE in selecting the role that I want. 3.667
35. The ICSM EPG is better than the PDF‐MBASE in explaining about activities in which each role has to participate. 4.667
VII. Process modeling and support 36. The ICSM EPG is better than the PDF‐MBASE in providing essential information for me to plan and execute my software development process. 4.667
37. The ICSM EPG is better than the PDF‐MBASE in allowing me to learn about software development process incrementally. 4.333
38. The ICSM EPG is better than the PDF‐MBASE in providing information that supports me in process adaptation to fit with your project status. 4
39. The ICSM EPG is better than the PDF‐MBASE in providing a framework for analyzing and estimating patterns of resource allocation and consumption in each phase of the software development life cycle.
4.167
140
40. The ICSM EPG is better than the PDF‐MBASE in providing complete information about software development process in the Inception Phase 4.167
41. The ICSM EPG is better than the PDF‐MBASE in providing complete information about software development process in the Elaboration phase 4.167
42. The ICSM EPG is better than the PDF‐MBASE in providing complete information about software development process in the Construction phase
4.167
43. The ICSM EPG is better than the PDF‐MBASE in providing more consistent Information 3.833
44. The ICSM EPG is better than the PDF‐MBASE in providing information about activities that have to be accomplished to achieve process objectives. 4.5
45. The ICSM EPG is better than the PDF‐MBASE in providing information about artifacts to be created and maintained. 4.333
46. The ICSM EPG is better than the PDF‐MBASE in providing an outline for what artifacts to produce for delivery to client. 4.5
47. The ICSM EPG is better than the PDF‐MBASE in providing information about tools to be used in the process. 3.5
VIII. Process Improvement 48. The ICSM EPG is better than the PDF‐MBASE in helping me to eliminate inconsistencies in the process specification. 3.833
49. The ICSM EPG is better than the PDF‐MBASE in providing information about quality model (i.e., provide ideal artifacts) 4
50. The ICSM EPG is better than the PDF‐MBASE in suggesting the steps to be accomplished in order to improve the quality of a software process 4
51. The ICSM EPG is easier to use compare to the PDF‐MBASE 4.66752. The notations/graphic representations used in the ICSM EPG are easier to understand compare to the PDF‐MBASE 4.167
53. It is easier to find specific information in the ICSM EPG 4.554. The ICSM EPG has a higher clarity of navigation compare to the ICSM EPG 4.833
EPG PDF
55. As a process guideline, do you prefer ICSM EPG or PDF‐MBASE? 100% 0%