"A framework for assessing the adequacy and effectiveness of software development methodologies
Post on 02-Jun-2018
218 Views
Preview:
Transcript
8/10/2019 "A framework for assessing the adequacy and effectiveness of software development methodologies
1/47
A Framework for Assessing the Adequacy and Effectiveness of
Software Development Methodologies
James
D
Arthur and Richard
E
Nance
1
Virginia Tech d :
ndex
Terms
Software Development Methodologies, Proc edural Evaluat ion, Evaluat ing M ethod-
ologies, S oftware Engineering Object ives, Software Engineering Principles , In duced
Attributes , Indicators .
1
Introduction
Over t he past two decades the software development process has changed dramatical ly. Early
software developm ent practices were guided by seat-of-the-pants prog ram ming styles. Recog-
nizing maintenance difficulties associated with such styles, the software development cornmunit?
began to inve stigate an d identify software engineering principles th a t could significantly enha nc e
the maintainabi l i ty an d qual i ty of a result ing product. Consequent ly, development techriques t ha t
exploi ted software en giv er ing principles l ike abstract ion [LISB75], information hiding [Pr\RD7 ]
and stepwise refinement [WIRN 71] were formulated and integrated into m any software
development
processes.
Th e subsequent de mand for increasingly complex software systems. however, mandated the
coordinated use of complementary principles, guided by an encompassing software development
philosophy th at recognized project level goals an d objectives, i .e. a methodological approach to
software developm ent. Today, myriad of tools, techniques, an d developm ent methodo logies
scl
dress th e chal lenging task of prod ucing high qual i ty software. For example. SC R
[HENKiY]
alitl
.
-..
-
Work supported by the
U S
Navy throu gh the Systems Research Cen ter under Basic Ordering
8/10/2019 "A framework for assessing the adequacy and effectiveness of software development methodologies
2/47
DARTS [GOMH8 ] are development methodologies that emphasize specific software engineering
goals (reduc ing software development costs and facilitati ng the design of real-tim e system s, respec-
tively). SREM [ALFM85] and SADT [ROSD77] are methodology based en v im nm cn k Both focus
on particular phases of the software life cyd e an d are suppo rted by unif ied se ts of complementary
tools.
The steady proliferation of design methodologies, however, has nct been without
a
price. In
particula r, users find increasing difficulty in choosing n appropr ia te methodolop jca l ap p i .~ a nd
recognizing reasonable expecta tions of a design or development methodology. Addressing th is
concern, the research described in this paper outlines a well-defined procedure for
eva luat ing the adequacy of a software development methodology relative
t
project goals.
a n d
assessing th e effectiveness of a methodology relative to th e quality of the produ ct produced.
The evaluation procedure is based on a substantiated set of l inkages among accepted software
engineering objectives, principles, and attributes. These linkages reflect an assessment perspective
structured by the needs, process, and product sequence for system development. and enable
comparative scale for determining the adequacy and effectiveness of the supporting development
methodology. Th e identification of code and docu men tation p roperties and t he definition of metrics
for these properties enables an accumulative determination of software engineering attributes.
principles and objectives.
To provide a uniform basis for discussion, Section
2
outlines the role of a me tho do log in
the software development process. Section discusses th e relationship of software engineering
objectives, pr inciples and attr i bute s to the software development effort . Section identif ies th e
commonly accepted objectives, pr inciples and attr ibutes , defines th e relationships am on g t ien i .
an d then discusses how one evalua tes a methodology based on the those relationships. Finally.
Section describes an application of th e assessment procedure to two Sav y software development
methodologies.
8/10/2019 "A framework for assessing the adequacy and effectiveness of software development methodologies
3/47
2. W h a t C o n s t i t u t e a M e t h o d o l o g y
Fundamental to the research presented in this paper is a common understanding of what
consti tutes a method dogyn. Simply state d, a methodology is a collection of complementary
methods , and a se t of rul s for applying them FREP771. More specifically,
a
methodology
1) organizes and structures the tasks comprising the effort to achieve a global objective,
establishing the relationships among tasks,
(2) defines methods for accomplishing individual tasks (within the framework of the global
objective), and
3 )
prescribes
n
order in which certain classes of decisions are made. and ways of making
those decisions th at lead to th e overall desired objective.
In
general, software development m ethodologies should be guided
by
accepted softw are engineerin s
prina ples th at , when applied to the defined process, achieve a desired goal. ased on this coninloll
understanding of what constitutes
a
methodology, the follo~.ving ections pres ent
a
procedural
approach to evaluating the dequ cy and eflectiveness of software development methodologies.
3 T h e R o le of O b j e ct i ve s , P r i n c ip l e s, a n d A t t r i b u t e s i n S o f t w a r e D e v e l o p m e n t
The development of large, complex software systems is considered a project ac t iv i ty , i~ ivo l v i~l g
several analysts an d programmers and a t least one manager. W hat then is the role of a methodology
in this setti ng and how does it relate to objectives, principles and attrib utes ? Figure assists ill
providing a n answ er to this question.
In general terms, an objective is 'something aimed at or striven for. More specific t o the
software development context, an objective pertains to a project desirable a characteristic tllat
can be definit ively judged only at the co m pl c. .~ ~,f th e project.
3
8/10/2019 "A framework for assessing the adequacy and effectiveness of software development methodologies
4/47
DOCUMP CTATlON
+)
PROGRAMS
Rchcd onplrriy
Wdl Dbmsd ~ L O C S
~rr b i l i~y ATlRIBUTES
Visibility ebtrmr
h l yBm
Figure
Illustration of the Relationships Among Objectives, Principles, Attributes
in the Software Development Process
A
software engineering
principle
describes an aspect of
how
the process of software developm ent
should b e done. T he process of software development, if it is to achieve the st ipul ated obje ctives,
mu st be governed by these 'rules of righ t conduct.
t tr ibutes ar e the intangible characteris tics of the product: the software produced by project
personnel following t he principles set fo rth by th e methodology. Unlike objectives, which per tain
o n ly t o t h e tot l project activity, attributes may be observed in one unit of the product a n d
absent in another . T h e dai m of presence o r absence of an a ttr ibute is based on th e recognit ion
o f
properties which contri bute evidence suppo rting th e claim. Properties are observable. a nd can b e
subjective as well as objective in nature.
8/10/2019 "A framework for assessing the adequacy and effectiveness of software development methodologies
5/47
Influenced by Fritz Ba uer s original defi nition of softwa re engineering
[BAUF72]
and reflecting
the abo ve description cf software engineering objectives, principles an d att ribu tes , t he rationale for
the evaluation procedure described in this paper is founded on the philosophical argument that:
T h e
mison
d etrr
of any software development methodology is the achievement of one
or more objec ivu th rough a
proiress
governed by defined
principles.
In turn. adherence
t o a process governed y those principles should result in a product ( p r o g r a m s a n d
docum entation) that possesses
at t r ibutes
considered desirable and beneficial.
This philosophy, exemplified by Figure 1, is tempered by practical concerns:
(1) While
a
se t of softw are engineering objectives can be identified, this se t might no t be
complete, and additions or modification should be permitted.
2) Objectives can be given different emphasis within a methodology or in applications of a
methodology.
(3) Att ribu tes of a large software product might be evident in one compone nt yet missing in
another.
4 A F r a m e w o r k
f ~ r
v a l u a t i n g S o f t w a r e a e v e l o p m e n t M e t h o d o l og i e s
A broad review
of
software engineering l i tera ture (BERG81,
CHMLSO
G A F J 7 8 , JXChIi5.
PARD79, PA-9D72, SCOL78 WARJIB] leads to the identification of seven objectives com~nonly
recognized in th e num erous methodologies:
1) Maintainability th e ease with which corrections can be made to respond to recogr.ized
inadequacies.
2) Corre ctness stric t adherence to specified requirements,
(3) Reusability he use of developed software in oth er applications,
4) Testab ility he ability to evaluat e conformance with requirements,
5) Relia bility the error-free perform ance of softw are over time.
6) Portab ility the ease in transferring software from one host system to anothe r. and
8/10/2019 "A framework for assessing the adequacy and effectiveness of software development methodologies
6/47
7) AdaptabiLity th e ease with which softw are
c n
accommodate to change.
The authors note that these definit ions,
as
well
s
others presented in this section, are abridged:
they are primarily intended to reflect
a working
understa nding based on general lite ratur e usage.
Achievement of these objectives comes through th e application of principles supported en-
couraged, enforced) by a methoddogy. Th e principles enumerated bdo w are extracted from the
references cited above and others)
as
mandatory in the creative process producing high qualie
programs and documentation.
1) Abstraction defining each program segment at a given level of refinement.
a ) Hierarchical Decomposit ion components defined in a t o p d w n manner.
b)
Functional Decomposition components partitioned along functional boundaries.
2 ) Information Hiding insulating the internal details of component behavior.
3)
Stepwise Refinement utilizing a convergent design.
4) Stru cture d Program ming using a restricted set of control construc ts.
5)
Concurrent Documentation management of supportin g docum ents system sp ec ifia -
tions, user manu al, etc.) thro ugh out the life cycle.
6) Life Cyc le Verification verification of requiremen ts thro ugh out the design. developmenr.
and ma inte nan ce phases of t he Life cycle.
Employment of well-recognized principles should result in software products possessing
z
tribu tes considered to be desirable an d beneficial. short definition of each att rib ute is
ivm
below.
1) Cohesion the binding of stat em ents within a software component.
8/10/2019 "A framework for assessing the adequacy and effectiveness of software development methodologies
7/47
2) Coup ling th e interdependence among sof tware components.
3) Complexity an abstract measure of work assodated with a software component relative
to human understanding and/or machine execution.
4) Well-defined Interfaces th e definitional da rit y and comp leteness of a shared bou ndary
between a pai r of comp onents hardw are or software).
5) Readabil ity th e diff iculty in understanding a software comp onent related to complex ity).
6) Ease of Ch ang e the ease with which software accomm odates enha ncem ents or extensions.
7) Traceability th e ease in retracing the complete history of a software comp onent from its
cur rent s t a tu s t o i t s des ign incept ion.
8) Visibility of. Beh avior th e provision of a review process for e n o r checking.
9) Early E rror Detection indication of faults in requirements specification and design prior
to implementa tion.
T h e software developm ent process, illustrated in Figure 1, depicts
a
natuml relationship that
l inks objectives to p r indp les and pr inciples to at tributes. T h a t is. one achieves the objectives of
a software developm ent m ethodology by applying fundamental principles which. in turn, induce
part icular code and documentation
attr ibutes
From a more detailed perspective. Figure 2 defines
the
specific
se t oflinka ges relating objectives to principles and principles to att ribu tes. -4s described
below, these linkages provide a framework for assessing both th e
adequacy
of a methodology
well as its effectiveness
4.1 ssessing h e dequacy of
a
M e t h o d o l o g y
T h e enunciat ion of objectives should be the f irst step in the definition of aso ftw are development
methodology. Closely
following is the sta teme nt of pr indples t ha t , employed properly, lead to the
attainment of those objectives.
In tur- . the applicat ion of those pr inc iples wi thin a j t r u c t u ~ d
software development process will yield a product th at exhibits desirable at tr ibute s. Th e importan t
8/10/2019 "A framework for assessing the adequacy and effectiveness of software development methodologies
8/47
Early Error Detection
Visibility of Behavior
Well-Defined Inurfacc
Figure
Linkages Among t h e Object ives, Principles and A ttribute s
correspondence between th e object ives an d principles and between th e principles an d at t r ibu tes is
shown in Fjgnre ; a literature confirmation of these relationships is discussed in [. \RTJSi].
T he ade quacy of asof tw are development methodology ca n be defined s its a bility t o achieve tlic
software engin eer kg object ives corresponding to those dictated by system needs and cequirem el~ts .
Intnitiveiy, the adequacy of
a
methodology is assessed through a top-down evaluation sc11e111e
s tar t ing wi th an examinat ion of s tated methoddogical object ives relat ive to system needs and
requirements. This step is then followed by a comparison of stated methodological principles
and at t r ibutes wi th those deemed most appropriate. An exa min ation of linkages defined by th e
e d n a t i o n p r o ce d ur e re ve als t h e most appropriate" set. Relative tr the framework depicted
by
F i v e and he s et s of l inkages defined in Figure 2,
an
application of th e evaluat ion proc edr~ re
o
the assessmezlt of method ological deqwcy is outlined below.
8/10/2019 "A framework for assessing the adequacy and effectiveness of software development methodologies
9/47
Objectives cf the Methodology: The identification of objectives and the relationships tying objec-
tives to needs an d requirements is usually accomplished by reading th e descriptions of software
development methodology. Evaluation a t this level is quite subjective; however, the absen ce of
de r
statement of objectives for a methodology should trigger
an
alarm: Is the 'methodology
only tool or an incomplete set of tools without coherent stru ctur e?
A
methodology should not
be faulte d, however, for emphasizing certain objectives at th e expense of others; such prioritization
can be highly depen dent on t h e application domain.
Principles Defining the Process Based on the objectives emphasized by the methodology and the
predefined se t of linkages am on g objectives and principles, th e next ste p in assessing the adequacy
of a methodology is an nvestigation of the software development process Tha t is, gven a stated set
of methodological objectives, on e asks: Are the principles supported by the methoddogy consistent
with th ose deemed necessary (through linkage exam ination ) to achieve the s tate d set of objecti*;es?
Th e presence of principles without corresponding objective(s) or vice versa should evoke concerns.
.iUthough this level of evaluation is inherently subjective, some analytical qualit:/ is introduced
through t h e established objective/principle correspondence.
Attributes of the Product T h e third st ep in the assessment process, formulating the set of erpected
product attributes, is based on the fact that principles govern the process by which a software
product i s produced. Th at is, given se t of principles should induce a consequent set of product
attribute s. Obviously, the expec ted set of product attr ibut es should correspond to those desired by
the software engineer , and t o some extent, be implied or stated in t he description cif the software
development methodology. Mo re objectivity is introduced a t this level because, although intangible.
evidence of the at tr ibu tes should be discernible in th e product.
4 2
Assessing the ffectiveness
o f
a
M e t h o d o l o g y
While a top-down evaluation process reveals deficiencies of a softw are deve lop ne ~i tmetliod-
ology, the
effectiveness
of a methodology is assessed through a bot tom -up evaluation process. is
8/10/2019 "A framework for assessing the adequacy and effectiveness of software development methodologies
10/47
the term implies, the effectiveness of a methodology is defined as the degree to which a method-
ology produces
a
desired result. In part icul ar, th e etfrctiveness of a metho dology is reflected by
a product s conformance t o the software development process de in ed by th at methodology. We
note, however, that elements independent of the methodology can influence its effectiveness, e.g.
an inadequa te un derstand ing and/or use of the methodology.
The E tence
of
Product Attributes:
Assessing methodological effectiveness starts with an exwni-
nation of the software produc t (cod e and docum entation) for the presence or absence of a t tr ibutes.
Because attr ib utes are intangible, subjective quali ties, the current evaluation is based o a defined
pr c~ er t ie s ha t p rovide evidence s to the presence or absence of attribu tes. More specifically.
the com puta tion of metric values reflect th e exte nt to which particula r properties are observed.
In
tur n, this information is used to synthesize the extant set of product at tr ibutes. Clearly , the
set of at tr ibutes determined from a product evaluation should agree with those induced hy tt.e
corresponding development methodology. Se t mismatch can imply an inapprop riate software de-
velopment methodology, an inadequate understanding of the methodology. or perhaps, the failure
of users to ad here to th e prinap les advocated by t he methodology.
Implied PTinriples and Object ives: Knowing which attributes are present in the product. and
the exte nt to which the y are assessed presen t, provides a basis for imply ing the use of software
engineering principles in t he software development process. Th e rationale for such a state me nt is
based o n the observation th at a p rinciple-to-a ttribute Linkage conversely indicates
an
a t t r ibu te- to -
principle relationship Implying principle usage must be tempered, however, because of the many-
te o n e relationships evist inq between at tr ibu tes and principles. Similarly . using the established
Linkages amo ng objectives and principles, on e can sp eculatr on the achievement of stated software
engineering objectives.
In
summ ary, the thr ee levels of examin ation defined by top-down eva luation process establishes
a procedure for determining how well a methodology can support perceived needs, requirements.
and t he software development process. Conversely, the bot tom -up evd ua tio n process reveals how
Yance
V P
age
1 ol
8/10/2019 "A framework for assessing the adequacy and effectiveness of software development methodologies
11/47
F i g u r e
Lllustration of the Evaluation Process
well the methodology
is
applied in the software deueloprr ent process through the
ns
o ~~uaniitolice
measures to support
an
objective,
qt~al i ta t ive
ssessment.
4 3 An Illustration
of
t h e
Evaluation Process
o illus trate how th e eva1ua:ion scheme can be applied. we r er th e reader t o F i g r e
3
whiie
considering th e single objective of
maintainnbility
Formally, maintainability can be deiined a s :he
ease with which m aintenance can be performed to
a
functional unit in accordance
with
prescribed
requirements. Accep ting maintainability as
n
object ive mandates the indusioa o f x p r i n d p e s
(hierarchical decomposit ion, functional decomposit ion, information hiding, st q w is e ref in em at,
structured programming and concurrent documentation) contr ibuting to the realization of tbat
8/10/2019 "A framework for assessing the adequacy and effectiveness of software development methodologies
12/47
object ive. T h a t is, if methodology em ph siz es maintainabili ty
s
an object ive, then i t should
also stress th e use of t he six principles :hat ar e related to maintainabil ity.
Exp and ing on one of those prinap les, information hiding, we note the five at t r ibu tes reduce d
coupling, enhanced cohesion, well-defined interfaces, ease of change, and low complexity) that
should be evident in software developed using a process governed by the principle of information
hiding. This set of expected a t t rib ute s is the n com pared to the desired set for correspondence.
n general , a methodology sho uld emphasize th e sam e set of software engineering object ives
derived from project level requirements. T h e methodology should correspondingly stress the set of
principles l inked to those object ives. Additionally, th e expected set of product at t r ibu tes defined
by th e linkages among prin aple s an d at t r ibu tes) should agree with the set deemed m ost desirable
by th e project manager. If the above condit ions are met . then the can dida te methodology is
assum ed t o b e a deq uate relat ive to project level , software engineering object ives.
On assessing the effectiveness of
an
methodology, le t us first observe th e relationship between
par t icu lar a t t r ibute
nd
specific pr o d ~ c t haracteris t ics . Reierring again to
Fiqurr? 3 nd
~ ~ a r r o w i i ~ q
ou r atte ntio n t o one of th e attrib ute s, well-defined interfaces, we identify one set of chara cteristics
related t o the well-defined interfaces at t ribu te. The se characteris t ics form the set of ob sert nblr
propert ies which contribute to t he claim th at a piece of software exhibi ts a w ell-defined interface.
Alth oug h th e prop erties shown a r e only subset of those previously identified [.ARTJSG]. they
represent k t confirmin g and co ntras ting eleme nts. For example, the use of global variables ha s
negat ive imp ac t on well-defined interfaces. T h e use of s truc tured d at a in para me ter cal ls . however.
h s
positive impact.
Hence, t o determine th e effect iveness of methodology one assesses the extent t o which p roduct
at t r ibu tes are present or absent ) , and then propagate the resu lt s o i tha t assessment through the
se ts of linkages defined by th e evaluation procedu re. As discussed by Kearney [ K E x J ~ ~ ]owever.
th at assessment process must be predicated o n val idated metrics .
8/10/2019 "A framework for assessing the adequacy and effectiveness of software development methodologies
13/47
5.
A p p l i c a t io n o f t h e E v a l u a t io n P r o c e d u r e
Based on the defined set of linkages among objectives, principles, and attributes, the opera-
tional specification of the evalu ation procedure is guided by two fundamen tal axioms:
1)
the methodology description and project requirements provide standards, conventions,
and gui elines t h a t descri e hour t o
produce
a product, and
(2) the project documentation, code, a nd code documentation reflect ow we l l the develop-
ment process prescribed by the methodology is followed.
A s
described below,
n
application of the evaluation procedure, guided by the above two
oms, illustrates t he utility and intrinsic prower of th e evaluation procedure in assessing the
adequa cy an d effectiveness of a methodology. Provide d in this illustration is a characteriza tion
of the c ompon ents used. in t he evaluation process, an individual assessment of two me th od ol og
descriptions, an analysis of associated products, and a summary of the results. Th e authors
note that a substantial part of the characterization and assessment process is embodied in the
operational aspects of applying the evaluation procedure.
Length restrictions, however, prevent
their discussion. For information on the operationa l aspects the autho rs refer the intereste d reader
t o
[ARTJ86 ARTJ87j.
5.1
ata S o u r c e s
A joint investigatio n of two comparable Navy software development methodologies and respec-
tive produc ts is detailed in [NANR SJ]. Th e investigation effort utilizes:
a
Four so ftware deve lopment m e t h o d o l o ~ ocuments for
1) identifying th e pronounced software engineering objectives. principles. an d a ttribu tej.
and
VPI
age
3of46
8/10/2019 "A framework for assessing the adequacy and effectiveness of software development methodologies
14/47
(2) assessing the adequacy of each methodology through the objective/prinaple/attribate
linkages defined by the evaluation procedure, and
Eight software system documents an d 118 routines, comprising 53 sou rce lines of code,
for
(1) determining th e evident set of product at tr ibute s, and
(2) via the
attribute/principle/objective
linkages, empirically assessing th e principles and
objectives emphasized during product development.
The following section provides
a
summary of the results and i l lustrates the uti l i ty
anc
2rsatiiity
of the procedural approa ch t o evaluating softwa re development methodologies. For simplicity. we
refer to the software systems
as
system A and system B
(and
methoddogy A. methodology B
respectively).
5 2 A n a l y z i n g t h X i e t h o d o l o g i c a l D e s c r i p t i o n a n d A s s o ci a te d P r o d u c t
Th e initial step in the evaluation process is to perform a ' top-down" analysis of rnethodolo gies
and B, t o reveal th e set of claimed software engineering objectives. principles, arid attri bute s.
Because bo th m ethodologies have experienced evolutionary development. a clear s ta tement o f t ie i r
respective methodologicai objectives is lacking. Nonetheless, as detailed in Figure
4
the docu-
mentation for methodology 4 does appear to stress the objectives of rrliability and correctness
supported by the principles of structured progmm ming hiemrchical decom position
d
uncticiml
decomposi t ion Following th e objective/principle relationships defined y the eva lua t ion pnce-
dure. for each objective stressed in methodology
A
only three of t he necessary six principles are
emp hasiz ed. The impli cation is, tha t unless th e principles of Life-cycle i-erification. inform ation
hiding an d stepwise ref inement are implicit ly assumed and utilized, correctness and reliability.
are
compromised.
8/10/2019 "A framework for assessing the adequacy and effectiveness of software development methodologies
15/47
Figure
Pronounced Objectives. Principles,
and
Xttributes
Using metric values and propert ies , a corresponding bot tom-u pn e~ am in at io n f product x
provides some interest ing resul ts . Th e Kiviat graph displayed in Figure ja i l lustrates t h e extent t
which each at t r ibute is
nssessed as
present in the product . (Att ribu te ratings are restricted to
~ Z
arbitrarily chosen 1-10 scale.) Note th at (r educe d) complexity atta ins the highest rating 8 0 out
o
10.0, closely followed by rea dability (7.4) and cohesion (6.8 ). Based o n th e thr ee principles stressed
i n m e t h o d o l o g
A,
th e evaluation procedure predicts th at (red uce d) complexity, readability. and
cohesion should, in fac t , be am ong the product a t t r ibutes .
n
concert with the stated objectives and principles for methodology A, f ig ure 5b reveal s tha t
s t ructured programming (7.7) is the prominent pnnaple used in developing system
A
fdlowed
by stepwise refinement (6.i), hierarchical decomposition 6.41,
and
functional decomposition 6.4).
Figure 5c depic ts th e results of emphasizing these principles in ter ms of methodology objectives.
In
8/10/2019 "A framework for assessing the adequacy and effectiveness of software development methodologies
16/47
Visibility Complexity
of Bchrvior
Interface
ase
of
Readability
Be
Hiemchid
Dccompositim
lmct id
Decomposition
Hid
Progmmhg R e f i e n t
Mainlainability
Figure
Detected Presence of Objectives, Principles, a nd Attri hu tes
particular, reliability is rated as the major software development objective 6.7). . i l though correct-
ness
s
also
stressed by methodology A, ascertaining correctness necessitates life-cycle verification.
This prin dp le is neithe r emphasized by methodology A, nor evident in th e software product . Is
i l lustrated by Figures Sa, 5b a nd 5c, other objectives an d pr inciples are given some em phas is du rin g
th e software developm ent process for system A. It is the auth ors opinions. however. t h a t because
they a re n ot explicit ly stressed in methodology A, the a ssod ated product suffers.
For methodology B, the objectives enun da ted in th e docu mentation are maintai~nabilif
udapl
ability, and reliability. Structured prcgmmming and concumnt documenta t ion
re
t h e e m p k i z e d
principles. Like methodology A however, a complete set of supporting principles are not -tared.
8/10/2019 "A framework for assessing the adequacy and effectiveness of software development methodologies
17/47
Hiemrchicnl decomposit ion, junctional decomposition, and to som e extent injon naiio n
hi ing
are
implici t ly assumed
as
underlying principles of methodology B. According to the l inkages among
objectives and pr inciples, l of the above pr inciples (both stated and assumed) are required
to
achieve the objectives exp liat l y state d in methodology B.
Subsequent analysis of product B and a bottom-up n propagation of the results through the
Linkages defined by th e evaluation procedure reveals stru ctu red prog ramm ing
as
the m ost prominent
principle
8.3) ,
closely followed by concurrent documentation
7.0).
hforeover, the evaluation also
indicates tha t the im plia t ly assumed pr inciples of methodology
B
are utilized ste pu ise refinement.
hierarchical decomposition, functional decomposition, and information hiding rate 6.9, 6.7.6.7 and
6.3
espectively. Finally, th e results imply th at d uring th e development of produ ct
B
the objectives
of maintainabil i ty, adaptabil i ty, and rel iabil i ty are most emphasized. The above assessments are
i ll u s tr a te d i n F igure s 5 4 5b, and 5 ~ .
To summarize, the evaluation procedure reveals th at both m ethodologies lack
a
d e a r s t at e m en t
of goals
and
objectives, as well
as
sufficient principles for achieving the ohjectives tha t a re em -
phtsized. hioreover. glaring deficieacies are apparent in both software development methodologies.
T h a t is, both fail to actively sup po rt the principle of information hiding an d also hav e difficulties
in incorporating the desirable at tr ibutes of traceabil i ty and well-defined interfaces in respective
system products. In general , the evaluation procedure does accurately assesses the sof tware engi-
neering
cbjectives, principles, and attri bu tes espoused by methodologies and B. Of particular
significance, however, is tha t the objectives an d principles determined to be 'emphasized du rin g
th e product development process, yet not state d in the methodology docum entation. are precisely
those tha t ar e implici t ly assumed imp ortant by the sof tware engineers developing products Iand
B.
A more detailed account of the evaluation can be found in [NXNR85].
0 onclusion
T d . techniques, environments, and methodologies dominate the sof tware enginer iny: l i t -
erature, but relat ively l i t t le research in the evaluation of nethodologies is evident . T 5 s work
8/10/2019 "A framework for assessing the adequacy and effectiveness of software development methodologies
18/47
reports
an
init ial at tempt to develop a procedural approaLh io evaluating software development
methodologies. Prom inent i n this approach are:
1)
a n explication of role of a methodo logy in th e software development process,
(2)
the development of a procedure based on linkages among objectives. principles, and
at t r ibu tes , and
3) t,he establis hme nt of basis for reduction of the subjec tive natu re of th e evalua tion thro ugh
th e introdu ction of properties.
in applica tion of the evaluation pro cedure to two Navy methodologies has provided consistent
results t h a t demo nstrate the u ti l i ty and versatil i ty of the evaluation procedure NANRSS]. Curren t
research efforts focus on the continued refinement of th e evaluation procedure thr oug h
(a ) th e the identif ication and integration of produc t quali ty indicators reflective of attribute
presence, and
( b ) t h e
ccllid tion
of metri cs sup por ting the measure of those indicators.
T he consequ ent refinement of the evaluation procedure offers promise of
a flexible approach that
admits t o change as the field of knowledge ma ture s. In conclusion, the procedural a ppr oac h
presented in this pap er represents a promising path toward th e end goal of objectively evalu ating
softvare engineering methoddogies.
ism
VPI
age
8
d 6
8/10/2019 "A framework for assessing the adequacy and effectiveness of software development methodologies
19/47
References
[ALFMS] ALford, M., SREM t the age of
Eight;
T h e Distributed Computing Design System."
*
IEEE Computer
V d .
18, No. 1 Ap ril 1985, pp. 36-54.
(ARTJ861 Art hur , J.D., Nance, R E . and Henry, S.M., "A P rocedural -1pproach to Ev alu atin g So ft-
w re
Development Methodologies: T h e Foundation," T ech nial Report T R - S 2 4 , Virginia
Tech, 1986.
[ARTJ87 Ar thu r, J.D. and Nance, R.E., "Developing an Automated Procedure for E~alua t ing
S o ft xa e Development Methoddogies and Assodated Products. ' T e ch n id Rep ort SRC-87-
007, Systems
Research
Center,
Virginia
Tech, 1987.
[BAUF72] Bauer, F.L. "Software Engineering," Information Pmcesing 71 North Holland Pub-
lishing Company, 1972.
(BERG811 Bergiand, G.D. A Guided Tour of Program Design Me & od ol og id Computer
v01.
14, Xo. 10, Oc tob er 1981, pp . 13-36.
[CHMLSO] Chmu ra, L.J. Norcio, A.f., and Wi ans ki, T.J., 'Evdu atiag Software Design Processes
by Analyzing Change Data Over Time." IEEE T m n s ~c t i o n s n S o f t w a y
Engineennc.
Val
16
No. 7,
J u l y
1990, pp. 729-740.
[FR EP 77] Freeman, P., "The Nat ur e of Design," -4 Tutorial on S o f tw m D esign Techniques.Second
edition, IEEE Computer Society Press, 1977,
pp.
29-36.
[ G A F J a l ] G d e n e y ,
J.
E.,
"Metrics in S oftw are Quality Assurance,'
Proceedings of
t h e
.Yational
AC U Conference November 1981. pp. 126-130.
[GOMHM]Gona a , H. "A Software Design Method for Real-Time S -stems," Communications of
the ACM ol. 27, No. 9, Sep tem be r 1984, pp. 938-9-19.
[HE NK 78] Heninger, K. L., J. W. Kallander, J. E. Shor e, and D. L. Parna s. %o ftw are Require-
ments for the A-7E Aircraft," NRL Me mora ndu m Report 3876. Yaval Research Labocztory,
Washington, D. C., November, 1978.
[JACM75] Jackson, M., Principles of rogram Design London: Academic Press. 1975.
[KEAJ86 Kearney,J., el .
al. 'Software Com plexit y Measurement.' C o m m n n i m t i o m of
fhe
A.C..U. Vo1.29, No. 11, Nov ember 1986, pp. 1044-1050. Yo ntere y,
CX
March 19.37.
pp.
238-252.
R
S-
VPI
1 9 d Y
8/10/2019 "A framework for assessing the adequacy and effectiveness of software development methodologies
20/47
[LISB75]
Liskov,
B.
Zlles, S., 5pecification Techniques For Data Abstraction,"
IEEE
Tmnsm-
t i o m o n S of tw ar e E n g i n ec h g
Vol. S E -I , NO.
1
March 1975, pp. 7-19.
[ N A N W ] N ance,
RE.
Arth ur, J.D . and Dande kar, A.V. "E valuation of Software Development
Methoddogies,"
X
Final R eport of t he Immediate Software Development Project, T h e
De-
partment of Computer Saences Virginia Tech, December 1985.
.
[PAR 376] P arn as,
D.
"On the Design and Development of Program
Families, IEEE Tmnsactiorrci
on Software Engineering V d SE-2,
NO. 1
March 1976, pp. 1-9.
(PARD721 Pa rna s, D.. "On the Criteria to be Used in Decomposing Systems into Modules.'
Communimtions
of
the A m ,
ol. 15,
NO.
5, May 1972, pp. 330-336.
[ROSD77] Ross D., "S truc ture d Analysis: A Language for Com mu nicati ng Ideas," I E E E T m m -
actions on Software Enginem'ng, Vol. SE-3, No. 1 January, 1977, pp. 16-34.
[SCOL78] Scott ,
L. .h
Engineering Methodology for Presenting Softw are Functional
.-\rchitec-
ture, Proceedings ofthe hird Internotiona Conference on Software Engineeriny YY
973.
pp.222-229.
[WARJ76] Warnier, J. Logical
C m l r u c t i o n o f
Pmgmms 3rd edition, trans . B. Flanagan, YY:Van
Nostrand Reinhold, 1976.
[W IR N 71 ] W ~ r t h ,
.
*Program Development by Stepwise Refinement,"
Comm unications of
the
ACM
Vol. 14, No.4, Xpril, 1971, pp. 221-227.
Y a u
VPI
Rge dU
8/10/2019 "A framework for assessing the adequacy and effectiveness of software development methodologies
21/47
VIEWGR PH
M TERI LS
FOR
THE
R
N NCE
PRESENT TION
8/10/2019 "A framework for assessing the adequacy and effectiveness of software development methodologies
22/47
FR MEWORK
FQR
SSESSING THE
DEQU CY ND EFFECTIVENESS OF
SOFTW RE DEVELOPMENT METHODOLOGIES
Presentation
to
the
Fifteenth nnual Software Engineering Workshop
Richurd
E
Nance
James D rthur
Systems Research C enter
and
Department of Computer Science
Virginia Polytechnic
Institute
and State University
Blacksburg Virginia
28
November 199
PRECEDING
P GE BL NK OT FILMED
8/10/2019 "A framework for assessing the adequacy and effectiveness of software development methodologies
23/47
TH ORI IN
Immediate Software Development Issues
for
Embedded Systems Applications
in
Surface Combatants
(25 March 5September 1985)
Issue: Multiple Software Development Methodologies
(a)
Review
two software development methodologies A and B)
b)Compare
and
evaluate
A and
c )
Assess
the costs and benefits of:
Continuing witn multiple sofnvare development
methodologizs
Using only one software development methodology
Transitioning to an alternate s o f ~ a r eevelopment
me hodology
8/10/2019 "A framework for assessing the adequacy and effectiveness of software development methodologies
24/47
OUTLINE
Evaluation Approach
Objectives Principles Attributes Framework
Developm ent of
an
Evaluation Procedure
Software Engineering and Software Development
Structured Evaluation Procedure
Data Sources
Application of th Evaluation Procedure
Summary of Sam ple Data
Illustration of Procedure Application
Summary of Results
Future Work
8/10/2019 "A framework for assessing the adequacy and effectiveness of software development methodologies
25/47
EV LU TION PPRO CH
1
Develop
n
Evaluation Procedure
hat is a methodology?
How
can they be
compared?
2 pply
the
Evaluation Procedure
In consonance with our Navy sponsor and with
Contributions from software development sites
and
oversight agencies.
8/10/2019 "A framework for assessing the adequacy and effectiveness of software development methodologies
26/47
O METHODOLOGES
W hat is a methodology?
methodology is
a
collection of complementary methods,
an d a set of rules for a.pplying them. More specifically, a
methodology
1) organizes and structures tasks comprising the effort to
achieve a global objective, establishing the
relationships am ong tasks,
2) defmes methods for accomplishing individual tasks
within the context of the global objective),
and
3) prescribes an order in
which
certain classes of
decisions are made,
and
ways of making those
decisions that lead to the desired objective.
8/10/2019 "A framework for assessing the adequacy and effectiveness of software development methodologies
27/47
RATIONALE FOR
TH EVALUATION
PROCEDURE
A set of ob ie ct iv es can be identified that include those
postulated by any software engineering methodology.
A
methodology defines those principles that characterize a
proper and appropriate development process. Adherence to a
process governed by these principles should result in a
prod uct program s and docum entation) that possesses
attributes considered desirable and beneficial.
Philosophy tempered by practical concerns:
1) Sets of objectives, principles, attributes areopen.
2) Prioritization of objectives recognized.
3)
Com ponents of large software system vary attribute
sampiing.
4)
Flexible application
of
evaluation procedure
consonant with project objectives.
8/10/2019 "A framework for assessing the adequacy and effectiveness of software development methodologies
28/47
FRAMEWORK FOR SOFTWARE DEVELOPMENT
O
ECTNES
Maintainability
Correctness
Reusability
Testability
Reliability
Portability
Adaptability
E Imaus
Hierarchical Decomposition
Functional Decomposition
Information
Hiding
Stepwise Refinement
Structured
rog raming
Life Cycle Verification
Concurrent Documentation
llln m
Reduced
Coupling
Enhanced Cohesion
Reduced Complexity
Well Defmed Interfaces
Readability
Ease of Change
Traceability
Visibility o Behavior
Early E m r Detection
PROCESS
PRODUCT
DOCUMENTAnON +) PROGRAMS
Properties
r n u T E S
Y a m
VP l
p ge l
8/10/2019 "A framework for assessing the adequacy and effectiveness of software development methodologies
29/47
PROCEDURE DEVELOPMENT
1 Identlfy Objectives
What qualities are desirable
2
Define Principles
How
are desirable qualities obtained
3 ink Principles to Objectives
Which principles contribute to each objective
4 Defme Resulting Attributes
Use of a principle induces what desirable attributes
5
Defme Properties Associated with Attributes
What properties give evidence of attribute presence or
absence
How to
measure properties
8/10/2019 "A framework for assessing the adequacy and effectiveness of software development methodologies
30/47
PRIMARY SOFTWARE
ENGINEERING
OB.TECTM3S
1)
Adaptability -
the ease with which software can
I
accomm odate to changing requirements
2) Correctness -
strict adherence to specifications
I
3)
Maintainability
-
the ease with which corrections
can e made to respond to recognized
inadequacies
I
4) Portability -
the ease
in
transferring software
to
another host environment
5)
Reliability
-
the error free behavior of software over
time
6)
Reusability -
the use of developed software
in
other
.
applications
.
4
7) Testability -
the ability to evaluate conformance
.
with specifications
8/10/2019 "A framework for assessing the adequacy and effectiveness of software development methodologies
31/47
PRIMARY SOFTWARE
ENGINEERING
PRINCTPLES
1)
Abstraction
defming each program segment at a
given level of refmement
a)
Hierarchical ecomposition
- components
defmed in a top-down manner
@ Functional ecomposition
components
partitioned along functional boundaries
2) Concurrent Documentation -management of
supporting documents system specifications, user
manuals, etc) throughout the life cycle
3)
Information Hiding
insulating the internal details
of com ponent behavior
4) Life Cycle Verification -
verification of
requirements throughout the design, development,
and maintenance phases of th life cycle
5) Stepwise Refinement utilizing convergent design
6) Structured Programming .:sing a restricted set of
program control constructs
8/10/2019 "A framework for assessing the adequacy and effectiveness of software development methodologies
32/47
OBJECTIVES PRINCIPLES LINK GES
daptability
Correctness
Maintainability
Portability
Reliability
Reusability
Testability
Concurrent Documentation
Functional Decomposition
Hierarchical Decomposition
Information Hiding
Life Cycle Verification
epwise Refinement
Structured
Programming
8/10/2019 "A framework for assessing the adequacy and effectiveness of software development methodologies
33/47
PRIM RY SOITWARE
ENGINEERING
ATTRIBUTES
1)
Cohesion The binding of statements within a
software component
2)
Complexity an abstract measure of work
associated with a software component
3)
Coupling the interdependence among software
components
4)
Early Error Detection
indication of faults
in
requirements, specification and design prior to
implementation
5 )
Ease of Change software that accommodates
enhancements or extensions
6)
Readability
the difficulty in ~nderstanding
software component
7) Traceability
the ease
in
retracing the complete
history of a software component from its current
status to its design
8) Visibility
of
Behavior
the
provision of a review
process for error checking
9)
Well-Defined Interfaces
the definitional clarity
nd
completeness of a shared boundary between
software and/or hardware software/software,
softw arehardware)
8/10/2019 "A framework for assessing the adequacy and effectiveness of software development methodologies
34/47
PRINCIPLES
TTRIBUTES
LINK GES
Documentation
Early
Enor Detection
Information Hiding
Well Defined Interfaces
8/10/2019 "A framework for assessing the adequacy and effectiveness of software development methodologies
35/47
ILLUSTR TION O
TH
EV LU TION PROCEDURE
Use
of lobal
1
Variables
xcessive
of arameters
Use
ofData
8/10/2019 "A framework for assessing the adequacy and effectiveness of software development methodologies
36/47
SETS
OF
DEFINED
LINK GES
a m
VPl
PalcUd 6
8/10/2019 "A framework for assessing the adequacy and effectiveness of software development methodologies
37/47
THE OPA FRAMEWORK FOR EVALUATION
SUMMARY
Fundm en ta l to the evaluation procedure are several sets of
linkages:
Linkages
Defined
Substantiated
Objectives
/
Principles
33).
33)
Principles
/
Attributes 24)
24)
Attributes
/
Properties
(125)
1 14)
66 Automatable
Assessing the adequacy of a methodology is achieved through a
op-down evaluatior? process.
Assessing the
effe tiveness
of a methodology is achieved through
a bottom-up evaluation process.
R S a m
VPI
Page
6 4 6
8/10/2019 "A framework for assessing the adequacy and effectiveness of software development methodologies
38/47
APPLICATION O F
THE
PROCEDURE
SUMMARY OF SAMPLE DATA
Documents Pr imary
A
The Combat System Developmext Plan
The Computer Programming M anual
The
rogram
Development
anual
SixNumbered Documents PDS, IDS)
Functional Description Document
Two
Numbered Documents PDS, IDS)
Source
Code:
A: Routines 17
SysProcs
2
Routines
99
SLOCS 117
SLOCS
137
SLOCS 5729
8/10/2019 "A framework for assessing the adequacy and effectiveness of software development methodologies
39/47
D T
SOURCES
ND
IMPLIC TIONS
Methodology
Description Standards Objectives
onventions do it Principles
Project Guidelines
Requirements
roject
Documentation
PPS
I S
How well {Principles
P S is it done
ttributes
ode and
Programs
ode Documentation
8/10/2019 "A framework for assessing the adequacy and effectiveness of software development methodologies
40/47
N ACCUMULATION OF
EVIDENCE
Demonstrating that software possesses a desired attribute
(or does not) is not a proof exercise; rather, it resembles
an exercise
in
civil litigation in that evidence
is
gathered
to
support oth contentions (the presence or absence) and
weighed on the scales of comparative judgement.
Offsettin
Measurement Scale
8/10/2019 "A framework for assessing the adequacy and effectiveness of software development methodologies
41/47
ELEMENTS METRICS
AND PROPERTIES
Relationship
Elements
Subjective
-
Code Example
Documentation
Cil)
of Distinct
Parameterless
Calls
Example
Awareness
of
V V
Y2)
of Distinct
Calls
8/10/2019 "A framework for assessing the adequacy and effectiveness of software development methodologies
42/47
ASSESSING METHODOLOGICAL EFFE TIVENESS
(ATTRIBUTES)
Coupling
Cohesion
Complexity
Well Defmed Interfaces
Readability
Ease of Change
Visibility of Behavior
Early Error Detection
Traceability
th
have difficulty with Traceability
and
Well Defied
Interfaces
u t
VP
4 d 6
8/10/2019 "A framework for assessing the adequacy and effectiveness of software development methodologies
43/47
Coupling
Ease of Readability
Change
K M T GR PH FOR
Methodology
Mc~lrodology
8/10/2019 "A framework for assessing the adequacy and effectiveness of software development methodologies
44/47
Hierarchical
Decomposition
oncumnt
/
I
Information
Lifccyclc Hiding
Verification
8/10/2019 "A framework for assessing the adequacy and effectiveness of software development methodologies
45/47
Maintainability
KIVI T
GR PII
FOR O D J E m
Methodology
Methodology
8/10/2019 "A framework for assessing the adequacy and effectiveness of software development methodologies
46/47
R SULTS
O PROCEDURE PPLIC TION
Assessing Methodological Adeauacv
A:
Stresses Objectives of Reliability and Correctness
Emphasizes Principle of Structured Program ming
Methodology A was (and is) an evolving
m e hodology
Stresses Objectives of Mainrainability,
Adaptability, Reliability,
and
Correcmess
Emphasizes Principles of Modular Decomposition,
Structured Programing and Concurrent
Documentation
At the Objectives level, both methodologies support stated
project objectives.
t th
Principles level, both methodologies lack the
enunciation of proper Principle usage.
No reference to desired Attributes is found
8/10/2019 "A framework for assessing the adequacy and effectiveness of software development methodologies
47/47
FUTUR
RESE RCH
pplying the Evaluation Procedure to
Sofbvare Quality ssurance
Predicting and/or assessing software quality necessitates
Systematic
approach to
s s ~ s s ing
product or process) confo rm a~ cewith
cceptance criteria
standards and guidelines)
The Evaluation Procedure
Currentiy
supports
a well-defmed, svstematic
approach for evaluating software products, and
Provides a rigorous framework for
Relating acceptance criteria based on attributes
to software engineering principles and
objectives, and
Defining acceDtance levels based on m easures
reflecting the achievement of objectives,
adherence to principles
nd
realization
of
attributes
top related