Software Engineering and Best Practices Sources: Various. Rational Software Corporation slides, OOSE textbook slides, Per Kroll talk, How to Fail with.

Software Engineering Software Engineering andand

Best PracticesBest Practices

Sources: Various. Sources: Various. Rational Software Corporation slides, Rational Software Corporation slides,

OOSE textbook slides, Per Kroll talk, How to Fail OOSE textbook slides, Per Kroll talk, How to Fail with the RUP article, textbookswith the RUP article, textbooks

Most slides have been modified considerablyMost slides have been modified considerably

What is Software What is Software Engineering?Engineering?

►The process of solving customers’ problems The process of solving customers’ problems by the systematic development and by the systematic development and evolution of large, high-quality software evolution of large, high-quality software systems within cost, time and other systems within cost, time and other constraintsconstraints

►Note:Note: Process, systematic (not ad hoc), evolutionary… Process, systematic (not ad hoc), evolutionary… Constraints: high quality, cost, time, meets Constraints: high quality, cost, time, meets

user requirementsuser requirements

Analysis of the Definition:Analysis of the Definition:► Systematic development and evolutionSystematic development and evolution

An engineering process involves applying well understood An engineering process involves applying well understood techniques in a organized and disciplined waytechniques in a organized and disciplined way

Many well-accepted practices have been formally standardizedMany well-accepted practices have been formally standardized► e.g. by the IEEE or ISOe.g. by the IEEE or ISO

Most development work is Most development work is evolutionevolution ► Large, high quality software systemsLarge, high quality software systems

Software engineering techniques are needed because large Software engineering techniques are needed because large systems cannot be completely understood by one personsystems cannot be completely understood by one person

Teamwork and co-ordination are requiredTeamwork and co-ordination are required Key challenge: Dividing up the work and ensuring that the parts Key challenge: Dividing up the work and ensuring that the parts

of the system work properly togetherof the system work properly together The end-product that is produced must be of sufficient qualityThe end-product that is produced must be of sufficient quality

► Cost, time and other constraintsCost, time and other constraints Finite resourcesFinite resources The benefit must outweigh the costThe benefit must outweigh the cost Others are competing to do the job cheaper and fasterOthers are competing to do the job cheaper and faster Inaccurate estimates of cost and time have caused many Inaccurate estimates of cost and time have caused many

project failuresproject failures

Comments:Comments:► $250 billion annually in US.$250 billion annually in US.► Over 175,000 projects!Over 175,000 projects!► Complexity, size, distribution, importance Complexity, size, distribution, importance

push our limits.push our limits.► Business pushes these limits:Business pushes these limits:

Great demands for rapid development and Great demands for rapid development and deploymentdeployment

► Incredible pressure to develop applications Incredible pressure to develop applications that are:that are: On time, Within budget, Meets the users’ On time, Within budget, Meets the users’

requirementsrequirements► Figures in the late 90s indicated that at mostFigures in the late 90s indicated that at most

70% of projects completed70% of projects completed Over 50% ran over twice the intended budgetOver 50% ran over twice the intended budget $81 billion dollars spent in cancelled projects!!$81 billion dollars spent in cancelled projects!!

► We are doing something wrong as an We are doing something wrong as an industry!industry!

What Happens in PracticeWhat Happens in PracticeSequential activities: (Traditional ‘Waterfall’ Process)

Requirements Design Code Integration Test

Late DesignBreakage

100%

Project Schedule

Dev

elop

men

t Pro

gres

s(%

cod

ed)

OriginalTarget Date

IntegrationBegins

Risk inadequately addressedProcess not receptive to ChangeProblems not really ‘seen’ until near delivery date! Until then, ‘all is well…’Big Bang approach – full deliveryLong development cycles…Little user involvement, etc. etc…

Symptoms of Software Symptoms of Software Development ProblemsDevelopment Problems

► Inaccurate understandingInaccurate understanding of end-user needs of end-user needs► Inability to deal with Inability to deal with changing requirementschanging requirements► Modules that don’t fit together (integration)Modules that don’t fit together (integration)► Software that’s hard to maintain or extend (brittle)Software that’s hard to maintain or extend (brittle)► Late discovery of Late discovery of serious project flawsserious project flaws (integration) (integration)► Poor software qualityPoor software quality (architecture, risks (architecture, risks

unanticipated…)unanticipated…)► Process Process not responsive to Change (Gantt Charts…)not responsive to Change (Gantt Charts…)► Unacceptable software performance (Risk, architecture, Unacceptable software performance (Risk, architecture,

…)…)► Team members in each other’s way, unable to Team members in each other’s way, unable to

reconstruct who changed what, when, where, why reconstruct who changed what, when, where, why (software architecture, …(software architecture, …

► ……and we could go on and on…and we could go on and on…

► We need a We need a processprocess that that Will serve as a framework for large scale and small Will serve as a framework for large scale and small

projectsprojects Adaptive – embraces ‘change!’Adaptive – embraces ‘change!’

► Opportunity for improvement not identification of failure!Opportunity for improvement not identification of failure! Iterative (small, incremental ‘deliverables’)Iterative (small, incremental ‘deliverables’) Risk-driven (identify / resolve risks up front)Risk-driven (identify / resolve risks up front) Flexible, customizable process (not a burden; Flexible, customizable process (not a burden;

adaptive to projects)adaptive to projects) Architecture-centric (breaks components into Architecture-centric (breaks components into

‘layers’ or ‘layers’ or common areas of responsibility…) common areas of responsibility…) Heavy user involvementHeavy user involvement

► Identify best ways of doing things – a better Identify best ways of doing things – a better process – acknowledged by world leaders…process – acknowledged by world leaders…

Need a Better Hammer!

Develop IterativelyDevelop Iteratively

Control ChangesControl Changes

Use Use ComponentComponent

ArchitecturesArchitectures

Manage Manage RequirementsRequirements

Model Model VisuallyVisually

VerifyVerifyQualityQuality

Best Practices of Software Best Practices of Software EngineeringEngineering

Know these!

Symptomsend-user needs

changing requirements

modules don’t fit

hard to maintain

late discovery

poor quality

poor performance

colliding developers

build-and-release

Root Causesinsufficient requirements

ambiguous communications

brittle architectures

overwhelming complexity

undetected inconsistencies

poor testing

subjective assessment

waterfall development

uncontrolled change

insufficient automation

Best Practicesdevelop iteratively

manage requirements

use component architectures

model the software visually

verify quality

control changes

Addressing Root Causes Addressing Root Causes Eliminates the SymptomsEliminates the Symptoms

Practice 1: Develop Software Practice 1: Develop Software Iteratively Iteratively

Develop IterativelyDevelop Iteratively

Control Changes

Use Component

Architectures

Manage Requirements

Model Visually

VerifyQuality

Considered by many practitioners to be the most significant of the six

Practice 1: Develop Software Practice 1: Develop Software IterativelyIteratively

►Until recently, we were developing Until recently, we were developing systems under the assumption that most systems under the assumption that most requirements can be identified up front.requirements can be identified up front.

►The research deconstructing this myth The research deconstructing this myth includes work by Capers Jones. (See next includes work by Capers Jones. (See next slide) In this very large study of 6,700 slide) In this very large study of 6,700 projects, projects, creepingcreeping requirements requirements — those — those not anticipated near the start—are a not anticipated near the start—are a very very significant fact of software development significant fact of software development lifelife, ranging from around 25% on average , ranging from around 25% on average projects up to 50% on larger ones.projects up to 50% on larger ones.

Interestingly, Interestingly, ► An initial design will likely be flawed with respect to its An initial design will likely be flawed with respect to its

key requirements. Requirements rarely key requirements. Requirements rarely fully knownfully known up up front!front!

► Late-phase discovery of design defects results in costly Late-phase discovery of design defects results in costly over-runs and/or project cancellation over-runs and/or project cancellation Oftentimes requirements change – during development!Oftentimes requirements change – during development!

► While large projects are more prone to cost overruns, While large projects are more prone to cost overruns, medium-size/small projects are vulnerable to medium-size/small projects are vulnerable to cancellation. cancellation.

► The key reasons continue to be The key reasons continue to be poor project planning and management, poor project planning and management, shortage of technical and project management expertise, shortage of technical and project management expertise, lack of technology infrastructure, lack of technology infrastructure, disinterested senior management, and disinterested senior management, and inappropriate project teams.”inappropriate project teams.”

Waterfall Delays RisksWaterfall Delays Risks

R

I

S

K

T I M E

IntegrationSystem

Test

Code

Design

Requirements

Waterfall risk

Walker Royce, 1995

Iterative DevelopmentIterative Development

• Earliest iterations address greatest risks • Each iteration produces an executable release• Each iteration includes integration and test

Iteration 1 Iteration 2 Iteration 3

Accelerate Risk ReductionAccelerate Risk Reduction

Iterative

T I M E

Iteration Iteration Iteration Iteration Iteration

Risk reductionRisk reduction

R

I

S

KWaterfall risk

Walker Royce, 1995

Iterative Development CharacteristicsIterative Development Characteristics► Critical risks are resolved before making Critical risks are resolved before making

large investmentslarge investments ► Initial iterations enable early user feedbackInitial iterations enable early user feedback

Easy to resolve problems early. Easy to resolve problems early. Encourages user feedback in meaningful waysEncourages user feedback in meaningful ways

► Testing and integration are continuousTesting and integration are continuous – – assures successful integration (parts all fit) assures successful integration (parts all fit) Continuous testing.Continuous testing.

► Objective milestones provide short-term focusObjective milestones provide short-term focus► Progress measured by assessing Progress measured by assessing

implementationsimplementations► Partial implementations can be deployedPartial implementations can be deployed

Waterfall method – no deliveryWaterfall method – no delivery Incremental development? May be some great Incremental development? May be some great

values in delivering key parts of application. values in delivering key parts of application. Critical components delivered first?Critical components delivered first?

► No big-bang approach!No big-bang approach!

Iterative Lifecycle GraphIterative Lifecycle Graph

In an In an iteration, , you may walk you may walk through all through all disciplinesdisciplines

CONTENT

STRUCTURE

T I M E

Executable Releases

RUP Iterations and PhasesRUP Iterations and Phases

An iteration is a distinct sequence of activitieswith an established plan and evaluation criteria,

resulting in an executable release.

PreliminaryIteration

Architect.Iteration

Architect.Iteration

Devel. Iteration

Devel. Iteration

Devel. Iteration

TransitionIteration

TransitionIteration

Elaboration Construction TransitionInception

Enables and encouragesEnables and encourages user user feedbackfeedback

Serious Serious misunderstandingsmisunderstandings evident early in the life cycleevident early in the life cycle

Development focuses on Development focuses on critical issues – break it down!critical issues – break it down!

Objective assessment thru Objective assessment thru testing testing

Inconsistencies detected earlyInconsistencies detected early

Testing starts earlier – Testing starts earlier – continuous!continuous!

Risks identified and addressed Risks identified and addressed early - via planned iterations!early - via planned iterations!

Problems Addressed by Iterative Problems Addressed by Iterative DevelopmentDevelopment

Root CausesRoot Causes SolutionsSolutions Insufficient requirementsInsufficient requirements Ambiguous Ambiguous

communicationscommunications Brittle architecturesBrittle architectures Overwhelming complexityOverwhelming complexity Subjective assessmentSubjective assessment Undetected Undetected

inconsistenciesinconsistencies Poor testingPoor testing Waterfall developmentWaterfall development Uncontrolled changeUncontrolled change Insufficient automationInsufficient automation

No Free LunchNo Free Lunch - Traps - Traps Abound…Abound…

► Major impacts on Project Managers, though….Major impacts on Project Managers, though….► Trap: When the initial risks are mitigated, new ones emergeTrap: When the initial risks are mitigated, new ones emerge Do not do just the easy stuff, to look good.Do not do just the easy stuff, to look good. Keep re-planning based on all new information.Keep re-planning based on all new information.► Trap: Remember that ‘some’ Rework enables you to enhanceTrap: Remember that ‘some’ Rework enables you to enhance your solutionyour solution Accommodate change early in the projectAccommodate change early in the project► Trap: Iterative development Trap: Iterative development does not meandoes not mean never to commit never to commit

to commit to a solutionto commit to a solution► Monitor ‘scrap and rework’Monitor ‘scrap and rework’► Trap: Must Control “requirement creep, ” however… SomeTrap: Must Control “requirement creep, ” however… Some clients will now naturally recognize many ‘musts…’clients will now naturally recognize many ‘musts…’

Many Traps in Iterative Many Traps in Iterative DevelopmentDevelopment

Here is another trap: Too long initial iterationHere is another trap: Too long initial iteration

► Winning is fun. A winning team works better than a Winning is fun. A winning team works better than a loosing teamloosing team

► Better to have a short initial iteration, than a too longBetter to have a short initial iteration, than a too long Cut scope if necessaryCut scope if necessary

► Avoid ‘analysis-paralysis’ by Avoid ‘analysis-paralysis’ by time-boxintime-boxing, you can g, you can enhance in later iterations (more later)enhance in later iterations (more later)

► Establish an even rhythm for project (at least within a Establish an even rhythm for project (at least within a phase)phase)

► Not completing iterations makes you lose most of the Not completing iterations makes you lose most of the benefitbenefit

► Focus on results and deliverables, not activitiesFocus on results and deliverables, not activities

Problem: Problem: Dangerous Iteration Dangerous Iteration PatternsPatterns

Teams not synchronized -> chaos

First iteration too loaded -> panic

Too much overlap -> no feedback loop

Problem: Fixed Plans Problem: Fixed Plans Produced UpfrontProduced Upfront

►Trap: Fine grained planning from start to Trap: Fine grained planning from start to endend

►Trap: Too much uncertainty makes Trap: Too much uncertainty makes detailed plans for the entire project detailed plans for the entire project meaningless (also not true!)meaningless (also not true!)Does not mean that we should not planDoes not mean that we should not planEstablish milestones and track progress Establish milestones and track progress relative to milestonesrelative to milestones

Solution: Plan With Evolving Solution: Plan With Evolving Levels of Detail – Be Levels of Detail – Be

Careful!!!Careful!!!

Current Iteration

Next Iteration

Phases and major milestones What and whenIterations for each phase Number of iterations Objectives and Duration

One For Entire Project

Fine-grained Plans: Iteration Plans

Coarse-grained Plan: Software Development Plan

Iterative does not mean less work and shorter schedule It is about greater predictability It is within the fine-grained plans that we have predictability…

Iterations Are Time-boxedIterations Are Time-boxed

►Work is undertaken within an Work is undertaken within an iteration.iteration.

►The iteration plan defines the The iteration plan defines the artifacts to be delivered, roles and artifacts to be delivered, roles and activities.activities.

►An iteration is clearly measurable.An iteration is clearly measurable.►Iterations are RISK DRIVENIterations are RISK DRIVEN

The Iteration Plan Defines….The Iteration Plan Defines….

TheThe deliverables for deliverables for that iteration.that iteration.

The to do list for the The to do list for the team membersteam members

Project progress is made Project progress is made against MILESTONESagainst MILESTONES

►Each phase is defined by a Each phase is defined by a milestonemilestone..

►Progress is made by passing Progress is made by passing milestonesmilestones..

►PhasesPhases - NOT TIMEBOXED. - NOT TIMEBOXED.►IterationsIterations ARE TIMEBOXED. ARE TIMEBOXED.►Milestones measure successMilestones measure success

Inception Elaboration Construction Transition

Major Milestones