Week 5 – Special Topics

Week 5 – Special Topics

Risk Management Best Practices

Traps, Alarms & Escapes From Navy Best Practices

Traps - think have risks covered by following procedures

Alarms – assumptions that cause trouble

Consequences – if nothing is done

Risk Checklist

• Personnel • Availability • Experience Levels • Mix of Disciplines • Requirements • Definition • Stability • Complexity • Resource Availability • Facilities • Hardware • Personnel • Funding Profile • Communications • Technology • Availability • Maturity Levels • System Complexity • Operating Environment • Design • Methods • Complexity • Software Tools • Testing, Modeling • Language • Operational Interfaces • Hardware • Sensitivities to Other Risks • Estimating Error • Schedule • Cost • Number of Critical Path Items

Potential Risk Item

Risk Management Training

Why RM Training is Needed Weak backgrounds in risk

management Mistaken concepts

Assessment is RM – skip planning & monitoring

Mitigation is only handling strategy All risk can be eliminated Focus on performance, omit cost &

schedule

Tailoring RM Training Adjust to different project team groups

Sr. management Working level engineers

Address issues each group is likely to face

Address tailoring RM activities to meet program needs – not one-size fits all

Software Risk Management

SW Risk as a Special Case Virtual

Can’t physically touch/feel to assess History of latent defects

Many interfaces Hardware-to-software Operating system-to-operating code Incompatible combinations

Update frequency Multiple versions Upward/downward compatibility

Taxonomy of SW RiskRisk Grouping Risk Issue

Project Level Requirements - excessive, immature, unstable, unrealisticLack of user involvementUnder estimation of complexity or dynamic nature

Project Attributes Performance - errors, qualityUnrealistic cost or schedule

Management Ineffective project management

Engineering Ineffective integration, assembly, testUnanticipated difficulties across user interface

Work Environment Immature design, process, technologiesInadequate configuration controlInappropriate methods, inaccurate metricsPoor trainiing

Other Legal, contractual issuesObsolescence (incl. excessive duration)Difficulties with subcontracted itemsUnanticipated maintenance & support costs

Boehm’s Top 10 Software Risks

Risk Item Management Techniques

Architecture, performance,quality

Architecture trade-off analysis, simulations,benchmarking, modeling, prototyping

Requirements changesChange thresholds, incrementaldevelopment, information hiding

Legacy softwareRestructuring, design recovery, wrappers,phase-out analysis

Externally-performed tasksReference checking, pre-award audits,award fee contracts, competitive design orprototyping, team building

Straining computer sciencecapabilities

Technical analysis, cost benefit analysis,prototyping

Boehm’s Top 10 Software Risks

Identification Strategies Review Schedule & Networks Review Cost Estimation Parameters Perform Interviews Revisit Lessons Learned Develop a Risk taxonomy Brainstorm and play “what if” Re-sort the watch list (e.g., by source)

Identification- Schedule Risks

Use the planning or baseline schedule Evaluate the activity network view Look for nodes with:

High fan-in (many activities terminate at a single node)

High fan-out (many activities emanate from a single node)

No predecessors

ID- Cost Risk Drivers

Consider specific areas of concern that can lead to problems: Personnel experience, availability Requirement complexity, firmness Scheduling and prediction of task and

partition times Hardware requirements, interfaces,

constraints

0.1Low

0.3Moderate

0.5High

0.7Very High

0.9Extremely

High

StabilityFactorMaturity Factor

ComplexityFactor

Technology exists andcan be used “as is”

Technology requiresminor change beforeuse (<25%)

Technology requiresmajor change beforeuse (<50%)

Technology requiressignificant design andengineering beforeuse (<75%)

State of the art, someresearch done

Simple relative tocurrent environment

DependencyFactor

Minor complexityrelative to currentenvironment

Moderately complexrelative to currentenvironment

Significantly complexrelative to currentenvironment

Extremely complexrelative to currentenvironment

Entirely within project control

Depends on existingproduct supplied fromoutside organization

Depends on supplyand modification ofexisting product fromoutside organization

Depends on newdevelopment fromoutside organization

Depends on findingdevelopment fromoutside organization

External factorswill not make anychanges

External factorswill make minorchanges (<25%)

External factorswill make majorchanges (<50%)

External factors willmake significantchanges (<75%)

External factorswill make constantchanges

ISO Risk Probability Table

ISO Risk Consequence Table

ISO Risk Contour

SW Risk Handling Avoidance - de-scoping objectives Assumption – latent defects Control – user acceptance testing Transfer – from software to

firmware or hardware

SW Metrics

S o f t w a r e R i s k I t e m s

Co

un

t

T i m e

C l o s e d R i s k s

N e w R i s k s

Commercial vs. DoD/NASA Perspective on Risk

Management

Commercial vs. Gov’t Perspective

Different market conditions Different best practices Different likelihoods for similar

issues

As always, tailor RM to program needs

Market Differences How is risk impacted?

Commercial DoD / NASA

Many small buyers Fewer large buyers

Many small suppliers Typically few suppliers of a given item

Market sets price Oligopoly pricing - biased to availble budget

Free movement in/out market Barriers to entry

Prices set by marginal costs Prices proportional to total cost

Once funding secured, usually stable May have unanticipated disruptions to funding

Capcity to supply adjusts to demand Moderate to large excess capacity

SW Development Best Practices

Commercial DoD / NASA

Evolutionary upgrades of existing systems Little reuse, many unique systems

Heavy buyer involvement (as team member) Formal development model - buyer oversees

Informal reviews Very formal reviews

Heavy user involvement Limited user involvement; buyer involved

Based on one or more industry stds Use gov't and industry stds

Prototyping common Limited prototyping

How is risk impacted?

Risk Category LikelihoodCategory Commercial DoD / NASA CommentCost Highly Likely Highly Likely Whenever new development is

requiredDesign Possible Likely Degree of design enhancement

requiredIntegration Possible Likely Driven by complexitySupport Possible Likely Commercial life cycles generally

shortManufacturing Likely Likely Varies with production rate &

resource availabilityTechnology Possible Likely Increases as push state of artManagement Possible Possible Depends on program complexity,

performance expectations

Political Unlikely Likely External issue to program

Overview of Risk Management Tools

Cautions in Tool Selection A good tool for one organization may

not be a good match for another Tailor RM to the program needs

The tool should never dictate the process Define process, then choose compatible

tool Be compatible with program culture

Effective Use of a Tool RM is more than using a RM tool Tool must efficiently & effectively

integrate into program Resources required Level of detail, complexity Focus of tool – e.g., program phase

RM Data Base Considerations Provide sufficient configuration control

Accessible to all team members Ability to accept anonymous comments

Support program needs Reporting Monitoring Captures lessons learned Fulfills contractual requirements

Balance costs/value

Tools Comparison @Risk & Crystal Ball – licensed

software Monte Carlo simulation add-in for Excel

Select desired distribution function & define parameters

Provide data and generate a plausible distribution function

Provides statistics and graphical output User provides risk analysis structure

Probability-Consequence Screening

Developed by the Air Force Risk events assigned a probability &

consequence for performance, schedule & cost

Position in consequence screening matrix determines risk score

User assigns Hi, Med, Low ranges Generates reports and graphical output www.pmcop.dau.mil

Probability-Consequence Screening

Probability-Consequence Screening

Risk Matrix Excel – based model

Collects inputs in watch list format Uses best practices (ordinal) breakout for

probability & consequence Orders events by Borda rank & assigns risk

level Generates action plan reports and

graphical output www.mitre.org

Risk Matrix

Risk Matrix

Risk Radar Access – based, licensed software

Can establish standard values for risk categorization

Manual or automatic risk prioritization Complies with ISO, SEI CMMI & Government

standards Generates detailed, summary & metrics

reports Demo available:

www.iceincusa.com/products_tools.htm

Risk Radar

Risk Radar

Risk Radar

TRIMS Technical Risk ID & Mitigation

Knowledge-based system Utilizes SEI & Navy Best Practices to collect

data on past experiences Measures technical risk rather than cost &

schedule Most applicable to design efforts Can tailor categories, templates & questions

Generates status, next action & overdue action reports

www.bmpcoe.org

TRIMS Technical Risk ID & Mitigation

TRIMS Technical Risk ID & Mitigation

DSM – Design Structure Matrix

Knowledge & simulation-based tool Assesses complexity of dependency

relationships between project tasks Measures risk in terms of schedule impact Most applicable to design efforts Ongoing development effort at MIT

Generates suggested sub-team groupings, & probability curves for task duration ranges

www.dsmweb.org

DSM – Design Structure Matrix

DSM – Design Structure Matrix

DSM – Design Structure Matrix

Final Exam Closed book, closed notes You have 90 minutes for exam. Any questions? Turn in Part II of your project

according to the schedule discussed last week