This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
•This course starts with brief model of human decision making (slides 14-27). Then it presents a crisp description of the tradeoff study processes (Slides 14-67), which includes a simple example of choosing between two combining methods.
•Then it shows a complex, but well-known tradeoff study example that most people will be familiar with: the San Diego airport site selection (Slides 68-75).
•Then we go back and examine many difficulties that could arise when designing a tradeoff study; we show many methods that have been used to overcome these potential problems (Slides 76-338).
•The course is summarized with slides 339-346.• In the Dog System Exercise, students create their own solutions for
a tradeoff study. These exercises will be computer based. The students complete one of the exercise’s eight parts. Then we give them our solutions. They complete another portion and we give them another solution. The computers will be preloaded with all of the problems and solutions. The students will use Excel spreadsheets and a simple program for graphing scoring (utility) functions.
•After the exercise there will be a mathematical summary of tradeoff methods. Students who are algebraically challenged may excuse themselves.
•The students should be able to Understand human decision making Use many techniques, including tradeoff
studies, to help select among alternatives Decide whether a problem is a good
candidate for a tradeoff study Establish evaluation criteria with weights of
importance Understand scoring (utility) functions Perform a valid tradeoff study Fix the do nothing problem Use several different combining functions Perform a sensitivity analysis Be aware of many tradeoff methods Develop a decision tree
decision problems that require DAR and incorporate them in their plans (e.g. SEMPs)
•DAR is a common process•Common processes are tools that the user
gets, tailors and uses•DAR is invoked throughout the whole
program lifecycle whenever a critical decision is to be made
•DAR is invoked by IPT leads on programs, financial analysts, program core teams, etc.
• Invoke the DAR Process in work instructions, in gate reviews, in phase reviews or with other triggers, which can be used anytime in the system life cycle
It’s not done just onceIt’s not done just once•A tradeoff study is not something that you do once at the beginning of a project.
•Throughout a project you are continually making tradeoffs creating team communication methods selecting components choosing implementation techniques designing test programsmaintaining schedule
•Many of these tradeoffs should be formally documented.
These tasks are drawn serially,but they are not performed in a serial manner. Rather, it is an iterative processwith many feedback loops, which are not shown.
Humans make four types of decisions:Humans make four types of decisions:•Allocating resources among competing projects* •Generating plans, schedules and novel ideas•Negotiating agreements•Choosing amongst alternatives Alternatives can be examined in series or
parallel. When examined in series it is called sequential
search When examined in parallel it is called a tradeoff
or a trade study “Tradeoff studies address a range of
problems from selecting high-level system architecture to selecting a specific piece of commercial off the shelf hardware or software. Tradeoff studies are typical outputs of formal evaluation processes.”*
Select evaluation methodsSelect evaluation methods• Select the source of the evaluation data and the
method for evaluating the data• Typical sources for evaluation data include
approximations, product literature, analysis, models, simulations, experiments and prototypes*
• Methods for combining data and evaluating alternatives include Multi-Attribute Utility Technique (MAUT), Ideal Point, Search Beam, Fuzzy Databases, Decision Trees, Expected Utility, Pair-wise Comparisons, Analytic Hierarchy Process (AHP), Financial Analysis, Simulation, Monte Carlo, Linear Programming, Design of Experiments, Group Techniques, Quality Function Deployment (QFD), radar charts, forming a consensus and Tradeoff Studies
Collect evaluation dataCollect evaluation data•Using the appropriate source (approximations, product literature, analysis, models, simulations, experiments or prototypes) collect data for evaluating each alternative.
Evaluate alternativesEvaluate alternatives•Evaluate alternative solutions using the evaluation criteria, weights of importance, evaluation data, scoring functions and combining functions.
•Evaluating alternative solutions involves analysis, discussion and review. Iterative cycles of analysis are sometimes necessary. Supporting analyses, experimentation, prototyping, or simulations may be needed to substantiate scoring and conclusions.
Select preferred solutionsSelect preferred solutions• Select preferred solutions from the alternatives
based on evaluation criteria.
• Selecting preferred alternatives involves weighing and combining the results from the evaluation of alternatives. Many combining methods are available.
• The true value of a formal decision process might not be listing the preferred alternatives. More important outputs are stimulating thought processes and documenting their outcomes.
• A sensitivity analysis will help validate your recommendations.
• The least sensitive criteria should be given weights of 0.
Who should come to the review?Who should come to the review?•Program Manager•Chief Systems Engineer•Review Inspector•Lead Systems Engineer•Domain Experts• IPT Lead•Facilitator •Stakeholders for this decision
Builder Customer Designer Tester PC Server
•Depending on the decision, the Lead Hardware Engineer and the Lead Software Engineer
Put in the PALPut in the PAL• Formal evaluations reviewed by experts
should be put in the organizational Process Asset Library (PAL) or the Project Process Asset Library (PPAL)
• Evaluation data for tradeoff studies come from approximations, analysis, models, simulations, experiments and prototypes. Each time better data is obtained the PAL should be updated.
• Formal evaluations should be designed with reuse in mind.
• Identify alternative solutions Linear addition of weight times scores,
Multiattribute Utility Theory (MAUT).* This method is often called a “trade study.” It is often implemented with an Excel spreadsheet. Analytic Hierarchy Process (AHP)**
In-class exerciseIn-class exercise•Use these criteria to help select your lunch today.Closeness, distance to the venue. Is it in the same building, the next building or do you have to get in a car and drive?Tastiness, including gustatory delightfulness, healthiness, novelty and savoriness.Price,* total purchase price including tax and tip.
Sensitivity analysis, simpleSensitivity analysis, simpleIn terms of Familiarity, MAUT was strongly preferred (5) over the AHP. Now change this 5 to a 3 and to a 7.
• Changing the scores for Familiarity does not change the recommended alternative.
• This is good.• It means the Tradeoff study is robust with
Sensitivity analysis, analyticSensitivity analysis, analyticCompute the six semirelative-sensitivity functions, which are defined as
which reads, the semirelative-sensitivity function of the performance index F with respect to the parameter is the partial derivative of F with respect to times with everything evaluated at the normal operating point (NOP).
• Improve the DAR process. Add some other techniques, such as AHP, to
the DAR web course Fix the utility curves document Add image theory to the DAR process Change linkages in the documentation system Create a course, Decision Making and Tradeoff
Wildlife HazardsJoint Use and National Defense CompatibilityExpandability
Ground AccessTravel Time, percentage of population in three travel time segments Roadway Network Capacity, existing and projected daily roadway volumes Highway and Transit Accessibility, distance to existing and planned
freeways Environmental Impacts
Quantity of residential land to be displaced by the airport developmentNoise Impact, population within each of three specific decibel ranges Biological Resources
Wetlands Protected speciesWater qualitySignificant cultural resources
PurposePurposeThe systems engineer’s job is to elucidate domain knowledge and capture the values and preferences of the decision maker, so that the decision maker (and other stakeholders) will have confidence in the decision.
The decision maker balances effort with confidence*
Part of a Pinewood Derby tradeoff studyPart of a Pinewood Derby tradeoff studyPerformance figures of merit evaluated on a prototype for a Round Robin with Best Time Scoring
Evaluation criteria
Input value
Score Weight Score times
weight 1. Average Races
per Car 6 0.94 0.20 0.19
2. Number of Ties 0 1 0.20 0.20 3. Happiness 0.87 0.60 0.52
• Looking at alternatives in parallel is not an innate human action.
• Usually people select one hypothesis and work on it until it is disproved, then they switch to a new alternative: that’s the scientific method.
• Such serial processing of alternatives has been demonstrated for Fire fighters Airline pilots Physicians Detectives Baseball managers People looking for restaurants*
•V. V. Krishnan has a model of animals searching for habitat (home, breeding area, hunting area, etc.)
•It uses the value of each habitat and the cost of moving between sites.
•When travel between sites is inexpensive, e. g. birds or honeybees* searching for a nest site, the search is often a tradeoff study comparing alternatives in parallel.
•When travel is expensive, e.g. beavers searching for a dam site, the search is usually sequential.
•When making decisions there is always uncertainty, too little time and insufficient resources to explore the whole problem space.
•Therefore, people cannot make rational decisions.
•The term satisficing was coined by Noble Laureate Herb Simon in 1955.
•Simon proposed that people do not attempt to find an optimal solution. Instead, they search for alternatives that are good enough, alternatives that satisfice.
RiskRisk•Systems engineers use risk to evaluate and manage bad things that could happen, hazards. Risk is measured with the frequency (or probability) of occurrence times the severity of the consequences.
•However, in economics and in the psychology of decision making, risk is defined as the variance of the expected value, uncertainty.*
Ambiguity, uncertainty and hazards*Ambiguity, uncertainty and hazards*•Hazard: Would you prefer my forest picked mushrooms or portabella mushrooms from the grocery store?
•Uncertainty: Would you prefer one of my wines or a Kendall-Jackson Napa Valley merlot?
•Ambiguity: Would you prefer my saffron and oyster sauce or marinara sauce?
Subjective expected utility theorySubjective expected utility theorymodels human decision making as maximizing
subjective expected utility maximizing, because people choose the set of
alternatives with the highest total utility, subjective, because the choice depends on the
decision maker’s values and preferences, not on reality (e.g. advertising improves subjective perceptions of a product without improving the product), and expected, because the expected value is used.
• This is a first-order model for human decision making.
Why teach tradeoff studies?Why teach tradeoff studies?•Because emotions, cognitive illusions, biases, fallacies, fear of regret and use of heuristics make humans far from ideal decision makers.
•Using tradeoff studies judiciously can help you make rational decisions.
•We would like to help you move your decisions from the normal human decision-making lower-right quadrant to the ideal decision-making upper-left quadrant.
Problem statementProblem statement•Stating the problem properly is one of the systems engineer’s most important tasks, because an elegant solution to the wrong problem is less than worthless.
•Problem stating is more important than problem solving.
•The problem statement describes the customer’s needs, states the goals of the project, delineates the scope of the problem, reports the concept of operations, describes the stakeholders, lists the deliverables and presents the key decisions that must be made.
Evaluation criteriaEvaluation criteria•are derived from high priority tradeoff requirements.
•should be independent, but show compensation.
•Each alternative will be given a value that indicates the degree to which it satisfies each criterion. This should help distinguish between alternatives.
•Evaluation criteria might be things like performance, cost, schedule, risk, security, reliability and maintainability.
Example criterion packageExample criterion package11
•Name of criterion: Percent Happy Scouts
•Description: The percentage of scouts that leave the race with a generally happy feeling. This criterion was suggested by Sales and Marketing and the Customer.
•Weight of importance: 10
•Basic measure:* Percentage of scouts who leave the event looking happy, contented or pleased
•Units: percentage
•Measurement method: Estimate by the Pinewood Derby Marshall
•Input: The domain is 0 to 100%. The expected values are 70 to 100%.
Second example criterion packageSecond example criterion package22
•Output: 0 to 1
•Scoring function: Biphasic hill shape with lower threshold of 0, lower baseline of 2, lower baseline slope of 0.67, optimum of 3.5, upper baseline of 4.5, upper baseline slope of -1 and upper threshold of 8.
Evaluation criteria are also called Evaluation criteria are also called • Attributes*• Objectives• Metrics• Measures• Quality characteristics• Figures of merit • Acceptance criteria
“Regardless of what has gone before, the acceptance criteria determine what is actually built.”
MoE versus MoPMoE versus MoP•Generally, it is not worth the effort to debate nuances of these terms. But here is an example.
•Measures of Effectiveness (MoEs) show how well (utility or value) a part of the system mission is satisfied. For an undergraduate student trying to earn a Bachelors degree, his or her class (Freshman, Sophomore, Junior or Senior) would be an MoE.
•Measures of Performance (MoPs) show how well the system functions.For our undergraduate student, their grade point average would be an MoP.*
•MoEs are often computed using several MoPs.
MoEs versus MoPsMoEs versus MoPs22
•The city of Tucson wants to widen Grant Road between I-10 and Alvernon Road. They want six lanes with a median, a 45 mph speed limit, and no traffic jams.
•MoEs cars per day averaged over two weeks cars per hour between 5 and 6 PM, Monday to
Friday, averaged over two weeks•MoPs number of pot holes after one year traffic noise (in dB) at local store fronts smoothness of the surface esthetics of landscaping straightness of the road travel time from I-10 to Alvernon number of traffic lights
Properties of good evaluation criteriaProperties of good evaluation criteria• Criteria should be objective• Criteria should be quantitative• Wording of criteria is very important• Criteria should be independent• Criteria should show compensation • Criteria should be linked to requirements • The criteria set should be hierarchical• The criteria set should cover the domain evenly• The criteria set should be transitive• Temporal order should not be important• Criteria should be time invariantOverview slide
Evaluation criteria propertiesEvaluation criteria properties• These properties deal with verification the combining function individual criteria sets of criteria
• But problems created by violating these properties can be ameliorated by reengineering the criteria
Evaluation criteria should be objective Evaluation criteria should be objective (observer independent)(observer independent)• Being Pretty or Nice should not be a criterion
for selecting crewmembers• In sports, Most Valuable Player selections are
often controversial• Deriving a consensus for the Best Football
Evaluation criteria should be worded in a Evaluation criteria should be worded in a positive manner, so that more is betterpositive manner, so that more is better**
• Use Uptime rather than Downtime.• Use Mean Time Between Failures
rather than Failure Rate.• Use Probability of Success, rather
than Probability of Failure.• When using scoring functions make
Exercise: rewrite this statementExercise: rewrite this statementWe have a surgical procedure that should cure your problem. Statistically one percent of the people who undergo this surgery die. Would you like to have this surgery?
Percent happy scoutsPercent happy scouts•The Pinewood Derby tradeoff study had these criteria Percent Happy Scouts Number of Irate Parents
•Because people evaluate losses and gains differently, the Preferred alternatives might have been different if they had used Percent Unhappy Scouts Number of Ecstatic Parents
Buying a new car, couple-2 criteria Buying a new car, couple-2 criteria •Wife
Safety•Husband
Maximum Horse Power Peak Torque Top Speed Time for the Standing Quarter Mile Engine Size (in liters) Number of Cylinders. Time to Accelerate 0 to 60 mph
Sometimes it is hard to get both Sometimes it is hard to get both independence and compensationindependence and compensation• If two criteria are independent,
they might not show compensation
• If they show compensation, they might not be independent
• Independence is more important for mandatory requirements
•Compensation is more important for tradeoff requirements
Temporal order Temporal order should not be importantshould not be important Criteria should be created so that the temporal order is not important for verifying or combining.
The temporal order of verifying The temporal order of verifying criteria should not be important criteria should not be important •Criteria requiring that clothing be Flame Proof
and Water Resistant would make the verification results depend on which we tested first
If the criteria depend on temporal order, then an expert system or a decision tree might be more suitable
The temporal order of combining The temporal order of combining criteria should not be important criteria should not be important • Consider a combining function (CF) that adds
two numbers truncating the fraction(0.2 CF 0.6) CF 0.9 = 0, however,(0.9 CF 0.6) CF 0.2 = 1,the result depends on the order.
• With the Boolean NAND* function ()(0 1) 1 = 0 however, (1 1) 0 = 1, the result depends on the order.
Order of presentation is importantOrder of presentation is important•The stared question is the only question that department and
college promotion committees look at. It is the only question reported in the TCE History.
•Larry Alimony’s CIEQ• I would take another course that was taught this way•The course was quite boring •The instructor seemed interested in students as individuals•The instructor exhibited a through knowledge of the subject matterWhat is your overall rating of this instructor’s teaching
effectiveness?
•TCE What is your overall rating of this instructor’s teaching
effectiveness?•What is your overall rating of the course?•Rate the usefulness of HW, projects, etc. •What is your rating of this instructor compared to other
instructors?•The difficulty level of the course is …
Evaluation cEvaluation criteria libraryriteria library•Criteria should be created so that they can be reused.
•Your company should have library of generic criteria.•Each criterion package would have the following slots Name DescriptionWeight of importance (priority) Basic measure UnitsMeasurement method Input (with allowed and expected range) Output Scoring function (type and parameters) Trace to (document)
Weights of importanceWeights of importanceThe decision maker should assign weights so that the more important criteria will have more effect on the outcome.
Part of a Pinewood Derby tradeoff studyPart of a Pinewood Derby tradeoff studyPerformance figures of merit evaluated on a prototype for a Round Robin with Best Time Scoring Figure of Merit Input
value Score Weight Score
times weight
1. Average Races per Car
6 0.94 0.20 0.19
2. Number of Ties 0 1 0.20 0.20 3. Happiness 0.87 0.60 0.52 Qualitative
Aspects that help establish weightsAspects that help establish weights
Reference: A Prioritization Process
Organizational Commitment Time Required Criticality to Mission Success Risk Architecture Safety Business Value Complexity Priority of Scenarios (use cases) Implementation
Difficulty Frequency of Use Stability Benefit Dependencies Cost Reuse Potential Benefit to Cost Ratio When it is needed
The status quoThe status quo"Selecting an option from a group of similar options can be difficult to justify and thus may increase the apparent attractiveness of retaining the status quo. To avoid this tendency, the decision maker should identify each potentially attractive option and compare it directly to the status quo, in the absence of competing alternatives. If such direct comparison yields discrepant judgments, the decision maker should reflect on the inconsistency before making a final choice."
The Do Nothing alternatives forThe Do Nothing alternatives forreplacing a Datsun 240Z Status quo, keep the 240Z Nihilism, do without a car, i.e., walk or take
If the Do Nothing alternative wins,If the Do Nothing alternative wins,your Cost, Schedule and Risk criteria may have overwhelmed your Performance criteria.
If a Do Nothing alternative winsIf a Do Nothing alternative wins22
• Just as you should not add apples and oranges, you should not combine Performance, Cost, Schedule and Risk criteria with each other Combine the Performance criteria (with their
weights normalized so that they add up to one)
Combine the Cost criteria Combine the Schedule criteria Combine the Risk criteria
•Then the Performance, Cost, Schedule and Risk combinations can be combined with clearly stated weights, 1/4, 1/4, 1/4 and 1/4 could be the default.
• If a Do Nothing alternative still wins, you may have the weight for Performance too low.
• One important purpose for including a do nothing alternative (and other bizarre alternatives) is to help get the requirements right. If a bizarre alternative wins the tradeoff analysis, then you do not have the requirements right.
• Similarly including sacred cows in the alternatives, will also test the adequacy of the requirements.
• “For a successful technology, reality must take precedence over public relations, for nature cannot be fooled.” -- Richard Feynman
•Treating CAIV means that you should do the tradeoff study with a specific cost and then go talk to your customer and see what performance, schedule and risk requirements he or she is willing to give up in order to get that cost.
•So if you want to treat CAIV, then keep your tradeoff study independent of cost: that is, do not use cost criteria in your tradeoff study.
Mandatory requirementsMandatory requirements•Mandatory requirements specify necessary and sufficient capabilities that the system must have to satisfy customer needs and expectations.
•They use the words shall or must.
•They are either passed or failed, with no in between.
•They should not be included in a tradeoff study.
•Here is an example of a mandatory requirement: The system shall not violate federal, state or
Tradeoff requirementsTradeoff requirements•Tradeoff requirements state capabilities that would make the customer happier.
•They use the words should or want. •They use measures of effectiveness and scoring functions.
•They are evaluated with multicriterion decision techniques.
•There will be tradeoffs among these requirements. •Here is an example of a tradeoff requirement: Dinner should have items from each of the five food groups: Grains, Vegetables, Fruits, Wine, Milk , and Meat and Beans.
•Mandatory requirements are often the upper or lower limits of tradeoff requirements.
UncertaintyUncertainty•Evaluation data (and weights of importance) should, when convenient, have measures of uncertainty associated with the data.
•This could be done with probability density functions, fuzzy numbers, variance, expected range, certainty factors, confidence intervals, or simple color coding.
Scoring function exampleScoring function exampleThis scoring function reflects the DM’s utility that he would be twice as satisfied if there were 91% happy scouts compared to 88% happy scouts.*
What is the best package of soda pop to buy?*What is the best package of soda pop to buy?*Regular price of Coca-Cola in Tucson, January 1995.The Cost criterion is the reciprocal of price.The Performance criterion is the quantity in liters.
Choosing Amongst Alternative Soda Pop Packages Data Criteria Trade-off Values
Scoring function for QuantityScoring function for Quantity**
A simple program that creates graphs such as these is available for free athttp://www.sie.arizona.edu/sysengr/slides.It is called the Wymorian Scoring Function tool.
Different Distributions of Alternatives in Different Distributions of Alternatives in Criteria SpaceCriteria Space** May Produce Different May Produce Different
Preferred AlternativesPreferred Alternatives
Tradeoff of requirements*Tradeoff of requirements*
Pareto OptimalPareto OptimalMoving from one alternative to another will improve at least one criterion and worsen at least one criterion, i.e., there will be tradeoffs.
“The true value of a service or product is determined by what one is willing to give up to obtain it.”
Real-world data will not fall neatly onto lines such as the circle in the pervious slide. But often they may be bounded by such functions. In the operations research literature such data sets are called convex, although the function bounding them is called concave (Kuhn and Tucker, 1951).
A lively baseball debateA lively baseball debate•For over 30 years baseball statisticians have argued over the best measure of offensive effectiveness.
•Two of the most popular measures are On-base plus slugging OPS = OBP + SLG Batter’s run average BRA = OBP x SLG
•I think their arguments ignored the most relevant data, the shape of the distribution of OBP and SLG for major league players.
Muscle force-velocity relationshipMuscle force-velocity relationship• (Force + F0 )(velocity + vmax) = constant, where F0 (the
isometric force) and vmax (the maximum muscle velocity) are constants.
• Humans sometimes use one combining function and sometimes they use another.
• If a bicyclist wants maximum acceleration, he or she uses the point (0, F0). If there is no resistance and maximum speed is desired, use the point (vmax, 0). These solutions result from maximizing the sum of force and velocity.
• However, if there is energy dissipation (e.g., Friction, air resistance) and maximum speed is desired, choose the maximum power point, the maximum product of force and velocity.
• This shows that the appropriate tradeoff function may depend on the task at hand.
Nonconvex data setsNonconvex data setsThe muscle force-velocity relationship fit neatly onto lines such as this hyperbola. This will not always be the case. But when it is not, the data may be bounded by such functions. In the operations research literature such data sets are called concave, although the function bounding them is called convex (Kuhn and Tucker, 1951).
Mini-summaryMini-summary•The Product Combining Function always favors alternatives with moderate scores for all criteria. It rejects alternatives with a low score for any criterion.
•Therefore the Product Combining Function may seem better than the Sum Combining Function. But the Sum Combining Function is used much more in systems engineering.
Which matchesWhich matcheshuman decision making?human decision making?•For a nonconvex distribution, the Sum Combining
Function will favor the points at either end of the distribution. Sometimes this matches human decision making. I usually buy a case of soda for my family. A person working in an office building on a
Sunday afternoon might buy a single can from the vending machine.
•A frugal person might want to maximize the product of cost and performance, i.e. the maximum liters/dollar (the biggest bang for the buck), which is the three liter bottle. This matches the recommendation of the Product Combining Function.
NBA teams seem to use NBA teams seem to use pp = = • When drafting basketball players
• Criteria are Height and Assists
• They want seven-foot players with ten assists per game (the ideal point)
• In years when there are many point guards but no centers, they draft the best point guards
• Choose the criterion with the maximum score (Assists) and then select the alternative whose number of Assists has the minimum distance to the ideal point
The moral of this storyThe moral of this storyThe perturbation step size (x – x0) should be small. Five and ten percent step sizes are probably too big, but we have been getting away with it, because we usually use the sum combining function.
Derivative of a function of two variablesDerivative of a function of two variables
•Let us examine the second-order terms,
those inside the { }, for two reasons to see if they are large and must be included in computing the first derivative to estimate the effects of interactions on the sensitivity analysis
What went wrong?What went wrong?In the previous computations, we changed both parameters at the same time and then compared the value of the function to the value of the function at its normal operating point. However, this is not the correct estimation for the second-partial derivative.
Estimating the sensitivity functionsEstimating the sensitivity functionsTo get the semirelative-sensitivity function we multiply the second-partial derivative by the normal values of Wt1 and S11 to get
Now, this is the same result that we derived in the analytic semirelative-sensitivity section.
Lessons learnedLessons learned•The perturbation step size should be small. Five and 10% perturbations are not acceptable.
•It is incorrect to estimate the second partial derivative by changing two parameters at the same time and then comparing that value of the function to the value of the function at its normal operating point. Estimating second derivatives requires evaluation of four not two numerator terms.
Other Techniques for Combining Data in Other Techniques for Combining Data in Order to Find the Preferred alternativesOrder to Find the Preferred alternatives
The preferred alternative is found by minimizing the distance to the ideal point using LP metrics.
where zk is the score of the kth criterion, wk is the weight of the kth criterion, z*k is the kth component of the ideal point, z*k is the kth component of the anti-ideal point and n is the number of criteria. The criteria index is k and the alternatives index is i.
Fuzzy Logic, rationaleFuzzy Logic, rationale•Some things are described well by probability theory. Such as the probability that John Wayne was a tall person is around 1.0.
•But what is the probability that George W. Bush is a tall person?
•This question does not have a good answer.
•The theory of Fuzzy Logic was invented to model such questions.
•With fuzzy logic the question becomes, “What is the possibility that George W. Bush belongs to the set of people called tall?”
Fuzzy rules for a single can Rule number Fuzzy premises Consequences
Cost Volume 1 Very Low Very Low 1 Can 2 Very Low Low 1 Can 3 Very Low Medium 1 Can 4 Very Low High 1 Can 5 Very Low Very High 1 Can 6 Low Very Low 1 Can 7 Low Low 1 Can 8 Low Medium 1 Can 9 Low High 1 Can
10 Low Very High 1 Can 11 Medium Very Low 1 Can 12 Medium Low 1 Can 13 Medium Medium 1 Can 14 Medium High 1 Can 15 Medium Very High 1 Can 16 High Very Low 1 Can 17 High Low 1 Can 18 High Medium 1 Can 19 High High 1 Can 20 High Very High 1 Can 21 Very High Very Low 1 Can 22 Very High Low 1 Can 23 Very High Medium 1 Can 24 Very High High 1 Can 25 Very High Very High 1 Can
1 Very Low 0.00 Very Low 0.65 1 Can 0.00 2 Very Low 0.00 Low 0.35 1 Can 0.00 3 Very Low 0.00 Medium 0.00 1 Can 0.00 4 Very Low 0.00 High 0.00 1 Can 0.00 5 Very Low 0.00 Very High 0.00 1 Can 0.00 6 Low 0.00 Very Low 0.65 1 Can 0.00 7 Low 0.00 Low 0.35 1 Can 0.00 8 Low 0.00 Medium 0.00 1 Can 0.00 9 Low 0.00 High 0.00 1 Can 0.00 10 Low 0.00 Very High 0.00 1 Can 0.00 11 Medium 0.00 Very Low 0.65 1 Can 0.00 12 Medium 0.00 Low 0.35 1 Can 0.00 13 Medium 0.00 Medium 0.00 1 Can 0.00 14 Medium 0.00 High 0.00 1 Can 0.00 15 Medium 0.00 Very High 0.00 1 Can 0.00 16 High 0.00 Very Low 0.65 1 Can 0.00 17 High 0.00 Low 0.35 1 Can 0.00 18 High 0.00 Medium 0.00 1 Can 0.00 19 High 0.00 High 0.00 1 Can 0.00 20 High 0.00 Very High 0.00 1 Can 0.00 21 Very High 1.00 Very Low 0.65 1 Can 0.65 22 Very High 1.00 Low 0.35 1 Can 0.35 23 Very High 1.00 Medium 0.00 1 Can 0.00 24 Very High 1.00 High 0.00 1 Can 0.00 25 Very High 1.00 Very High 0.00 1 Can 0.00
Rules with non-zero degree of fulfillment (DoF) Rule number
Cost Volume Package DoF
21 Very High 1.00 Very Low 0.65 1 Can 0.65 22 Very High 1.00 Low 0.35 1 Can 0.35 37 Medium 0.46 Low 1.00 1 liter 0.46 42 High 0.54 Low 1.00 1 liter 0.54 58 Low 0.44 Medium 1.00 2 liter 0.44 63 Medium 0.56 Medium 1.00 2 liter 0.56 78 Very Low 0.12 Medium 0.87 6 pack 0.10 79 Very Low 0.12 High 0.13 6 pack 0.02 83 Low 0.88 Medium 0.87 6 pack 0.77 84 Low 0.88 High 0.13 6 pack 0.11 109 Low 0.82 High 1.00 3 liter 0.82 114 Medium 0.18 High 1.00 3 liter 0.18 125 Very Low 0.44 Very High 1.00 12 pack 0.44 130 Low 0.56 Very High 1.00 12 pack 0.56 150 Very Low 0.62 Very High 1.00 24 pack 0.62 155 Low 0.38 Very High 1.00 24 pack 0.38
Can we use this fuzzyCan we use this fuzzyrule base to give advice?rule base to give advice?22
• Suppose our customer says, “A few of my friends and I cashed in all our empty bottles. We want to buy some soda pop and put it in this little cooler.”
• We would convert that to, “Cost = Low AND Quantity = Medium.”
• Two rules succeed: one for the 2 liter bottle and one for the 6 pack. The highest DoF is for the 6 pack. Therefore, we would recommend, “Buy a 6 pack, DoF = 0.77.”
Killer tradesKiller trades•We do not have time to analyze all 60 possibilities. So we limit the number of things to be studied by doing killer trades. That is, we answer certain questions and kill off large parts of the decision tree.
•In this example we will say that a formal evaluation is necessary, we will use approximation data and the sum combining function.
•This means that our tradeoff study matrix only needs three columns, one for each alternative.
Should we walk this famous slugger?Should we walk this famous slugger?
*Includes hit by pitch, error, etc.**Indicates preferred option†Utility is runs plus expected future runs, from an initial condition of no runners on base and no outs. For the pitching team, less utility is best.
PhrasingPhrasing•The way you phrase the question may determine the answer you will get.
•When asked whether they would approve surgery in a hypothetical medical emergency, many more people accepted surgery when the chance of survival was given as 99 percent than when the chance of death was given as 1 percent.
Factors affecting human decisionsFactors affecting human decisions the decision maker corporate culturethe decision maker’s valuespersonality typesrisk aversenessbiases, illusions and use of heuristics
information displayedwording of the questioncontext
the decisioneffort required to make the decisiondifficulty of making the decisiontime allowed to make the decisionneeded accuracy of the decisioncost of the decisionlikelihood of regret
Temporal orderTemporal order•You will get more consistent results if youfirst work on the criteria then fill in the matrix of evaluation data row by rowassign weights last, that way criteria that have no affect on the outcome can be given minimal weights
When you get When you get “The Wrong Answer” “The Wrong Answer” you could change you could change
• Weights of importance• Scores for the alternatives• Parameters of the scoring functions• Parameters of the combining function• The combining function itself• The tradeoff method
Possible missing requirementsPossible missing requirements• Need for Storage Space• Time Before Soda Loses Carbonization• Need for a Glass• Availability of Cold Soda in the Desired Size• Ziggy’s Trips to the Restroom
The feeling in your stomach testThe feeling in your stomach test**
•Assume you are trying to make an important decision, like “Should I quit my job and become a consultant?”
•You have done a tradeoff study, but the results are equivocal.
•How should you decide?
•Get a coin. Assign heads and tails, e.g. heads I quit my job, tails I keep my job. Flip the coin and look at the result. What is the immediate feeling in your stomach?
•If it was heads, but your stomach is in turmoil, then keep your job.
LimitationsLimitations•Limited time and resources guarantee that a tradeoff study will never contain all possible criteria.
•Tradeoff studies produce satisficing (not optimal) solutions.
•A tradeoff study reflects only one view of the problem. Different tradeoff analysts might choose different criteria and weights and therefore would paint a different picture of the problem.
•We ignored human decision-making mistakes for which we have no corrective action, such as closed mindedness, lies, conflict of interest, political correctness and favoritism.
UncertaintyUncertainty•We studied two independent tradeoff studies that had a variability or uncertainty statistic associated with each evaluation datum.
•These statistics were carried throughout the whole computational process, so that at the end the recommended alternatives had associated uncertainty statistics.
•Both of these studies were incomprehensible.
•Therefore, we did not try to accommodate uncertainty, changes and dependencies in the evaluation data.
COTS-Based Engineering ProcessCOTS-Based Engineering Process•When choosing commercial off the shelf
(COTS) products the following generic criteria may be convenient: Percent of requirements satisfied Vendor viability Total life cycle cost Apparent interface ease Architectural compatibility Foreign components User interface ease of use Observable states
Specific criteriaSpecific criteria• For tradeoff study tools these specific criteria may be
convenient: Rationale is easy to understand Can verify calculations with paper and pencil Works with nonconvex distributions of alternatives Implements scoring functions (utility curves) Has multiple combining functions Performs sensitivity analyses
A tradeoff study on A tradeoff study on tradeoff study toolstradeoff study tools•A tradeoff study was performed starting with 60 COTS decision analysis tools.
•These were the final Preferred alternatives Pinewood by Bahill Intelligent Computer Systems Hiview by Catalyze Ltd. Logical Decisions for Windows by Logical Decisions
Inc. Expert Choice by Expert Choice Inc.
See A Tradeoff Study of Tradeoff Study Tools http://www.sie.arizona.edu/sysengr/sie554/tradeoffStudyOfTradeoffStudyTools.doc
Create a Tradeoff StudyCreate a Tradeoff Study**11
Iteration: 2.1Brief Description: Tradeoff Analyst completes the four modules of the tradeoff study tool and gives the results to the decision maker. Every aspect of a tradeoff study requires extensive discussion with the decision maker and other stakeholders.
Added Value: This helps a decision maker to make better decisions and it documents the process that was used to make these decisions.
Level: User goalScope: Applies to a decision problem that is appropriate for a tradeoff study.
Primary Actor: Tradeoff Analyst (this could be a person or a team).
Supporting Actors: Tradeoff Analyst will get the tradeoff study tool and documents from Company Resources. Tradeoff Analyst will put the results of the tradeoff study in the project assets library (PAL).
Frequency: Company wide, once a week
Precondition: A decision maker has asked Tradeoff Analyst to perform a tradeoff study. Preliminary criteria, weights, alternatives and criteria values must already be defined and be in the hands of Tradeoff Analyst.
Tradeoff Analyst can stop the system at any time; all entered data and intermediate results will be saved [exit use case].
Postcondition: Tradeoff Analyst has planed a tradeoff study.
Specific Requirements
Functional Requirements:
Note: Transferring data from the Criteria Module into other modules and interchanging information with Company Resources and the PAL are supplementary requirements.
Concrete inclusion use casesConcrete inclusion use casesThe next two use cases are concrete inclusion use cases to the Create a Tradeoff Study use case.
Iteration: 2.1Brief Description: Tradeoff Analyst enters data into the Criteria Module and designs scoring functions. If this inclusion use case is called by the base use case, then it is context sensitive; the spreadsheet that is open is the spreadsheet that is used. If the actor initiates the use case, then the name of the spreadsheet to be used must be queried.
Added Value: Tradeoff Analyst understands the criteria and develops scoring functions.
Frequency: Company wide, once a weekPrecondition: Criteria must already be defined and be in the hands of Tradeoff Analyst.
Trigger: This use case is initiated by the Create a Tradeoff Study use case or by the Tradeoff Analyst.
Main Success Scenario:1a. When triggered by the Create a Tradeoff Study use case, Tradeoff Analyst replaces criteria of the template with problem domain criteria and describes these criteria in the notes section.
2. Tradeoff Analyst works on the criteria one at a time and may rewrite, decompose or derive criteria.
Nonfunctional Requirements:NFR2-1 Scoring function graphs must be updated within 100 milliseconds of a change in a parameter.
NFR2-2 Computing normalized weights shall take less than 100 milliseconds.
Business Rules:BR-1. The weights entered by Tradeoff Analyst shall be numbers (usually integers) in the range of 0 to 10, where 10 is the most important.
Iteration: 2.1Brief Description: Tradeoff Analyst enters criteria values for the alternatives into the Input Module. If this inclusion use case is called by the base use case, then it is context sensitive, the spreadsheet that is open is the spreadsheet that is used. If the actor initiates the use case, then the name of the spreadsheet to be used must be queried.
Added Value: These criteria values can be used to compute preferred alternatives.
Level: Low levelScope: Input ModulePrimary Actor: Tradeoff AnalystFrequency: Company wide, once a week
•Decompose criteria into subcriteria •Put subcriteria in separate columns •Normalize weights •Derive evaluation data approximations product literature analysis models and simulations experiments prototypes
•Create scoring functions •Combine data in separate areas•Add columns for alternatives
• Good industry practices for ensuring success of tradeoff studies include having teams evaluate the data evaluating the data with many iterations peer review of the results and
As you do a better job of getting the requirements right, the preferred alternatives of different teams converge.
Speculation
As you do a better job of getting the necessary and sufficient requirements, the preferred alternatives of the various tradeoff combining techniques will converge.
•Emotions, illusions, biases and use of heuristics make humans far from ideal decision makers.
•Using tradeoff studies thoughtfully can help move your decisions from the normal human decision-making lower-right quadrant to the ideal decision-making upper-left quadrant.
Tradeoff study exercise, detailsTradeoff study exercise, details0. Read the problem statement (dogProb0.doc) and write
some preliminary requirements, 5 minutes, wait for solutions (dogSol0.doc), 2 minute discussion.
1. Identify key system decisions and their alternatives (dogProb1.doc). 8 minutes, wait for solutions (dogSol1.doc), 7 minute discussion.
2. Fill in the Decision Tree Worksheet, use text boxes or do it on paper (dogProb2.doc). 8 minutes, wait for solutions (dogSol2.doc), 7 minute discussion.
3. Use the Decision Resolution Worksheet (dogProb3.doc) to perform the Killer Trades. 8 minutes, wait for solutions (dogSol3.doc).
4. Define the tradeoff studies that still need to be done and list them on the Decision Resolution Worksheet (dogProb4.doc). 5 minutes, wait for solutions (dogSol4.doc).
Tradeoff study exerciseTradeoff study exercise6. Perform a tradeoff study using the Tradeoff
Matrix Spreadsheet (dogProb6.xls). 30 minutes, wait for solutions, In 10 minutes discuss both dogSol6.xls and dogSol6.doc.
For scoring functions open the folder named SSF and use the tool named SSF.exe
7. Fix the Do Nothing problem (dogProb7.doc and dogProb7.xls). 5 minutes, wait for solutions. In 5 minutes discuss the sensitivity analysis in dogSol7.doc and dogSol7.xls.
Tradeoff study exerciseTradeoff study exercise8. Recompute your tradeoff matrix using a
combining function other than the sum of weighted scores (dogProb8.doc and dogProb8.xls). 15 minutes, wait for solutions. In 5 minutes discuss the sensitivity analysis in dogSol8.doc and also the solutions in dogSol8.xls.
EquationsEquations•The following section uses algebraic equations to summarize the tradeoff methods we have just discussed. These slides are located at
AHP Analytic Hierarchy Process BCS Bowl Championship Series BM Basic Measure CDR Critical Design Review CF Combining Function CMMI Capability Maturity Model Integrated COTS Commercial Off The Shelf DAR Decision Analysis and Resolution DM Decision Maker DoF Degree of Fulfillment EV Expected Value IPT Integrated Product Development Team IQ Intelligence Quotient MAUT Multi-Attribute Utility Technique NFL National Football League NOP Normal Operating Point PAL Process Asset Library PC Personal Computer PDR Preliminary Design Review QFD Quality Function Deployment SEMP Systems Engineering Management Plan SRR System Requirements Review Wt Weight
Course materialsCourse materials•This slide show, we present this in Vista For the “Humans are not rational2” slide, bring two $2
bills, a coin, two $1 bills, a lottery ticket and the last two slides of this presentation.
•Dog System Exercise problems and solutions this is 21 files plus one folder we need computers for this exercise Load the files onto the desktop of the PCs before the
class
•Mathematical Summary MS Word Slides
•The student computers will need PowerPoint, MS Word and Excel.
•Optional handouts include Ben Franklin’s letter and the GOAL/QPC Creativity Tools Memory Jogger.
HistoryHistory•This course is based on material from Terry
Bahill’s Systems Engineering Process course at the University of Arizona.
•Bahill adapted it for BAE in the Fall of 2004 where it was reviewed by Rob Culver, Bill Wuersch, and John Volanski and it was piloted October 12-13, 2004.
•The human decision making material was added at the UofA in Fall 2005.