Interactive Requirements Prioritization Using Search Based Optimization Technique and Constraint Solver Acknowledgement: Dr. Paolo Tonella and Dr. Angelo Susi University of Trento, Italy Fondazione Bruno Kessler (FBK), Trento, Italy Francis Palma [email protected]Ecole Polytechnique de Montreal PhD. Supervisor: Dr. Yann-Gaël Guéhéneuc, Ecole Polytechnique Co-supervisor: Dr. Naouel Moha, UQAM Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions
64
Embed
Interactive Requirements Prioritization - Francis Palma, PhDfrancis-palma.net/wp-content/uploads/2016/04/Palma... · Interactive Requirements Prioritization Using Search Based Optimization
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Interactive Requirements Prioritization Using Search Based Optimization Technique and Constraint Solver
Acknowledgement: Dr. Paolo Tonella and Dr. Angelo Susi
University of Trento, Italy Fondazione Bruno Kessler (FBK), Trento, Italy
Problem: ‘Prioritization of Requirements’ To find the best ordering of requirements in each successive release to ensure quality & value of the delivered system, trade-off constraints & end-user satisfaction
Do Trade-off between user needs & real constraints
Now, highest quality & best valued system!
Extracted from Requirement Documents
Problem Description: An Example Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions
2 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
3 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
Requirements Prioritization?
A task performed in Software Project Management to determine which candidate requirements should be included in a certain release, negotiating all technical & non-technical constraints.
Requirements Prioritization matters, why?
Expectations ~ high; Timelines ~ short; Resources ~ limited, Make sure product contains the most essential functions, maximizing customer satisfaction and best resource utilization.
Prioritization dimensions may include:
Stakeholder expectations, business value, risk, cost, difficulty, time-constraints, dependencies among requirements etc.
5 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
req/software analyst
Classification: State-of-the-Art approaches
Prioritization Approaches Classification
using user knowledge either performing pair-wise comparison or not
using domain knowledge
using both
ü Domain Knowledge: Encoded, reusable requirements details or requirement specifications ü User Knowledge: When Requirement Analyst aware of complete domain knowledge
6 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
State-of-the-Art approaches (cont) Pairwise Comparison based approaches Analytic Hierarchy Process (AHP): comparing all unique pairs of requirements to determine which of the two is of higher priority, and to what extent [T.L. Saaty, ‘80]
Bubble Sort: compares two requirements & swap them if they are in the wrong order [Karlsson, ‘98] Cost-Value Approach: each requirement is determined on (i) the value to the users (ii) the cost of implementing, also uses the AHP technique [Karlsson & Ryan, ‘97] Case Based Ranking (CBRank): exploits a machine learning algorithm to guide the elicitation of user preferences during the prioritization process [Paolo Avesani et al ‘05]
7 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
State-of-the-Art approaches (cont.)
Non-Pairwise comparison based approaches Numerical Assignment: grouping requirements in different priority groups [Bradner, ‘97] MoScoW: groups all requirements into four priority groups MUST have, SHOULD have, COULD have, and WON’T have [Tudor and Walter, ‘06] Simple Ranking: requirements are ranked from integer 1 to N [Berander & Andrews, ‘05] Binary Search Tree: each node represents a requirement, lower priority requirements placed in left subtree, higher priority ones in right subtree, than node priority [Karlsson, ‘98] $100 Method: each stakeholder is asked to assume having $100 to distribute over the requirements in a ratio scale [Berander & Andrews ‘05]
Combining Techniques based approaches Planning Game: combination of two prioritization techniques i.e. Numerical Assignment & Simple Ranking [Beck, ‘99]
Domain Knowledge based approaches Priority Groups: dividing requirements into separate groups. then groups are ranked by using AHP [Joachim Karlsson] Genetic Algorithm: optimization is an application of GA & used in the problem of requirements prioritization too; uses domain knowledge [John Holland]
8 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
State-of-the-Art approaches (cont)
Summary Approach Cons
AHP Scalability Bubble Sort Scalability, time CBRank unable to accept constraints i.e. Dep Cost-Value Approach Time consuming
BST Sensitivity; a single error may build wrong tree GA Can’t resolve contradictory; $100 Method Longer; less confidence, biased MoScoW Ambiguous final ordering Simple Ranking unable handling complex scenarios
Scalability & handling of technical dependencies are the very common problems of all state-of-art approaches!
11 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
Genetic Algorithm 1. Acquisition and coding of set of Requirements and Domain Knowledge 2. Apply Genetic Algorithm (no user knowledge) 3. Output of the ranking (the most promising individual)
12 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
Genetic Algorithm Pseudo Code
The Canonical GA (a simple pseudo code is presented here):
1. choose initial population 2. evaluate each individual’s fitness REPEAT: 3. select best-ranking individuals to reproduce 4. apply crossover operator 5. apply mutation operator 6. evaluate each individual’s fitness until terminating condition (e.g. until at least one individual has the
desired fitness or enough generations have passed)
Disagreement: Between a pair of ordering (i.e. Pr1 and Prio | Dep | Eli), disagreement is the count of pairs that are inverted in two orderings. § Lower disagreement defines higher fitness
Formally define:
17 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver !
dis(pr1, pr2) = {(r,s)" pr1* | (s,r)" pr2*}
of 63
Disagreement Calculation Indv. ID Individual Disagr
20 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
GA Crossover Operator We used cut-head/fill-in-tail and cut-tail/fill-in-head…
R2 R3
Positions 5-6 as cut points
R8
Pr2
Pr3
Pr2’
R4 R1 R5 R8 R6 R7 R9
R5 R8 R1 R3 R7 R2 R9 R4 R6
R7 R9 R6 R2 R3 R4 R1 R5
- variation allows searching out different available niches, find better fitness values and subsequently better solutions - never produce chromosomes containing duplicate genes.
23 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
Our IGA Approach 1. Acquisition and coding of set of Requirements and Domain Knowledge 2. Apply Interactive Genetic Algorithm (exploiting User Knowledge) 3. Output of the ranking (the most promising individual)
25 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
Our Approach: User Feedback
Ties appears in population, - Contradictory information w.r.t. the initial constraints - Nothing is expressed explicitly in the initial constraints. etc... ... Simple example: Why (R7, R8)? Contradictory or Ambiguous w.r.t. Prio & Dep..
29 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
The Case Study: Target System (ACube) § Prioritize requirements for a real software system, as part of the project *ACube (Ambient Aware Assistance)
designing a highly technological monitoring environment to be deployed in nursing homes to support medical and assistance staff
§ After end-user requirements analysis phase, 60 user requirements and 49 technical requirements Four macro-scenarios have been identified.
ID Macro-Scenario # of requirements
FALL Monitoring falls 26
ESC Monitoring escapes 23
MON Monitoring dangerous behavior 21
ALL The three scenarios 49
*ACube is a social welfare project coordinated by Fondazione Bruno Kessler (FBK) and funded by Autonomous Province of Trento under Bando Grandi Progetti, 2006. *http://acube.fbk.eu/
30 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
Case Study: The Overall Process Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions
31 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
§ Collected user and technical requirements § Identified macro-scenarios § Tech. requirements has priority values (range 1-500) defined as a function that indicates priority-level of the tech. requirement w.r.t. the priority of the user requirements § Software architect analyzed the requirement documents & extracted requirement implementation dependencies § Software architect also defined a Gold Standard (GS) i.e. the ordering that would be followed at the end --- ---
§ With Prio info, Dep info and GS, we applied IGA Algorithm and generated Eli info during runtime, and evolve… § IGA returns an ordering applying GA operators with user’s (Requirements Analyst) assistance. § We then measure fitness against Gold Standard (GS)
of 63
Gold Standard (GS)
§ For each of the four macro-scenarios, we obtained the Gold Standard (GS) prioritization from the Software Architect of the ACube project
ü The GS prioritization is the ordering given by the software architect to the requirements when planning their implementation during the ACube project.
Why Gold Standard?
ü To measure disagreements with respect to GS. ü To evaluate our approach in terms of disagreement against other non-interactive approaches using the same GS.
33 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
Research Question - 1 RQ1 (Convergence) Can we observe convergence with respect to the finally elicited fitness function?
- Convergence is not obvious immediately, as Eli Graph is evolving at early stages. - Although the full fitness function is known only at end of elicitation process.
35 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
Research Question - 3 RQ3 (Role of initial precedence constraints) How does initial availability of precedence constraints affect the performance of IGA?
- Different types of Domain Knowledge affect IGA significantly - The improvement of IGA over GA is even higher when limited ranking information is available
37 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
General Discussion § Up to now we propose an Interactive approach for requirements prioritization
§ We experimented on a real project i.e. ACube
§ We also verified robustness w.r.t other non-interactive approaches Analytical Hierarchy Process (AHP):
- Most common approach for requirements prioritization… - Decision maker decides for every unique pairs based on different criteria… - For a system with N requirements and C criteria, AHP will have, total C × (C − 1) + C × N × (N − 1) / 2 comparisons (a huge manual task!) So, only with 25/50/100 (max) pairwise comparisons and acceptable performance IGA makes AHP obsolete
39 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
• Based on Sa$sfiability Modulo Theory* and on Interac9ve Pairwise User Feedback – aims at minimizing the disagreement between
• a total order of prioritized requirements (Gold Standard) • constraints that are either encoded with the requirements
or expressed iteratively by the user during prioritization process
* a decision problem for logical formulas expressed in classical
first-order logic with equality or inequality
We try to overcome some of the limits of other approaches: exploiting user knowledge, minimizing the requests to users to pay attention to scalability problems, considering the possibility of arbitrary constraints and assuring robustness to errors
40 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
• A set of Requirements
• Domain Knowledge: available as requirements documenta9on (e.g., cost of the implementa9on, value for the stakeholders, dependencies between requirements) that can be converted into total or par9al rankings of the requirements
Req Cost Dependencies
R1 High R2, R3 R2 Low R3 R3 Low
R4 Medium R3 R5 Medium
• Evaluation from users in terms of orderings between pairs of requirements
The Input Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions
41 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
1. Acquisi9on and coding of Domain Knowledge for the set of requirements under analysis
2. Computa9on of solu9ons via SMT, also exploi9ng evalua9ons from decision makers
3. Output of the ranking
R3 R2
R1 R4
R1 R2
R3
R4
R5
Domain knowledge (constraints) User feedback
R1 R2 R3 R4
Set of requirements
Interac<ve SMT
R2
R1
R3
R4
<,> ?
The Main Approach Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions
42 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
2.1. Computa9on of a set of solu9ons via SMT solver (Yices* tool) 2.2. If the number of solu9ons is grater than 1
– then: iden9fica9on of conflicts (9es) between the solu9ons and the constraints
– else: exit and return the solu9on 2.3. Request of knowledge to users to decide about conflicts 2.4. If max number of itera9ons / max pair comparison
44 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
Req Value (Priori<es) Dependencies
R1 High R2, R3 R2 Low R3 R3 Low
R4 Medium R3 R5 Medium
Transform the domain knowledge into graphs
R3 R2
R1
R5 R4
Priorities
R3 R2
R1 R4
Dependencies
Domain Knowledge coding into Graphs Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions
45 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
requirements and the various constraints that are either en-coded with the requirements or that are expressed itera-tively by the user during the prioritization process. Weuse SMT solvers to achieve such a minimization, takingadvantage of interactive input from the requirement engi-neer whenever the solver produces more than one prioritizedlist of requirements having the same disagreement with theavailable constraints. The prioritization process terminateswhen a unique solution at minimum disagreement is foundor the maximum allowed number of pairwise comparisons isreached.
Req Prio DepsR1 High R2, R3
R2 Low R3
R3 LowR4 Medium R2
R5 Medium
4 5
1
2 3
R
R R
R R
3 2
1 4
R R
R R
Prio Deps
Table 2: Requirements with priority and dependen-cies
Let us consider the five requirements listed in Table 2.For each requirement we consider the priority expressed bythe requirement engineer who collected them and the depen-dencies the requirement may have with other requirements.For conciseness, in Table 2 we omit other important ele-ments of a requirement (e.g., the textual description). Con-straints upon requirements can be represented by means ofa constraint graph. In a constraint graph, an edge betweentwo requirements indicates that, according to the relateddependency or priority, the requirement associated with thesource of the edge should be implemented before the targetrequirement. Edges may be weighted, to actually quantifythe strength or importance of each constraint. When en-coded in the input language of the SMT solver, such con-straints will be retractable constraints which are given theweight labeling the edge (or 1 if no weight is given). Aninfinite weight is used for constraints that must necessarilyhold in the final ordering of the requirements. These will benon-retractable constraints for the SMT solver.
At the right of Table 2, we show the constraint graphinduced by the priority (Prio) property of the requirementsand the constraint graph induced by the dependencies (Deps)between requirements. Requirements with High priority shouldprecede those with Medium priority. Hence the edges (R1, R4)and (R1, R5) in the graph Prio. Similarly, the precedence be-tween Medium and Low priority requirements induces thefour edges at the bottom of the constraint graph Prio. Re-quirement R1 depends on R2, R3, hence the implementationof R2, R3 should precede that of R1, which gives raise to theedges (R2, R1) and (R3, R1) in Deps.
It is possible to encode the requirements prioritizationproblem as a MAX-SAT2 problem. An SMT solver willfind an integer assignment which maximizes the weight ofthe retractable constraints that are satisfied by the solution(minimum cost of unsatisfied constraints). The encoding isdescribed formally in the next section. Intuitively, it con-sists of an assignment of positions (integers between 1 and2Technically we are exploiting a MAX-SMT solver.
Figure 1: Encoding of the constraints in Table 2 forthe SMT solver Yices
N , for N requirements) to the components of an integerarray x of size N . The assignment must define a permu-tation of the positions, hence there should be no repetitionof positions. This is easily encoded as a set of inequalitiesbetween pairs of positions (x[i] != x[j] " i != j). The otherconstraints (from the constraint graphs) are encoded as in-equalities (e.g., x[1] < x[4] and x[1] < x[5] for the two edgesat the top of the constraint graph Prio). The solutions tothe MAX-SAT problem produced by the SMT solver are allrequirement orderings that violate the minimum number ofretractable constraints (e.g., Deps and Prio), or have min-imum cost of violation, in case di!erent weights are givento di!erent constraints. The encoding of the requirementprioritization problem in Table 2 for the SMT solver Yices3
Table 3: Prioritized requirements and related mini-mum disagreement
After encoding the constraint graphs in Table 2 as re-tractable assertions (assert+ in Figure 1) and running anSMT solver (e.g., Yices) on them, we obtain the four priori-tized lists of requirements shown in Table 3 (the solver mustbe run four times to obtain the complete set of prioritiza-tions, each time negating the previously found solutions).These are all prioritized lists having the same, minimumcost of retracted assertions (i.e., 4). Such a cost is called dis-agreement and it represents the number of constraints thatare not respected by the resulting requirement positions.The first two prioritizations Pr1 and Pr2 are in complete
3http://yices.csl.sri.com/
R3 R2
R1 R4
Dependencies
R3 R2
R1
R5 R4
Priorities
Retractable assertions
Problem SMT Encoding
Retractable assertions: assert+ can be retracted. (details on http://yices.csl.sri.com/)
46 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
(assert+ [expr] [weight])
of 63
requirements and the various constraints that are either en-coded with the requirements or that are expressed itera-tively by the user during the prioritization process. Weuse SMT solvers to achieve such a minimization, takingadvantage of interactive input from the requirement engi-neer whenever the solver produces more than one prioritizedlist of requirements having the same disagreement with theavailable constraints. The prioritization process terminateswhen a unique solution at minimum disagreement is foundor the maximum allowed number of pairwise comparisons isreached.
Req Prio DepsR1 High R2, R3
R2 Low R3
R3 LowR4 Medium R2
R5 Medium
4 5
1
2 3
R
R R
R R
3 2
1 4
R R
R R
Prio Deps
Table 2: Requirements with priority and dependen-cies
Let us consider the five requirements listed in Table 2.For each requirement we consider the priority expressed bythe requirement engineer who collected them and the depen-dencies the requirement may have with other requirements.For conciseness, in Table 2 we omit other important ele-ments of a requirement (e.g., the textual description). Con-straints upon requirements can be represented by means ofa constraint graph. In a constraint graph, an edge betweentwo requirements indicates that, according to the relateddependency or priority, the requirement associated with thesource of the edge should be implemented before the targetrequirement. Edges may be weighted, to actually quantifythe strength or importance of each constraint. When en-coded in the input language of the SMT solver, such con-straints will be retractable constraints which are given theweight labeling the edge (or 1 if no weight is given). Aninfinite weight is used for constraints that must necessarilyhold in the final ordering of the requirements. These will benon-retractable constraints for the SMT solver.
At the right of Table 2, we show the constraint graphinduced by the priority (Prio) property of the requirementsand the constraint graph induced by the dependencies (Deps)between requirements. Requirements with High priority shouldprecede those with Medium priority. Hence the edges (R1, R4)and (R1, R5) in the graph Prio. Similarly, the precedence be-tween Medium and Low priority requirements induces thefour edges at the bottom of the constraint graph Prio. Re-quirement R1 depends on R2, R3, hence the implementationof R2, R3 should precede that of R1, which gives raise to theedges (R2, R1) and (R3, R1) in Deps.
It is possible to encode the requirements prioritizationproblem as a MAX-SAT2 problem. An SMT solver willfind an integer assignment which maximizes the weight ofthe retractable constraints that are satisfied by the solution(minimum cost of unsatisfied constraints). The encoding isdescribed formally in the next section. Intuitively, it con-sists of an assignment of positions (integers between 1 and2Technically we are exploiting a MAX-SMT solver.
Figure 1: Encoding of the constraints in Table 2 forthe SMT solver Yices
N , for N requirements) to the components of an integerarray x of size N . The assignment must define a permu-tation of the positions, hence there should be no repetitionof positions. This is easily encoded as a set of inequalitiesbetween pairs of positions (x[i] != x[j] " i != j). The otherconstraints (from the constraint graphs) are encoded as in-equalities (e.g., x[1] < x[4] and x[1] < x[5] for the two edgesat the top of the constraint graph Prio). The solutions tothe MAX-SAT problem produced by the SMT solver are allrequirement orderings that violate the minimum number ofretractable constraints (e.g., Deps and Prio), or have min-imum cost of violation, in case di!erent weights are givento di!erent constraints. The encoding of the requirementprioritization problem in Table 2 for the SMT solver Yices3
Table 3: Prioritized requirements and related mini-mum disagreement
After encoding the constraint graphs in Table 2 as re-tractable assertions (assert+ in Figure 1) and running anSMT solver (e.g., Yices) on them, we obtain the four priori-tized lists of requirements shown in Table 3 (the solver mustbe run four times to obtain the complete set of prioritiza-tions, each time negating the previously found solutions).These are all prioritized lists having the same, minimumcost of retracted assertions (i.e., 4). Such a cost is called dis-agreement and it represents the number of constraints thatare not respected by the resulting requirement positions.The first two prioritizations Pr1 and Pr2 are in complete
3http://yices.csl.sri.com/
R3 R2
R1 R4
Dependencies
R3 R2
R1
R5 R4
Priorities
Completing the Encoding Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions
47 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
Pairs to be Evaluated from the Decision Maker Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions
49 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
R3 R2
R1
R5 R4
Priorities
R3 R2
R1 R4
Dependencies
Why (R4,R5) ? Nothing is said about (R4,R5) in the Priorities and Dependencies graphs
Why (R3,R4) ?
Contradiction in ordering (R3,R4)
TIE PAIRS
Pr1, Pr2, (R4, R5)
Pr1, Pr3 (R3, R4)
… … <>?
R3
R4 R4
R5
User Preference Graph eliOrd
< or >
User Feedback Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions
50 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
R3 R2
R1
R5 R4
Priorities
R3 R2
R1 R4
Dependencies
R3
R4
R5
User Preference Graph eliOrd
New Graph eliOrd from User Feedback Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions
51 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
Evaluation of Approach B
52
• Priori9ze requirements for a real soaware system, as part of the project ACube (Ambient Aware Assistance) – designing a highly technological monitoring environment to be deployed in nursing
homes to support medical and assistance staff • Aaer user requirements analysis phase,
– 60 user requirements where retrieved that produced 49 technical requirements – Four macro-‐scenarios have been iden9fied.
Id Macro-‐scenario # of requirements
FALL Monitoring falls of pa9ents 26
ESC Monitoring escapes of pa9ents 23
MON Monitoring dangerous behavior of pa9ents 21
ALL The three scenarios 49
Case Study: ACube Project Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions
53 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
• Two sets of technical constraints: – Prio graph: a func9on that associates technical requirement to a number
indica9ng the priority of the technical requirement with respect to the priority of the user requirements it is intended to address
– Dep graph: is defined on the basis of the dependencies between technical requirements
• For each of the four macro-‐scenarios, we considered as the Gold Standard (GS) the priori9za9on from the soaware architect of the ACube project – The GS priori9za9on is the ordering given by the soaware architect to the
requirements when planning their implementa9on during the ACube project
• Comparison with “Incomplete Analy9c Hierarchy Process” and “Interac9ve Gene9c Algorithm”
Ingredients for the Evaluation Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions
54 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
• Is a variant of the Analy9c Hierarchy Process (AHP) • Pairwise comparison itera9ve process in which the decision
maker is required to specify a preference between two requirements Ri and Rj as an integer value pij∈[1 . . . 9]
coming from other sources.We evaluated our SMT approach comparing it with other
state-of-the-art interactive prioritization techniques (IAHPand IGA in particular). Results indicate that SMT substan-tially outperforms them, while keeping the user e!ort (interms of number of elicited pairs) acceptable. Moreover, wecompared interactive and non-interactive prioritization, ob-serving an increased performance when interaction with thedecision maker takes place. We also evaluated the robust-ness of our method with respect to decision maker’s errors.
This paper is structured as follows: in Section 2 we de-scribe the background for our work, Section 3 presents theproposed approach and algorithm. In Section 4 we commenta set of experimental evaluations on e!ectiveness, robust-ness and e"ciency of the approach. The empirical assess-ment was conducted on a set of requirements from a realhealthcare project. Relevant related works are discussed inSection 5. Conclusions and future work are presented inSection 6.
2. BACKGROUNDHere we briefly introduce two quite successful, state-of-
the-art methods for interactive prioritization of requirements:Incomplete Analytic Hierarchy Process (IAHP) [4] and In-teractive Genetic Algorithm (IGA) [14]. Both methods havethe objective of synthesizing an approximation of the rank-ing of a set of N requirements, minimizing the e!ort asso-ciated with the interactive input required from the decisionmaker (consisting of pairwise comparisons).
2.1 Incomplete AHPIAHP [4] is a variant of the Analytic Hierarchy Process
(AHP) [12] that can deal with incomplete information (pair-wise comparisons) from the user. It was designed to supportscalability of AHP to requirements whose size make AHPprohibitively expensive (the number of pairs that must beelicited to apply AHP is quadratic with the number of re-quirements).
In particular, AHP implements a pairwise comparison it-erative process in which the decision maker is required tospecify a preference as an integer value pij ! [1 . . . 9], be-tween two requirements Ri and Rj . The value can be mappedto a qualitative measure of the preference relation, as sum-marized in Table 1. After assigning a preference value to oneof the two requirements (say, Ri) compared to the other one(Rj), the preference value for the least preferred requirement(Rj) is the reciprocal of pij , so, pji = 1/pij . When all thepairs have been elicited, a ranking is synthesized throughthe computation of the principal eigenvector of the matrixpij , the components of which determine the rank of eachrequirement.
Preference pij Definition
1 Ri Equally important to Rj
3 Ri Moderately more important than Rj
5 Ri Strongly more important than Rj
7 Ri Very strongly more important than Rj
9 Ri Extremely more important than Rj
2, 4, 6, 8 For compromise between the above values
Table 1: A possible fundamental scale in the interval[1 . . . 9] used with AHP
IAHP overcomes the AHP scalability problem by mini-mizing the number of pairs elicited from the decision maker,while maintaining a good trade-o! between the precision ofthe final solution and the e!ort of the decision maker. Thisis done by calculating, at each iteration of the elicitationprocess, a prediction of the next most promising pair to beasked to the decision maker in order to find a stable andgood approximation of the target ranking, early before elic-iting all the N(N "1)/2 pairs for the set of N requirements.
The high level steps of the IAHP process are: (i) thedecision maker provides N "1 judgments which form a con-nected graph; (ii) using the available pairwise comparisons,the missing comparisons are estimated by taking the geomet-ric mean of the intensities over a spanning tree (this resultsin a weight matrix); (iii) the derivatives of the weight matrixwith respect to the missing matrix elements are computedand the next pairwise comparison to ask is determined basedon such derivatives; (iv) if the stopping criteria are met (e.g.,maximum number of elicitations or convergence metrics be-low threshold), then the computation stops and a rank isproduced by calculating the principal eigenvector of the es-timated matrix; otherwise the selected comparison is elicitedand the algorithm iterates.
2.2 Interactive GAIn [14] we presented an Interactive Genetic Algorithm
(IGA) to achieve the minimization of the disagreement be-tween a total order of prioritized requirements and the var-ious constraints coming from the domain knowledge, thatare either encoded with the requirements or expressed iter-atively by the user during the prioritization process.
In the approach we exploit the interactive input from theuser whenever the fitness function cannot be computed pre-cisely based on the information available. Specifically, eachindividual in the population being evolved by the genetic al-gorithm represents an alternative prioritization Pri of theN requirements. Two examples of such individuals are:Pr1 = #R1, R5, R4, R3, R2$, or Pr2 = #R1, R4, R5, R3, R2$.When individuals having a high fitness (i.e., a low disagree-ment with the constraints) cannot be distinguished, sincetheir fitness function evaluates to a plateau, user input isrequested interactively, in the form of preferences betweenpairs of requirements, so specifying if Ri is before or afterRj in the user ideal ranking, so as to make the fitness func-tion landscape better suited for further minimization. Inthis way, knowledge about the constraints in the domaincan help the method to minimize the amount of knowledgeelicited from the decision maker, so decreasing her/his ef-fort. The prioritization process terminates when a low dis-agreement is reached, a time out is reached or the allocatedelicitation budget is over.
3. APPROACHThe prioritization approach we propose aims at minimiz-
ing the disagreement between a total order of prioritizedrequirements and the various constraints that are either en-coded with the requirements or that are expressed itera-tively by the user during the prioritization process. Weuse SMT solvers to achieve such a minimization, takingadvantage of interactive input from the requirement engi-neer whenever the solver produces more than one prioritizedlist of requirements having the same disagreement with theavailable constraints. The prioritization process terminates
**P. T. Harker. Incomplete Pairwise Comparisons in the Analytic Hierarchy Process. Math. Modelling, 9(11):837 – 848, 1987 **T. L. Saaty and L. G. Vargas. Models, Methods, Concepts & Applications of the Analytic Hierarchy Process. Kluwer Academic, 2000
55 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
• The IGA method consists in a Gene<c Algorithm associated to a Pairwise Interac<ve and Itera<ve process to acquire user preferences.
• Differently from IAHP, the priori9za9on elicited by the user is strict (Ri less or more important than Rj, no range as to be specified)
P. Tonella, A. Susi, and F. Palma. Using Interactive GA for Requirements Prioritization. In 2nd International Symposium on Search Based Software Engineering, pages 57–66. IEEE, 2010
56 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
RQ1 (Comparison) Does the SMT-based method produce improved prioritizations compared to IAHP and IGA?
Figure 2: Disagreement with GS after eliciting atmost 25 (top), 50 (middle) and 100 (bottom) pairsfrom the user in the ALL scenario
RQ3 (Robustness).In Figure 4, we show that even if the performance of the
SMT-based interactive approach degrades at increasing usererror rates, it always produces an improved prioritizationcompared to IGA and IAHP with a user who does not makeany mistake. We show this for 50 elicited pairs, but sim-ilar results are available for a di!erent number of elicitedpairs. With no error SMT produces a prioritization with adisagreement of 90. At 5%, 10% and 20% error rate, thedisagreement becomes 92, 95 and 98 respectively. On theother hand, even if the user makes no error (i.e. error-freeuser), the disagreement of the prioritizations produced byIGA and IAHP is 120 and 208. We obtained similar plotsfor the other three macro scenarios (MON, FALL, ESC).
In Figure 5 we compare the robustness of SMT with thetwo other approaches considered in this work, IGA and IAHP.Here we show the results obtained for ALL. Similar resultshave been obtained for the other macro scenarios. The SMT-based interactive approach is substantially more robust thanthe others. At increasing user error rates, the performanceof IGA and IAHP is degrading to a higher degree. SMT
Figure 3: Disagreement (top) and average dis-tance (bottom) with GS for interactive and non-interactive SMT after eliciting 25, 50, 100 pairs fromthe user in the ALL scenario
has also a degrading performance at increased error rates,but it outperforms IGA and IAHP to a large extent. For anerror rate of 5% SMT, IGA and IAHP have a disagreementof 92, 120 and 236 respectively. For a 10% error rate, theseare 95, 119 and 277. For an error rate of 20%, the disagree-ments with GS become 98, 123 and 331 respectively. So,clearly the SMT-based interactive approach is more robustthen IGA and IAHP.
RQ4 (Performance).Table 8 shows the comparison among execution times of
SMT, IGA and IAHP. All execution times in the table areobtained from the system time of each execution and aremeasured in seconds, with a very small observable variabil-
Figure 4: Robustness of SMT: Disagreement forSMT at di!erent user error rates; IGA and IAHPwith error-free user, after eliciting 50 pairs in theALL scenario
57 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
RQ2 (Role of interaction) Does SMT-based interactive prioritization produce improved prioritizations compared to non-interactive SMT-based prioritization?
Figure 2: Disagreement with GS after eliciting atmost 25 (top), 50 (middle) and 100 (bottom) pairsfrom the user in the ALL scenario
RQ3 (Robustness).In Figure 4, we show that even if the performance of the
SMT-based interactive approach degrades at increasing usererror rates, it always produces an improved prioritizationcompared to IGA and IAHP with a user who does not makeany mistake. We show this for 50 elicited pairs, but sim-ilar results are available for a di!erent number of elicitedpairs. With no error SMT produces a prioritization with adisagreement of 90. At 5%, 10% and 20% error rate, thedisagreement becomes 92, 95 and 98 respectively. On theother hand, even if the user makes no error (i.e. error-freeuser), the disagreement of the prioritizations produced byIGA and IAHP is 120 and 208. We obtained similar plotsfor the other three macro scenarios (MON, FALL, ESC).
In Figure 5 we compare the robustness of SMT with thetwo other approaches considered in this work, IGA and IAHP.Here we show the results obtained for ALL. Similar resultshave been obtained for the other macro scenarios. The SMT-based interactive approach is substantially more robust thanthe others. At increasing user error rates, the performanceof IGA and IAHP is degrading to a higher degree. SMT
Figure 3: Disagreement (top) and average dis-tance (bottom) with GS for interactive and non-interactive SMT after eliciting 25, 50, 100 pairs fromthe user in the ALL scenario
has also a degrading performance at increased error rates,but it outperforms IGA and IAHP to a large extent. For anerror rate of 5% SMT, IGA and IAHP have a disagreementof 92, 120 and 236 respectively. For a 10% error rate, theseare 95, 119 and 277. For an error rate of 20%, the disagree-ments with GS become 98, 123 and 331 respectively. So,clearly the SMT-based interactive approach is more robustthen IGA and IAHP.
RQ4 (Performance).Table 8 shows the comparison among execution times of
SMT, IGA and IAHP. All execution times in the table areobtained from the system time of each execution and aremeasured in seconds, with a very small observable variabil-
Figure 4: Robustness of SMT: Disagreement forSMT at di!erent user error rates; IGA and IAHPwith error-free user, after eliciting 50 pairs in theALL scenario
!
dis(pr1, pr2) = {(r,s)" pr1* | (s,r)" pr2*}
RQ2: Role of Interaction Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions
58 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
RQ3 (Robustness) Is the SMT-based method more robust than IAHP and IGA with respect to errors committed by the user during the elicitation of pairwise comparisons?
Figure 2: Disagreement with GS after eliciting atmost 25 (top), 50 (middle) and 100 (bottom) pairsfrom the user in the ALL scenario
RQ3 (Robustness).In Figure 4, we show that even if the performance of the
SMT-based interactive approach degrades at increasing usererror rates, it always produces an improved prioritizationcompared to IGA and IAHP with a user who does not makeany mistake. We show this for 50 elicited pairs, but sim-ilar results are available for a di!erent number of elicitedpairs. With no error SMT produces a prioritization with adisagreement of 90. At 5%, 10% and 20% error rate, thedisagreement becomes 92, 95 and 98 respectively. On theother hand, even if the user makes no error (i.e. error-freeuser), the disagreement of the prioritizations produced byIGA and IAHP is 120 and 208. We obtained similar plotsfor the other three macro scenarios (MON, FALL, ESC).
In Figure 5 we compare the robustness of SMT with thetwo other approaches considered in this work, IGA and IAHP.Here we show the results obtained for ALL. Similar resultshave been obtained for the other macro scenarios. The SMT-based interactive approach is substantially more robust thanthe others. At increasing user error rates, the performanceof IGA and IAHP is degrading to a higher degree. SMT
Figure 3: Disagreement (top) and average dis-tance (bottom) with GS for interactive and non-interactive SMT after eliciting 25, 50, 100 pairs fromthe user in the ALL scenario
has also a degrading performance at increased error rates,but it outperforms IGA and IAHP to a large extent. For anerror rate of 5% SMT, IGA and IAHP have a disagreementof 92, 120 and 236 respectively. For a 10% error rate, theseare 95, 119 and 277. For an error rate of 20%, the disagree-ments with GS become 98, 123 and 331 respectively. So,clearly the SMT-based interactive approach is more robustthen IGA and IAHP.
RQ4 (Performance).Table 8 shows the comparison among execution times of
SMT, IGA and IAHP. All execution times in the table areobtained from the system time of each execution and aremeasured in seconds, with a very small observable variabil-
Figure 4: Robustness of SMT: Disagreement forSMT at di!erent user error rates; IGA and IAHPwith error-free user, after eliciting 50 pairs in theALL scenario
60 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
Algorithm selection
Experiment parameters setting
Pairwise elicitation
The Prioritization Tool & Technical Report
TR-FBK-SE-2011-3: Francis Palma, Angelo Susi, Paolo Tonella: Using an SMT Solver for Interactive Requirements Prioritization. Tech. Rep. FBK-SE (Mar. 2011) 61 of 63
Conclusions & Future Works q We proposed an Interactive Genetic Algorithm & an Interactive SMT to collect pairwise information useful to prioritize the requirements
q We also verified the robustness of the algorithms with respect to increased error-prone responses from analysts
q We evaluated the approach in a real project (ACube)
q In summary, we contributed NOVEL APPROACH to prioritize requirements What’s Next? Experiment
- On-line: Empirical Study with real object (i.e. human/analyst)
62 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
Publications
- Paolo Tonella, Angelo Susi, and Francis Palma. Using Interactive GA for Requirements Prioritization. In 2nd International Symposium on Search Based Software Engineering (SSBSE 2010), pages 57–66. IEEE, 2010 (Under minor review for the Journal of Information and Software Technology (IST)) - Francis Palma, Angelo Susi, Paolo Tonella, Using an SMT solver for interactive requirements prioritization, in Tibor Gyimothy and Andreas Zeller, SIGSOFT FSE, ACM, 2011, pp. 48-58 (19th ACM SIGSOFT Symposium on the Foundations of Software Engineering, Szeged, Hungary)da 09/05/2011 a 09/09/2011