This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
IBM T.J. Watson Research Center & Northwestern University
Initialize a model of configuration complexity and demonstrates its value for a change management system.
Metrics that indicate some configuration complexity, including execution complexity, parameter complexity, and memory complexity.
Example: complexity of J2EE provisioning
DB2:
WAS:
createuser
install wiz.(db2)
selectfeatures
(2)
specifycfg. params(7, 5 def.)
reboot app cmd(cli)
env var(3)
cfg file(1 line)
app cmd(cli)
start svc(db2) ...
cfg file(3 lines)
install wiz(db2)
selectfeatures
(4)
app cmd(cli)
app cmd(cli)
install wiz(was)
...
user/pass
db2host
instance
db2_portsvc_name
db_name
node alias
= control flow= cfg data flow
DB2:
WAS:
createuser
install wiz.(db2)
selectfeatures
(2)
specifycfg. params(7, 5 def.)
reboot app cmd(cli)
env var(3)
cfg file(1 line)
app cmd(cli)
start svc(db2) ...
cfg file(3 lines)
install wiz(db2)
selectfeatures
(4)
app cmd(cli)
app cmd(cli)
install wiz(was)
...
DB2:
WAS:
createuser
install wiz.(db2)
selectfeatures
(2)
specifycfg. params(7, 5 def.)
reboot app cmd(cli)
env var(3)
cfg file(1 line)
app cmd(cli)
start svc(db2) ...
cfg file(3 lines)
install wiz(db2)
selectfeatures
(4)
app cmd(cli)
app cmd(cli)
install wiz(was)
...
createuser
install wiz.(db2)
selectfeatures
(2)
specifycfg. params(7, 5 def.)
reboot app cmd(cli)
env var(3)
cfg file(1 line)
app cmd(cli)
start svc(db2) ...create
userinstall wiz.
(db2)
selectfeatures
(2)
specifycfg. params(7, 5 def.)
reboot app cmd(cli)
env var(3)
cfg file(1 line)
app cmd(cli)
start svc(db2) ...
cfg file(3 lines)
install wiz(db2)
selectfeatures
(4)
app cmd(cli)
app cmd(cli)
install wiz(was)
cfg file(3 lines)
install wiz(db2)
selectfeatures
(4)
app cmd(cli)
app cmd(cli)
install wiz(was)
...
user/pass
db2host
instance
db2_portsvc_name
db_name
node alias
user/pass
db2host
instance
db2_portsvc_name
db_name
node alias
= control flow= cfg data flow= control flow= cfg data flow
Process complexity: manual
• Execution• 59 steps, 27 context switches
• Parameter• 32 parameters used 61 times,
18 outside of source context • Source score: 125
• Memory (LIFO stack model)• Size: max 8, avg 4.4
5 1
17 17
094
0 0
automated
See: Brown, A.B., A. Keller, and J.L. Hellerstein. A Model of Configuration Complexity and Its Application to a Change Management System. Proceedings of the Ninth IFIP/IEEE International Symposium on Integrated Network Management (IM 2005), Nice, France, May 2005.
IBM T.J. Watson Research Center & Northwestern University
Overview– 3 experiments, 10 testcases with 1 warm-up– 1st stage, 35 users– 2nd stage, 23 users, with refined experiment
Metrics– Average time spent per step (e.g. time / no. of steps)– User rating (in the end of each experiment)– Error rate (user picked non-optimal path)
Analysis approach– Step I: general statistical analysis of all data
• Each testcase measured as an independent data point• Goal: identify factors that explain the most variance
– Step II: pair-wise testcase comparisons• Get more insight into specific effects of factor value• Goal: remove inter-user variance
IBM T.J. Watson Research Center & Northwestern University
– Depending on its goal (user, time or error rate), optimization for less complexity will have different focus, examples:• An installation procedure with easily-located clear info (e.g. wizard-
based prompts) for the next step will reduce both task time and user-perceived complexity
• A procedure with feedback on the current state of the system and the effect of the previous action (e.g. message windows following a button press) will only reduce user-perceived complexity, but unlikely to improve task time or error rate
• Omitting positional feedback (i.e., not showing users effects of their actions) may, counterintuitively, increase user accuracy, but at cost of significantly higher perceived complexity and task time
IBM T.J. Watson Research Center & Northwestern University