Combining aspect-oriented modeling with property-based reasoning to improve user interface adaptation

Combining Aspect-Oriented Modelingwith Property-Based Reasoning

to Improve User Interface Adaptation

Arnaud BlouinIRISA, Triskell, [email protected]

Brice MorinSINTEF ICT, Oslo

[email protected]

Olivier BeaudouxESEO-GRI, Angers

[email protected] Nain

INRIA, Triskell, [email protected]

Patrick AlbersESEO-GRI, Angers

[email protected]

Jean-Marc JezequelIRISA, Triskell, Rennes

[email protected]

ABSTRACTUser interface adaptations can be performed at runtimeto dynamically reflect any change of context. Complexuser interfaces and contexts can lead to the combinato-rial explosion of the number of possible adaptations.Thus, dynamic adaptations come across the issue ofadapting user interfaces in a reasonable time-slot withlimited resources. In this paper, we propose to com-bine aspect-oriented modeling with property-based rea-soning to tame complex and dynamic user interfaces.At runtime and in a limited time-slot, this combina-tion enables efficient reasoning on the current contextand on the available user interface components to pro-vide a well suited adaptation. The proposed approachhas been evaluated through EnTiMid, a middleware forhome automation.

Author KeywordsMDE, user interface, context, adaptation, aspect, run-time, malai

ACM Classification KeywordsH.5.2 Information Interfaces and Presentation: User In-terfaces—Theory and methods, User Interface Manage-ment Systems (UIMS); D.2.1 Software Engineering: Re-quirements/ Specifications—Methodologies; H.1.0 In-formation Systems: Models and Principles—General

General TermsDesign

INTRODUCTIONThe number of platforms having various interactionmodalities (e.g., netbook and smart phone) unceasingly

Author version

increases over the last decade. Besides, user’s prefer-ences, characteristics and environment have to be con-sidered by user interfaces (UI). This triplet <platform,user, environment>, called context1 [8], leads user in-terfaces to be dynamically (i.e. at runtime) adaptableto reflect any change of context.

UI components such as tasks and interactions, enabledfor a given context but disabled for another one, causea wide number of possible adaptations. For example,[14] describes an airport crisis management system thatleads to 1,474,560 possible adaptations. Thus, an im-portant challenge is to support UI adaptation of com-plex systems. This implies that dynamic adaptationsmust be performed in a minimal time, and respectingusability.

The contribution of this paper is to propose an ap-proach that combines aspect-oriented modeling (AOM)with property-based reasoning to tackle the combina-torial explosion of UI adaptations. AOM approachesprovide advanced mechanisms for encapsulating cross-cutting features and for composing them to form models[1]. AOM has been successfully applied for the dynamicadaptation of systems [20]. Property-based reasoningconsists in tagging objects that compose the systemwith characterizing properties [14]. At runtime, theseproperties are used by a reasoner to perform the adap-tation the best suited to the current context. Reasoningon a limited number of aspects combined with the useof properties avoids the combinatorial explosion issue.Although these works tackle system adaptation at run-time, they do not focus on the dynamic adaptation ofUIs. Thus, we mixed these works with Malai, a mod-ular architecture for interactive systems [4], to bringcomplex and dynamic user interface adaptations undercontrol. We have applied our approach to EnTiMid, amiddleware for home automation.

The paper is organized as follows. The next sectionintroduces background research works used by our ap-

1In this paper, the term ”context” is used instead of ”con-text of use” for conciseness.

inria

-005

9089

1, v

ersi

on 1

- 5

May

201

1Author manuscript, published in "ACM SIGCHI Symposium on Engineering Interactive Computing Systems (2011) 85--94"

DOI : 10.1145/1996461.1996500

proach. Then, the process to create an adaptive UIusing our approach is explained. Next, the adaptationprocess that is automatically executed at runtime is de-tailed. Following, our approach is evaluated throughEnTiMiD, a middleware for house automation. Thepaper ends with the related work and the conclusion.

BACKGROUNDThe work presented in this paper brings an interac-tive system architecture and a software engineering ap-proach together. Thus, this section starts with the pre-sentation of the Malai architecture. The software engi-neering approach applied to Malai to allow complex UIadaptations at runtime is then introduced.

The Malai ArchitectureThe work presented in this paper is based on Malai, anarchitectural model for interactive systems [4]. In Malaia UI is composed of presentations and instruments (seeFigure 1). A presentation is composed of an abstractpresentation and a concrete presentation. An abstractpresentation is a representation of source data createdby a Malan mapping (link ¬). A concrete presentationis the graphical representation of the abstract presen-tation. It is created and updated by another Malanmapping (link ­) [5]. An interaction consumes eventsproduced by input devices (link ®). Instruments trans-form input interactions into output actions (link ¯). Anaction is executed on the abstract presentation (link °);source data and the concrete presentation are then up-dated throughout a Malan mapping (link ±).

Data

Presentation

presentationAbstract

presentationConcrete

Action

Interaction

Instrument

User Interface

Event

Abstract

Concrete

¬

±­

°

®

¯

Figure 1. Organization of the architectural model Malai

Malai aims at improving: 1) the modularity by consid-ering presentations, instruments, interactions, and ac-tions, as reusable first-class objects; 2) the usability bybeing able to specify feedback provided to users withininstruments, to abort interactions, and to undo/redo ac-tions. Malai is well-suited for UI adaptation because ofits modularity: depending on the context, interactions,instruments, and presentations, can be easily composedto form an adapted UI. However, Malai does not pro-vide any runtime adaptation process. The next section

introduces the research work on dynamic adaptive sys-tem that has been applied to Malai for this purpose.

Dynamically Adaptive SystemsThe DiVA consortium proposes an adaptation meta-model to describe and drive the adaptation logic of Dy-namically Adaptive Systems (DAS) [14]. The core ideais to design DAS by focusing on the commonalities andvariabilities of the system instead of analyzing on allthe possible configurations of the system. The featuresof the system are refined into independent fragmentscalled aspect models. On each context change, the as-pect models well adapted to the new context are se-lected and woven together to form a new model of thesystem. This model is finally compared to the currentmodel of the system and a safe migration is computedto adapt the running system [20].

The selection of the features adapted to the currentcontext is performed by a reasoning mechanism basedon multi-objective optimization using QoS Properties.QoS properties correspond to objectives that the rea-soner must optimize. For example, the properties of thesystem described in [14] are security, CPU consumption,cost, performances, and disturbance. The importanceof each property is balanced depending on the context.For instance, if the system is running on battery, mini-mizing CPU consumption will be more important thanmaximizing performances. The developer can specifythe impact of the system’s features on each property.For example, the video surveillance feature highly max-imizes security but does not minimize CPU consuming.The reasoner analyzes these impacts to select featuresthe best suited to the current context.

If DiVA proposes an approach that tames dynamicadaptations of complex systems, it lacks at consider-ing UI adaptations. The following sections describe thecombined use of Malai and DiVA to bring adaptationsof complex interactive systems under control.

CONCEPTION PROCESSThis section describes the different steps that develop-ers have to perform during the conception of adaptableUIs. The first step consists in defining the context andthe action models. Then a mapping between these twomodels can be defined to specify which context elementsdisable actions. The last step consists in defining thepresentations and the instruments that can be selectedat runtime to compose the UI.

All these models are defined using Kermeta. Kermetais a model-oriented language that allows developers todefine both the structure and the behavior of models[21]. Kermeta is thus dedicated to the definition of ex-ecutable models.

Context DefinitionA context model is composed of the three class modelsUser, Platform, and Environment that describe each

inria

-005

9089

1, v

ersi

on 1

- 5

May

201

1

context component. Developers can thus define theirown context triples without being limited to a specificcontext metamodel.

Each class of a class model can be tagged with QoSproperties. These properties bring information aboutobjectives that demand top, medium, or low priorityduring UIs adaptation. For instance, Listing 1 definesan excerpt of the user class model for a home automa-tion system. Properties are defined as annotations onthe targeted class. This class model specifies that auser can be an elderly person (line 3) or a nurse (line6). Class ElderlyPerson is tagged with two properties.Property readability (line 1) concerns the simplicity ofreading of UIs. Its value high states that the readabilityof UIs must be strongly considered during adaptationsto elderly people. For instance, large buttons wouldbe more convenient for elderly people than small ones.Property simplicity (line 2) specifies the simplicity ofthe UI. Since elderly people usually prefer simple inter-action, this property is set to high on class ElderlyPer-son.

1 @readab i l i ty ”high ”2 @s impl i c i ty ”high ”3 class Elder lyPerson inherits User {4 }56 class Nurse inherits User {7 }

Listing 1. Context excerpt tagged with QoS properties

By default properties are set to ”low”. For examplewith Listing 1, property readability is defined on classElderlyPerson but not on class Nurse. It means thatby default Nurse has property readability set to ”low”.

All the properties of the current context should be max-imized. But adapting UIs is a multiobjective problemwhere all objectives (i.e. QoS properties) cannot bemaximized together; a compromise must be found. Forexample, a developer may prefer productivity to theaesthetic quality of UIs even if maximizing both wouldbe better. Values associated with properties aim at bal-ancing these objectives.

Our approach does not provide predefined properties.Developers add their own properties on the UI compo-nents and the context. The unique constraint for thedevelopers is to reuse in the context model propertiesdefined in UI components and vice versa. Indeed, prop-erties of the current context are gathered at runtimeto then select the most respectfully UI components to-wards these properties. The efficiency of the reasonerthus depends on the appropriate definition of propertiesby the developers.

Actions DefinitionActions are objects created by instruments. Actionsmodify the source data or parameters of instruments.The main difference between actions and tasks, such

as CTT tasks [22], is that the Malai’s action meta-model defines a life cycle composed of methods do,canDo, undo, and redo. These methods, that an actionmodel must implement, bring executability to actions.Method canDo checks if the action can be executed.Methods do, undo, and redo respectively executes, can-cels, and re-executes the action. An action is also as-sociated to a class which defines the attributes of theaction and relations with other actions.

1 abstract class NurseAction inherits Action { }23 class AddNurseVisit inherits NurseAction , Undoable{4 reference ca l endar : Calendar5 attribute date : Date6 attribute t i t l e : S t r ing7 attribute event : Event89 method canDo ( ) : Boolean i s do

10 result := ca lendar . canAddEvent ( date )11 end12 method do ( ) : Void i s do13 event := ca lendar . addEvent ( t i t l e , date )14 end15 method undo ( ) : Void i s do16 ca lendar . removeEvent ( t i t l e , date )17 end18 method redo ( ) : Void i s do19 ca lendar . addEvent ( event )20 end21 }2223 class Cal lEmergencyService inherits NurseAction{24 // . . .25 }

Listing 2. Excerpt of nurse actions

Listings 2 defines an excerpt of the home automationaction model in Kermeta. Abstract action NurseAction(line 1) defines the common part of actions that nursescan perform. Action AddNurseVisit (line 3) is a nurseaction that adds an event into the nurse calendar (seemethod do line 12). Method canDo checks if the eventcan be added to the calendar (line 9). Methods undoand redo respectively remove and re-add the event tothe calendar (lines 15 and 18). Action CallEmergency-Service in another nurse action that calls the emergencyservice (line 23).

Mapping Context Model to Action ModelActions can be disabled in certain contexts. For in-stance elderly people cannot perform actions specific tothe nurse. Thus, action models must be constrainedby context models. To do so we use Malan, a declar-ative mapping language [5]. Because it is used withinthe Malai archetecture, the Malan language has beenselected. Context-to-action models consists of a set ofMalan expressions. For instance, one of the constraintsof the home automation system states that elderly peo-ple cannot perform nurse actions. The Malan expres-sion for this constraint is:

Elder lyPerson −> ! NurseAction

where NurseAction means that all actions that inheritfrom action NurseAction are concerned by the mapping.

inria

-005

9089

1, v

ersi

on 1

- 5

May

201

1

Another constraint states that nurses can call ambu-lances only if the house has a phone line. The corre-sponding Malan expression is:

House [ ! phoneLine ] −> ! Cal lEmergencyService

where the expression between brackets (i.e., !phone-Line) is a predicate that uses attributes and relationsof the corresponding class of the context (i.e. House inthe example) to refine the constraint.

By default all the actions are enabled. Only actionstargeted by context-to-action mappings can be disabled:on each context change, mappings are re-evaluated toenable or disable their target action.

Presentation DefinitionDevelopers can define several presentations for the sameUI: several presentations can compose at runtime thesame UI to provide users with different viewpoints onthe manipulated data; defining several presentations al-lows to select at runtime the presentations the bestsuited to the current context. For instance, the cal-endar that the nurse uses to add visits can be presentedthrough two presentations: 1) a 2D-based presentationthat displays the events of the selected month or week;2) a list-based presentation that shows the events intoa list widget.

1 class Agenda {2 attribute name : St r ing3 attribute events : Event [ 0 . . ∗ ]4 attribute dates : Date [ 0 . . ∗ ]5 }6 class Event {7 attribute name : St r ing8 attribute d e s c r i p t i o n : S t r ing9 attribute p lace : S t r ing

10 reference date : Date11 attribute s t a r t : TimeSlot12 attribute end : TimeSlot13 }14 // . . .

Listing 3. Excerpt of the 2D-based abstract presentation

1 @aesthet i cQua l i ty ”high ”2 @space ” low”3 class AgendaUI {4 attribute t i t l e : S t r ing5 attribute l i n e sU I : LineHourUI [ 0 . . ∗ ]6 attribute hand le rSta r t : Handler7 attribute handlerEnd : Handler8 attribute eventsUI : EventUI [ 0 . . ∗ ]9 attribute datesUI : DateUI [ 0 . . ∗ ]

10 }11 class EventUI {12 attribute x : Real13 attribute y : Real14 attribute width : Real15 attribute he ight : Real16 }17 // . . .

Listing 4. Excerpt of the 2D-based concrete presentationand its QoS properties

Listings 3 and 4 describe parts of the 2D-based presen-tation of the nurse agenda. Its abstract presentation

defines the agenda model (see Listing 3). An Agendahas a name, contains Event and Date instances. Anevent has a name, a place, a description, a starting andan ending Timeslot instances. A time-slot specifies thehour and the minute. A date defines its day, month andyear.

The concrete presentation defines the graphical rep-resentation of the nurse agenda (see Listing 4). Thegraphical representation of agendas (class AgendaUI )contains representations of days, events, and time-slotlines (respectively classes DayUI, EventUI and Line-HourUI ). These representations have coordinates x andy. Classes DayUI and EventUI also specify their widthand height. An agenda has two handlers associated tothe selected event. These handlers are used to changethe time-slot of the selected event.

Similarly to context models, presentations can be taggedwith QoS properties. These properties provide contextreasoner with information about, for example, the easi-ness of use or the size of the presentation. For instance,the 2D-based and list-based presentations have charac-teristics well-suited for some platforms and users. List-ing 4 shows the QoS properties of the 2D-based presen-tation defined as annotations: the 2D-based presenta-tion optimizes the aesthetic quality (property aesthetic-Quality ”high”) but not space (property space ”low”).By contrast, the list-based presentation optimizes spaceto the detriment of the aesthetic quality.While properties specified on contexts define objectivesto optimize at runtime, properties on presentations de-clare characteristics used to select appropriated presen-tations depending on the current context and its objec-tives. For instance, if the current context states thatthe the aesthetic quality must be highly considered, the2D-based presentation will be selected.

Instrument DefinitionInstruments transform input interactions into outputactions. Instruments are composed of links and of aclass model. Each link maps an interaction to a result-ing action. Instrument’s class model defines attributesand relations the instrument needs. Widgets handledby instruments and that compose the UI are notablydefined into the class model of instruments.

VisitTypeSelector is an instrument operating on thenurse agenda. This instrument defines the type of visitto add to the agenda. The selection of the type of visitcan be performed using different widgets: several togglebuttons (one of each visit type) or a list can be used.While toggle buttons are simpler to use than a list (asingle click to select a button against two clicks to se-lect an item of a list), lists are usually smaller than aset of toggle buttons. The choice of using such or suchwidgets thus depends on the current context: if spaceis a prior objective, list should be privileged; otherwise,toggle buttons should be selected.

inria

-005

9089

1, v

ersi

on 1

- 5

May

201

1

SetVisitType

VisitTypeSelector

Link

Class Model

(a) Incomplete instrument

buttons0..*

SetVisitType ButtonPressed

VisitTypeSelector

ToggleButtonAspect

Link

Class Model

(b) Completed using toggle buttons

Aspect

SetVisitType ItemChanged

VisitTypeSelector

Listlist1

Link

Class Model

(c) Completed using a list

Figure 2. Instrument VisitTypeSelector

One of the contributions of our work consists of be-ing able to choose the best suited interaction for a linkat runtime: while defining instruments, developers canlet interactions undefined. Interactions and widgetsare automatically chosen and associated to instrumentsat runtime depending on the current context. For in-stance, Figure 2(a) describes the model of instrumentVisitTypeSelector as defined by developers. This modelis composed of an incomplete link that only specifiesthe produced action SetVisitType; the way this actionis performed is let undefined. The class model of thisinstrument only defines a class corresponding to the in-strument (class VisitTypeSelector). This class modelwill also be completed at runtime.Figure 2(b) corresponds to the model of Figure 2(a)completed at runtime. Toggle buttons have been chosento perform action SetVisitType. The interaction corre-sponding to the click on buttons (interaction Button-Pressed) is added to complete the link. A set of togglebuttons (class ToggleButton) is also added to the classmodel. This interaction and these widgets come froma predefined aspect encapsulating them. We defined aset of aspects for WIMP2 interactions (i.e. based onwidgets) that can automatically be used at runtime tocomplete instrument models.Figure 2(c) corresponds to another completed model.This time, a list has been chosen. Interaction Item-Changed, dedicated to the handle of lists, completes thelink. A list widget (class List) has been also added tothe class model. This widget and its interaction alsocome from a predefined aspect.

Figure 3 presents an example of the instrument Times-lotSetter completed with interactions. This instrumentchanges the time-slot of events of the nurse agenda(action SetTimeslotEvent). Figure 3(a) shows this in-strument completed with a drag-and-drop interaction(DnD) and handlers. Handlers surround the selectedevent. When users drag-and-drop one of these handlersthe time-slot of the event is modified. This interaction

2”Windows, Icons, Menus and Pointing device”

and these handlers were encapsulated into an aspectdefined by the developer.

Figure 3(b) shows another aspect defined by the de-veloper for instrument TimeslotSetter : when the cur-rent platform supports bi-manual interactions, such assmartphones or tabletops, time-slot setting can be per-formed using such interactions instead of using a DnDand handlers.

handlers0..*

SetTimeslotEvent DnD

TimeslotSetter

HandlerAspect

Link

Class Model

(a) Completed using a drag-and-drop inter-action

SetTimeslotEvent BimanualInteraction

TimeslotSetter

Aspect

Link

Class Model

(b) Completed using a bimanual interaction

Figure 3. Instrument TimeslotSetter

Such flexibility on interactions and widgets is performedusing QoS properties. Widgets and interactions aretagged with properties they maximize or minimize.Widgets are also tagged with properties correspondingto simple data type they are able to handle. For in-

inria

-005

9089

1, v

ersi

on 1

- 5

May

201

1

stance, the toggle button widget is tagged with fourproperties: property simplicity high means that togglebuttons are simple to use; property space low meansthat toggle buttons do not optimize space; propertiesenum and boolean mean that toggle buttons can be usedto manipulate enumerations and booleans. At runtime,these properties are used to find widgets appropriate tothe current context.

ADAPTATION PROCESS AT RUNTIMEThis section details the adaptation process at runtime.This process begins when the current context is mod-ified. The context reasoner analyzes the new contextto determine actions, presentations, interactions, andwidgets that will compose the adapted UI. The weaverassociates WIMP interactions and widgets to instru-ments. The UI composer adapts the UI to reflect themodifications.

Reasoning on ContextThe context reasoner is dynamically notified aboutmodifications of the context. On each change, the rea-soner follows these different steps to adapt actions, pre-sentations, instruments, interactions, and widgets, tothe new context:

foreach Context change do1

Re-evaluate mappings to enable/disable2

actionsDisable instrument’s links that use disabled3

actionsEnable instruments’s links that use enabled4

actionsDisable instrument’s links which interaction5

cannot be performed anymoreDisable instruments with no more link enabled6

Select presentations by reasoning on7

propertiesSelect interactions/widgets for instruments by8

reasoning on propertiesend9

Algorithm 1. Context reasoner process

The process of enabling and disabling actions (line 2of Algorithm 1) is performed thanks to the context-to-action mapping: if the change of context concerns amapping, this last is re-evaluated. For instance with thehome automation example, when the user switches fromthe nurse to the elderly person, mappings described inthe previous section are re-evaluated. Actions that in-herit from NurseAction are then disabled.Once actions are updated, instruments are checked: in-strument’s links that use the disabled, respectively en-abled, actions are also disabled, respectively enabled(lines 3 and 4). Links using interactions that cannot beperformed anymore are also disabled (line 5). For exam-ple, vocal-based interactions can only work on platformsproviding a microphone. Instruments with no more linkenabled are disabled (line 6).

Presentations that will compose the UI can now be se-lected (line 7). This process selects presentations byaligning their properties with those of the current con-text. In the same way, WIMP interactions and widgetsare selected for instruments (line 8) using properties.These selections can be performed by different kind ofoptimization algorithms such as genetic algorithms orTabu search. These algorithms are themselves compo-nents of the system. That allows to change the algo-rithm at runtime when needed.

We perform this reasoning on properties using the ge-netic algorithm NSGA-II [12]. Genetic algorithms areheuristics that simulate the process of evolution. Theyare used to find solutions to optimization problems. Ge-netic algorithms represent a solution of a problem as achromosome composed of a set of genes. Each genecorresponds to an object of the problem. A gene isa boolean that states if its corresponding object is se-lected. For example with our UI adaptation problem,each gene corresponds to a variable part of the UI (thenurse actions, the toggle button aspect, the list aspect,the different presentations, etc.). The principle of ge-netic algorithms is to randomly apply genetic opera-tions (e.g. mutations) on a set of chromosomes. Thebest chromosomes are then selected to perform anothergenetic operations, and so on. The selection of chromo-somes is performed using fitness functions that maxi-mize or minimize objectives. In our case, objectives areproperties defined by the developer. For instance read-ability is an objective to maximize. For each chromo-some its readability is computed using the readabilityvalue of its selected gene:

freadability(c) =∑n

i=1 propreadability(ci)xi

Where freadability(c) is the fitness function computingthe readability of the chromosome c, ci is the gene at theposition i in the chromosome c, propreadability(ci) thevalue of the property readability of the gene ci, and xi

the boolean value that defines if the gene ci is selected.For example :

freadability(001100111001011) = 23

The fitness functions are automatically defined at de-sign time from the properties used by the interactivesystem.

Chromosomes that optimize the result of fitness func-tions are selected. Constraints can be added to geneticalgorithm problems. In our case a constraint can statethat the gene corresponding to the calling emergencyservice action can be selected only if there is a line phonein the house.When genetic algorithms are stopped, they provide aset of solutions that tend to be the best ones.

inria

-005

9089

1, v

ersi

on 1

- 5

May

201

1

Weaving Aspects to Complete ModelsOnce interactions and widgets are selected, they mustbe associated with their instruments. To do so, we reusethe process proposed in the DiVA project to weave as-pects with models. An aspect must specify where itscontent (in our case the interaction and possible wid-gets and components) must be inserted: this is the roleof the pointcut. In our case pointcuts target instrumentsand more precisely an action and the main class of theinstrument. An aspect must also define its compositionprotocol that describes how to integrate the content ofthe aspect into the pointcut.

Composing and Updating the User InterfaceThe goal of the UI composer is two-fold: 1) It composesthe selected presentations and widgets at startup. 2)Once composed, the UI composer updates the UI oncontext changes if necessary. Because modifications ofthe UI must be smooth enough not to confuse the users,the UI must not be recomposed from scratch using 1).The existing UI must be updated to minimize graphicalchanges and to keep usability.

EVALUATIONOur proposal is based on two hypotheses: 1) it tamesthe combinatorial explosion of complex interactive sys-tems adaptations; 2) adaptations performed using ourproposal are well adapted to the current context. Weevaluated these two hypotheses by applying our pro-posal to EnTiMid, a middleware for home automation.Each component of the UI of EnTiMid is developed withthe Kermeta implementation of Malai. At the end ofthe conception time, executable models are compiled asOSGi components [25] to run on the top of DiVA. Theuse of OSGi permits instruments, actions, and presen-tations to be easily enabled and disabled at runtime.

The experiments described in this section have beenperformed on Linux using a laptop with a Core2Duo at3.06GHz and 4Gb of RAM. Each result presented belowis the average result of 1000 executions.

EnTiMid: a Middleware for Home AutomationEnTiMid is a middleware for home automation. It no-tably addresses two issues of the home automation do-main, by providing a sufficient level of abstraction.The first issue is about interoperability of devices. Builtby many manufacturers, devices are often not compati-ble with one another because of their communicationprotocol. EnTiMid offers a mean to abstract fromthese technical problems and consider only the prod-uct’s functionalities.The second issue is about adaptation. Changes inthe deployed peripherals or in the user’s habits implychanges in the interactive system dealing with the home.Moreover, many people with different skills will have tointeract with the interactive system, and the UI mustadapt to the user. Considering models at runtime, En-TiMid permits such dynamic adaptation.

Figure 4. Part of the EnTiMid UI that controls thelights, the heaters, and the shutters of the home

Figure 4 shows a part of the EnTiMid’s UI that man-ages home devices such as heaters, shutters, and lights.A possible adaptation is if the home does not haveany shutter, related actions will be disabled and theUI adapted to not provide the shutter tab.

Hypothesis 1: Combinatorial explosion tamingWe evaluate this hypothesis by measuring the adapta-tion time of five versions of EnTiMid, called v1 to v5.These versions have an increasingly level of complexity,respectively around 0.262, 0.786, 4.7, 42.4, and 3822millions of configurations. These different levels of com-plexity have been obtained by removing features fromversion v5. A configuration defines which componentsof the interactive system are enabled or disabled.

The adaptation time starts after a change of context andends when the UI is adapted accordingly. The adapta-tion time is composed of: the time elapsed to select theoptimal possible configuration in a limited time; thetime elapsed to reconfigure the interactive system andits UI.

0

500

1000

1500

2000

0.262(v1)

0.786(v2)

4.718(v3)

42.46(v4)

3822(v5)

Avera

ge t

ime o

n 1

00

0 a

dap

tati

ons

(ms)

Possible configurations (in millions)

Reasoning timeConfiguration time

Total time

Figure 5. Average adaptation time of EnTiMid using anincreasingly number of possible configurations

Figure 5 presents our results using the reasoner basedon the NSGA-II genetic algorithm. It shows that thereasoning time remains linear between 600 and 800ms.That because the parameters of the reasoner (e.g. thenumber of generations, the size of the population) are

inria

-005

9089

1, v

ersi

on 1

- 5

May

201

1

automatically modified in function of the complexity ofthe system to run between 500 and 1000ms. Figure5 also shows that the configuration time (i.e. whenthe system and its UI are modified) remains constantaround 200ms. That brings the full adaptation timeto around 1 second for the most complex version ofEnTiMid.

Hypothesis 2: Adaptations qualityFinding a configuration in a limited time makes senseonly if the configuration found is of good quality. Thus,we now evaluate the quality of the configurations foundby the genetic reasoner in the limited time-slots de-scribed above. We compared these configurations withthe optimal configurations. Optimal configurations areconfigurations giving the best results using the fitnessfunctions. These optimal configurations have been com-puted by an algorithm exploring all the solutions. Suchcomputations took 4.5s, 10s, 480s, and 7200s for respec-tively v1, v2, v3, and v4. We were not able to computethe optimal solutions of v5 due to time and resourceconstraints.

0

5

10

15

20

25

0.262(v1)

0.786(v2)

4.718(v3)

42.46(v4)

Best

Config

ura

tions

Found

Possible configurations (in millions)

Average number on 1000 adaptationsMinimal number on 1000 adaptationsMaximal number on 1000 adaptations

Figure 6. Comparison between the optimal solutions andsolutions found by the genetic reasoner

Figure 6 presents the number of optimal configurationsfound by the genetic reasoner with v1, v2, v3, and v4.In average the reasoner always found optimal configu-rations for every version of EnTiMid tested. However,the performance slightly decreases while the complex-ity increases. For example with v4, several adaptationsamong the 1000 performed did not find some of theoptimal configurations. This result is normal since wecannot obtain same quality results in the same limitedtime for problems whose complexity differ.

We can state that the genetic reasoner gives good re-sults for EnTiMid. But it may not be the case for lesscomplex or different interactive systems. One of the ad-vantages of our proposal is that the reasoner is also acomponent that can be selected in function of the con-text. For instance with a simple interactive system (e.g.10000 configurations), the selected reasoner should be areasoner that explores all the configuration since it willnot take more than 0.5s.

Threats to validityAn important remark on this evaluation is that in ourcurrent implementation the configuration quality doesnot include the evaluation of the usability of adapta-tions, nor the user’s satisfaction. For example our pro-cess may perform two following adaptations provokingbig changes in the UI, that may disturb the user. Suchevaluations can be performed by:

• The reasoner while selecting a configuration. In thiscase, the previous UI will be integrated into the ge-netic algorithm under the form of fitness functionsmaximizing the consistency of the adapted UI.

• A configuration checker that would evaluate the bestconfiguration among the best ones found by the rea-soner.

The configurations found by the genetic reasoner mainlydepend on the properties defined on the components ofthe interactive systems. The developers have to balancethem through simulations to obtain good results [14].

This paper does not focus on the UI composition. TheUI composer used in this evaluation is basic and takesa negligible amount of time during the reconfiguration.The use of a more complex composer will slow down theconfiguration process.

RELATED WORKThe conception of dynamic adaptable systems has beenwidely tackled in the software engineering domain [20].Software engineering approaches use model-driven en-gineering (MDE) to describe the system as a set ofmodels. These models are sustained at runtime to re-flect the underlying system and to perform adaptations.This process thus bridges the gap between design timeand runtime. Yet these approaches do not focus onthe adaptation of UIs. For example in [9], Cetina etal. propose an approach to autonomic computing, andthus to dynamic adaptation, applied on home automa-tion. This approach lacks at considering the system asan interactive system whose UI needs adaptations.

Based on MDE, UI adaptation has been firstly tack-led during design time to face the increasing number ofplatforms (e.g., Dygmes [11], TERESA [19] and Florinset al.. [15]). These adaptation approaches mainly fol-low the CAMELEON top-down process composed of 1)the task model 2) the abstract UI 3) the concrete UI4) the final UI [8]. Using the CAMELEON process, de-velopers define several concrete UIs using one abstractUI to support different platforms. Users and the en-vironment have been also considered as adaptation pa-rameters, such as in UsiXML [18] and Contextual Con-curTaskTrees [3]. A need to adapt at runtime UIs thusappear to face to any change of user, environment andplatform.

Approaches have been proposed to consider models ofUIs at runtime [2, 24, 6]. In [7, 6], Blumendorf et al.

inria

-005

9089

1, v

ersi

on 1

- 5

May

201

1

propose a framework for the development and execu-tion of UIs for smart environments. Their proposalshares several points with ours: the use of a mappingmetamodel to map models; they consider that bridgingdesign time and runtime implies that models are exe-cutable. However, they focus on the link between themodels and the underlying system while we focus onthe adaptation of complex interactive systems.

In [24], Sottet et al. propose an approach to dynam-ically adapt plastic UI. To do so, a graph of modelsthat describe the UI is sustained and updated at run-time. The adaptation is based on model transforma-tions: in function of the context change, the appro-priate transformation is identified and then applied toadapt the UI. This process follows the event-condition-action paradigm where the event is the context changeand the action the corresponding transformation. Themain drawbacks of this approach are that: transforma-tions must be maintained when the interactive systemevolves; the development of complex interactive systemswill lead to the combinatorial explosion of the numberof needed transformations.

CAMELEON-RT is a conceptual architecture referencemodel [2]. It allows the distribution, migration, anddynamic adaptation of interactive systems. Adapta-tions are performed using rules predefined by developersand users, or learned by the evolution engine at run-time. A graph of situations is used to perform adapta-tions: when the context changes, the corresponding sit-uation is searched into the graph. The found situationis then provided to the evolution engine that performsthe adaptation. This approach focuses on the usabilityof adaptations. However, it can hardly deal with com-plex systems because of the need to define a graph ofsituations.

ReWiRe is a framework dedicated to the dynamic adap-tation of interactive systems [26]. As in our approach,ReWiRe’s architecture uses a component-based systemthat facilitates the (de-)activation of the system’s com-ponents. But ReWiRe suffers from the same main lim-itation than CAMELEON-RT: it can hardly deal withcomplex systems because of the increasing complexityof the ontology describing the whole runtime environ-ment.

In [13], Demeure et al. propose a software architecturecalled COMETs. A COMET is a task-based interac-tor that encapsulates different presentations. It alsoembeds a reasoner engine that selects the presentationthe more adapted to the current context. While wedefine a unique reasoner for the entire interactive sys-tem, COMETs defines one reasoner for each widget. Wethink that tagging widgets with properties that a globalreasoner analyzes is a process that requires less effortthan defining several reasoners.The approach presented in [17] is close to COMETswhere UI components can embed several presentations

and an inference engine deducing from the context thepresentation to use.

In [23], Schwartze et al. propose an approach to adaptthe layout of UIs at runtime. They show that the UIcomposer must also be context-aware to layout UIs infunction of the current user and its environment. Forexample, our reasoner decides the components that willcompose the UI, but not their disposition in the adaptedUI. It is the job of the UI composer that analyzes thecontext to adapt the layout of the UI accordingly.

DYNAMO-AID is a framework dedicated to the devel-opment of context-aware UIs adaptable at runtime [10].In this framework, a forest of tasks is generated from themain task model and its attached abstract description.Each task tree of this forest corresponds to the taskspossible for each possible context. Because of the com-binatorial explosion, such process can be hardly scalableto complex interactive systems.

In [16], Gajos and Weld propose an approach, calledSupple, that treat the generation of UIs as an optimiza-tion problem. Given a specific user and device, Supplecomputes the best UI to generate by minimizing theuser effort and respecting constraints. This approachis close to our reasoning step. However, Supple is notMDE-driven and only consider user effort as objectivewhile our approach allows developers to define their ownobjectives.

CONCLUSIONAdapting complex interactive systems at runtime is akey issue. The software engineering community has pro-posed approaches to dynamically adapt complex sys-tems. However, they lack at considering the adaptationof the interactive part of systems. In this paper, wehave described an approach based on the Malai archi-tectural model and that combines aspect-oriented mod-eling with property-based reasoning. The encapsulationof variable parts of interactive systems into aspects per-mits the dynamic adaption of user interfaces. TaggingUI components and context models with QoS proper-ties allows the reasoner to select the aspects the bestsuited to the current context. We applied the approachto a complex interactive system to evaluate: the timespent adapting UIs on context changes; the quality ofthe resulting adapted UIs.

Future work will focus on the consideration of adap-tations quality during the reasoning process. It willassure consistency between two adapted UIs. Work oncontext-aware composition of UIs will be carried out aswell.

ACKNOWLEDGMENTSThe research leading to these results has received fund-ing from the European Communitys Seventh Frame-work Program FP7 under grant agreements 215412

inria

-005

9089

1, v

ersi

on 1

- 5

May

201

1

(http://www.ict-diva.eu/) and 215483 (http://www.s-cube-network.eu/).

REFERENCES1. International Workshop on Aspect-Oriented

Modeling. http://www.aspect-modeling.org.

2. L. Balme, A. Demeure, N. Barralon, J. Coutaz,and G. Calvary. CAMELEON-RT: A softwarearchitecture reference model for distributed,migratable, and plastic user interfaces. In EUSAI,pages 291–302, 2004.

3. J. V. d. Bergh and K. Coninx. Contextualconcurtasktrees: Integrating dynamic contexts intask based design. In Proc. of PERCOMW ’04,page 13, 2004.

4. A. Blouin and O. Beaudoux. Improvingmodularity and usability of interactive systemswith Malai. In Proc. of EICS’10, 2010.

5. A. Blouin, O. Beaudoux, and S. Loiseau. Malan:A mapping language for the data manipulation. InProc. of DocEng ’08, pages 66–75, 2008.

6. M. Blumendorf, G. Lehmann, and S. Albayrak.Bridging models and systems at runtime to buildadaptive user interfaces. In Proc. of EICS’10, 2010.

7. M. Blumendorf, G. Lehmann, S. Feuerstack, andS. Albayrak. Executable models forhuman-computer interaction. In Proc. ofDSV-IS’08, 2008.

8. G. Calvary, J. Coutaz, D. Thevenin, Q. Limbourg,L. Bouillon, and J. Vanderdonckt. A unifyingreference framework for multi-target userinterfaces. Interacting With Computers,15(3):289–308, 2003.

9. C. Cetina, P. Giner, J. Fons, and V. Pelechano.Autonomic computing through reuse of variabilitymodels at runtime: The case of smart homes.Computer, 42:37–43, 2009.

10. T. Clerckx, K. Luyten, and K. Coninx.DynaMo-AID: A design process and a runtimearchitecture for dynamic model-based userinterface development. In Proc. of EIS’04, 2004.

11. K. Coninx, K. Luyten, C. Vandervelpen, J. V. denBergh, and B. Creemers. Dygimes: Dynamicallygenerating interfaces for mobile computing devicesand embedded systems. In Proc. of MobileHCI’03,pages 256–270, 2003.

12. K. Deb, A. Pratap, S. Agarwal, and T. Meyarivan.A fast and elitist multiobjective genetic algorithm:Nsga-ii. IEEE Transactions on EvolutionaryComputation, 6:182–197, 2002.

13. A. Demeure, G. Calvary, and K. Coninx.COMET(s), a software architecture style and aninteractors toolkit for plastic user interfaces. InProc. of DSV-IS’08, pages 225–237, 2008.

14. F. Fleurey and A. Solberg. A domain specificmodeling language supporting specification,simulation and execution of dynamic adaptivesystems. In Proc. of MODELS’09, 2009.

15. M. Florins and J. Vanderdonckt. Gracefuldegradation of user interfaces as a design methodfor multiplatform systems. In Proc. of IUI ’04,pages 140–147, 2004.

16. K. Gajos and D. S. Weld. Supple: automaticallygenerating user interfaces. In Proc. of IUI ’04,pages 93–100, 2004.

17. A. Hariri, D. Tabary, S. Lepreux, and C. Kolski.Context aware business adaptation toward userinterface adaptation. Communications of SIWN,3:46–52, 2008.

18. Q. Limbourg, J. Vanderdonckt, B. Michotte,L. Bouillon, M. Florins, and D. Trevisan. UsiXML:a user interface description language for specifyingmultimodal user interfaces. In Proc of WMI’2004,2004.

19. G. Mori, F. Paterno, and C. Santoro. Design anddevelopment of multidevice user interfaces throughmultiple logical descriptions. IEEE Transactionson Software Engineering, 30:507–520, 2004.

20. B. Morin, O. Barais, G. Nain, and J.-M. Jezequel.Taming Dynamically Adaptive Systems withModels and Aspects. In Proc. of ICSE’09, 2009.

21. P.-A. Muller, F. Fleurey, and J.-M. Jezequel.Weaving executability into object-orientedmeta-languages. In Proceedings ofMODELS/UML’2005, pages 264–278, 2005.

22. F. Paterno, C. Mancini, and S. Meniconi.ConcurTaskTrees: A diagrammatic notation forspecifying task models. In Proc. of INTERACT’97, pages 362–369, 1997.

23. V. Schwartze, S. Feuerstack, and S. Albayrak.Behavior-sensitive user interfaces for smartenvironments. In Proc of ICDHM ’09, pages305–314, 2009.

24. J.-S. Sottet, V. Ganneau, G. Calvary, J. Coutaz,J.-M. Favre, and R. Demumieux. Model-drivenadaptation for plastic user interfaces. In Proc. OfINTERACT 2007, pages 397–410, 2007.

25. The OSGi Alliance. OSGi service platform corespecification, 2007.http://www.osgi.org/Specifications/.

26. G. Vanderhulst, K. Luyten, and K. Coninx.ReWiRe: Creating interactive pervasive systemsthat cope with changing environments by rewiring.In Proc. of the 4th International Conference onIntelligent Environments, pages 1–8, 2008.

inria

-005

9089

1, v

ersi

on 1

- 5

May

201

1