A Toolkit for Managing User Attention in Peripheral Displays

A Toolkit for Managing User Attention in Peripheral Displays Tara Matthews, Anind K. Dey*, Jennifer Mankoff, Scott Carter, Tye Rattenbury

EECS Department, UC Berkeley*Intel-Berkeley Researchcurrently of HCII, Carnegie Mellon

Peripheral Display Toolkit

Problematic ScenarioBus scheduleRequires a task switch to useDoesnt afford continuous monitoring

RTE LAWR GRIZ EUCL CNTR NUM HALL SR. GRIZ SHAT 65 605a 616a 630a 65 635a 646a 700a 65 703a 716a 730a 65 735a 746a 800a 65 803a 816a 830a 65 835a 846a 900a 65 903a 916a 930a 65 935a 946a 1000a 65 1003a 1016a 1030a 65 1035a 1046a 1100a 65 1103a 1116a 1130a 65 1135a 1146a 1200p 65 1203p 1216p 1230p 65 1235p 1246p 100p 65 103p 116p 130p 65 135p 146p 200p 65 203p 216p 230p 65 235p 246p 300p 65 303p 316p 330p 65 335p 346p 400p


Solution: Peripheral DisplaysProvide awareness with min attentionSeparate from primary taskBus LED DisplayGet bus info at a glanceDynamically manages attention it attractsHard to accomplish


Why is creating PDs hard?Need to abstract info to be glance-ableNeed mechanisms for dynamically managing attention PDs attract:Deciding attention levels to attract (notification levels)Displaying info appropriately (transitions)Our contribution:Peripheral Display Toolkit (PTK) supports these key issues in PD creation


Example PTK ApplicationsRemote ActivitySocial GuitarAudio MonitorMotion MonitorRemote Awareness DisplayBus DisplaysBus MobileBus LEDInstant Messenger StatusIM Picture FrameSocial GuitarBus LED BusMobileOrb showing remote activity


Example PTK ApplicationsManage attention with abstraction, notification levels, & transitionsGlance-ableFew lines of codeSimplified code design & code re-useIM Picture FrameSocial GuitarBus LED BusMobileOrb showing remote activity


Simple Example: Motion Monitor

no/low motion

medium motion

high motion


OutlinePTK: Motivation & ExamplesManaging attentionAbstractionNotification levelsTransitionsToolkit Architecture & LibraryConclusion


Managing User AttentionAttention is a crucial issue in designSignificant impact on successMore info can be monitored w/o overburdeningAttention management is difficultUsed to designing apps that attract full attentionAttention attracted should match info urgency

Literature surveys led to key characteristics of attention management


Attention Management Characteristics From survey of peripheral displays:Abstractionenables lower-attention info consumptionNotification levelshelp determine how much attention to attract based on urgency of infoTransitionsenable display of info to match urgency


AbstractionEnables lower-attention consumption of infoExtracting features or reducing fidelity of infoMotion Monitor: replaces video images with general motion amount (high, medium, none)

video abstraction amount ofimages motion high motion medium motion no/low motion


Notification LevelsIndicate the urgency of infoLow levels = non-urgent infoHigh levels = more urgent info Motion Monitor: doesnt show urgent info, so uses low notification levels


More on Notification LevelsAttention is (roughly) scalar PD designers must decide where on the scale displayed info should fall


More on Notification LevelsAttention Type1. None


More on Notification LevelsAttention Type1. NoneNotification LevelIgnore


More on Notification LevelsAttention Type1. None2. InattentionNotification LevelIgnore


More on Notification LevelsAttention Type1. None2. InattentionNotification LevelIgnoreChange blind


More on Notification LevelsAttention Type1. None2. Inattention3. DividedNotification LevelIgnoreChange blind


More on Notification LevelsAttention Type1. None2. Inattention3. DividedNotification LevelIgnoreChange blindMake aware


More on Notification LevelsAttention Type1. None2. Inattention3. Divided4. FocusedNotification LevelIgnoreChange blindMake aware


More on Notification LevelsNotification LevelIgnoreChange blindMake awareInterruptAttention Type1. None2. Inattention3. Divided4. Focused


More on Notification LevelsNotification LevelIgnoreChange blindMake awareInterruptDemand actionAttention Type1. None2. Inattention3. Divided4. Focused5. Focused + interaction


TransitionsEnable display of info to match urgencyDepend on notification levelIgnore: no changesChange blind: fading, slow motion, tiny updatesMake aware: discrete, abrupt updatesInterrupt: flashing, beeping, vibrationDemand action: interrupt until user does actionMotion Monitor: varies pulse speeds for different notification levels


OutlinePTK: Motivation & ExamplesManaging attentionAbstractionNotification levelsTransitionsToolkit Architecture & LibraryConclusion


PTK ArchitectureSupport for managing impact on human attention using abstraction, notification levels, and transitions Simplified code design and code re-use Library of common PD componentsOutput TransitionInputNotification MapAbstractor


Motion Monitor Architecture


Motion Monitor Architecturemotion = medium


Motion Monitor Architecturemotion = mediumnotification level = change blind


Motion Monitor Architecture5 lines of logic code


Remote Awareness DisplayCount # people, How much motion?Phone ring? Voices?


Motion Monitor Remote Awareness Display


Remote Awareness Display54 total lines of code


General Architecture Diagram


Library ComponentsInputaudio, camera, Phidgets, Context Toolkit, online calendars, news, stocks, weather, Web page parser, serial port communication


Library ComponentsInputaudio, camera, Phidgets, Context Toolkit, online calendars, news, stocks, weather, Web page parser, serial port communicationOutputticker text, Ambient Orb, Phidgets


Library ComponentsInputaudio, camera, Phidgets, Context Toolkit, online calendars, news, stocks, weather, Web page parser, serial port communicationOutputticker text, Ambient Orb, PhidgetsAbstractorsmotion, people counting, voices, phone ringing


Library ComponentsInputaudio, camera, Phidgets, Context Toolkit, online calendars, news, stocks, weather, Web page parser, serial port communicationOutputticker text, Ambient Orb, PhidgetsAbstractorsmotion, people counting, voices, phone ringingNotificationexact match, threshold, contains, degree of change


TransitionsPTK provides architectural support for implementing as animationsTimingNumber of stepsFlashing (single, repeated)Tied to outputDepend on output modalityExample transitions: color changing and pulsing (orb); fading, motion, & flashing (ticker text)


Future Work & ConclusionPTK supports the creation of PDsFocuses on managing user attentionTo do this, uses abstraction, notification levels, transitionsValidated with 7 example applicationsSupports attention managementFacilitates code re-useFuture workIncorporate interruptibility and local context sensingCross-modality animationsPD evaluation support


Questions?

For more information & downloads:www.cs.berkeley.edu/projects/io/ptk/

[email protected]

Thanks toNSF, Ed De Guzman, Jeff Heer, Scott Klemmer, Berkeley & CMU


What about user context?PDs must 2 sets of attention issues:User contextinterruptibility, primary activity, focus of attentionAttention managementbalancing info urgency and user attentionPTK focuses on attention managementUser context invasive to sense, not practical to assume PDs have accessManagement issues are independent of user context Context input can be considered on a library level (in notification maps)


Related WorkAROMA: architectural support for abstractionContext Toolkit: infrastructure for context input and some abstractionPhidgets: library of hardware widgets for input and outputReal World Interfaces Toolkit: library of X10 output widgetsiStuff: infrastructure for distributed inputs and outputs


Focus AttentionPeripheral display + attracting focused attention = contradiction? Nope!Notifications enable displays to be peripheral when not displaying urgent infoAttention is scalar: PDs should cover entire range


PTK ValidationCreated 7 example applications that illustrate:Supports attention management3 used abstraction5 used notification levels and transitionsEasier to create5-55 lines of logic codeFacilitates code re-useused same input for 4 displays

IM Picture FrameSocial GuitarBus LED BusMobileOrb showing remote activity


Notification MapsIf event value equal to:Set notification level to: no motion ignore med motion change blind high motion make aware (null) interrupt (null) demand actionExact Match Notification Map


Notification MapsExact matchchecks event for equality w/specified valuesThresholddetermines if event within specified rangesContainschecks if event contains specified stringsDegree of changedetermines if event has changed by a specified amount since last event


IntroductionPeripheral displays (PDs) enable you to monitor many info sources while maintaining a calm environmentBut, only calm if designed to manage the attention they attractPDs are hard to createNo existing tools or guidelines for managing human attentionPeripheral Display Toolkit (PTK):Enables easier creation of PDs that manage human attention


Why Peripheral Displays?Provide awareness w/ minimal attentionSeparate from users primary activityNon-focal display not used unless providing peripheral infoImportant to ubicomp vision of many devices to one user


Peripheral Display Toolkit (PTK)Supports the creation of PDs1. Focuses on managing user attention2. Simplifies code3. Provides library of common componentsManagers user attention withAbstractionNotification levelsTransitions


Characteristics of Attention ManagementAbstraction of info to be glance-ableDynamically managed attentionNotification levels for deciding how much attention to attractTransitions for displaying info appropriately


Peripheral Display Toolkit (PTK)Supports the creation of PDsFocuses on managing user attentionBalance info urgency and user attentionTo do this, usesAbstractionNotification levelsTransitions


More on Notification Levels


x = attention

100%

0%

y = awareness

indirect cognitive

direct cognitive

influence

divided

attention

focused

attention

pre

-

attention

setting (layout, color, objects, names)

inattention

conscious perception (x > 0)

change blind

ambient

alerting

influence

Provide awareness with the minimal attention appropriate to the informationThese are hard to accomplish.In interviewsHow do you make a display glanceable and dynamically manage attention?These key issues were identified thru a literature survey of existing displays

Why is the bus mobile hard to create?- to make the info readable at a glance, you have to plan for the amount of attention it should attract- there are no existing tools or guidelines to help you with this management of user attentionYellow border around picture

The PTK enables the creation of a whole range of displays that manage the attention they attract from users. For example, we can represent remote activity on an Ambient Orb, IM status on a physical picture frame, remote audio on a physical guitar, bus schedules with LEDs or moving, physical tokens.glanceabledynamically manage attention attracted using abstraction, notification levels, and transitionscan be created in only a few lines of code because of the PTKs extensive library of input, output, and attention management componentsSimplified display design and code re-use: several apps re-use audio and video input (orbs, guitar, ticker), two use the same bus input Library of common PD componentsaudiophidgetsIMphidgetsbusphidgets (LEDs)busmotors (non-phidget)audioorbcameraorbaudio+cameraorb+ticker

Yellow border around picture







This example makes it possible to easily illustrate key points in the rest of the talk. Its simple, but useful.

Imagine you have colleagues in another office and you would like to have an idea of their activity. The Motion Monitor is meant to help you maintain a sense of connectedness with remote colleagues by giving you a sense of how much motion is occurring in their office.

Motion is sensed in the remote office and displayed on a commercial Ambient Orb

When there is no or very low motion in their office, the orb pulses white and light blue-ish.Where there is a moderate amount of motion in their office, the orb pulses blue a little quicker.When there is a high level of motion in their office, the orb pulses red even more quickly.

The PTK was used the create this application in only 6 lines of code.explain the topics and why presented in this ordermanaging attention: based on cog sci and peripheral display literature survey, we developed a framework for creating displays that dynamically manage user attentionthen Ill talk about how we instantiated this in our toolkit architectureconclude with validation and future workIn interviews of 10 PD creators, we found that difficulty prototyping these displays has led to fewer iterations in practice.

Finding this balance also allows the user to monitor more info at once without being overburdened. In a ubicomp environment, this is important in making good use of multiple devices while maintaining a calm environment

Roughly, attention is scalar, ranging from no attention to focal attention: pd designers have to decide where in this range displayed info should fall.

In the PTK, we focus on supporting managing user attention in the design of peripheral displays.

Stock exampleAlthough it is more complicated, attention is roughly scalarThe dotted line represents a point where you are not consciously aware that you are attending to something.To facilitate this, weve defined our notification levels to correspond to different attention types along this scale.change blind slow or splash (were talking about slow, not splash)Mechanism for creating effects on a display that attract an appropriate amount of attention from the user, based on infos urgency or notification level.

Change blind updates dont attract attention, but the info is up to date when the user glances at the displaynow Ill talk about how we instantiated this in our toolkit architectureconclude with validation and future workcode? display?This is the architecture of the MM. It is distributed across multiple machines: the camera input is in the remote location you are monitoring. The MM is on a different machine where you are. The PTK Discovery Server enables the input and output to be connected.

Composing the MM are the motion abstractor, an exact match notification map, and an orb output widget that includes transitions.pulse here, if it works automaticallyThe only code you have to write is represented by the green box; it puts the components of the app together. This is 5 lines of code.Lets look at a more complex display to explore more features of the PTK.

The Remote Awareness Display builds upon the MM and Audio Monitor to provide a more complete sense of the activity in the remote location. This application combines an Ambient Orb and a textual ticker, so a user can check the activity level from anywhere in the room by glancing at the Ambient Orb. If the user wants more detailed information, she can look at a ticker.

Camera input is abstracted for motion and for the # of people. Audio input is abstracted to determine if a phone is ringing and if people are talking.

On the output side, the orb behaves the same as the MM, but it pulses yellow when a phone rings and purple when there are voices. The ticker shows the # of people, the amount of motion, if a phone is ringing, and if people are talking. We are going to walk through the transformation from the MM to the RAD. Here is the MM.

The PTK enables abstraction to occur on the input-side. First well move the motion abstractor.This is useful when you do not want to send large video files across machines, or if you wanted to protect privacy by keeping the video on one machine.Next we add the people counting abstractor.Then we add audio input and the phone and voice abstractors.

The PTK allows multiple inputs. The PTK Discovery Server enables them to be re-used in multiple applications. Right now these inputs are being sent to the RAD, but they could be sent to other applications as well.

Now we add notification maps to deal with the different types of input.

The PTK enables abstractors and notification maps to be chained as shown, so that the output of one is the input to the next. This allows more complex abstractions and notification setting.Next we wrap the orb widget in order to customize the colors it displays for different inputs.10 put app together 7 initialize GUI ticker 8 orb wrapper 29 ticker wrapper

Finally, we add the ticker widget and wrapper.

The PTK allows an application to have as many outputs as it needs.

This app requires 54 total lines of code, most of which are customizing the message displayed on the ticker.Here is a general architecture diagram. Again, you can see multiple inputs feeding into the Server which distributes their events to multiple applications, allowing for re-use of input.Abstraction can occur on the input or output side, or bothWithin applications, abstractors and notification maps can be chained allowing more complex abstractions and notification setting; andmultiple output widgets can be included.The PTK has abstractors for motion and people counting in video and voices and phone rings in audio.The PTK includes several parameterizable notification maps that were sufficient to build all of the displays weve built so far.

The library is necessarily limited so we have focused on making it very easy to create new library components. The modular nature of these components makes it very easy to put them together in different ways and reuse them.PTK represents groundwork for future research in peripheral displays Be prepared for:user context (next slide)related work (slide after)lines of code for RAD (slide after)question about chaining in RAD:in this case, it doesnt really chain the abstractors; we enable flexibility: you can chose to not replace the raw input in the event and instead add to the abstracted data to the event which is what we did for the RAD both motion and people counting abstractors used the raw input and added to the event. You can instead chose to replace the raw input with the abstracted data and the next abstractor in the chain will have to use the last abstractors output.(slide) transitions why not pluggable?tied to output because they heavily depend on how the information is displayedwe provide architectural support for implementing transitions as animationscontrol over timingnumber of stepsbased on previous work in toolkit level animation support

Peripheral displays must deal with 2 sets of attention issues:user context (like interruptibility and reasoning about a users primary activity), andattention management (or balancing info urgency and user attention)

While we are interested in solving both parts of the problems, weve tackled one half, dealing with attention management first. The reasons for our decision are thatUser context is invasive to sense, and it is not practical to assume PDs will have access to it; and also thatManagement issues are independent of user context

With the PTK, developers could incorporate results from an existing interrituptibility sensing tool into the decisions made by a notification mapWE BELIEVE Peripheral displays should cover the entire range

(Scott: should I not say attention is scalar so often? Will someone object?)To validate the PTK, we created 7 example applications, some pictured here. These illustrate that the PTK supports attention management (); the PTK makes peripheral displays easier to create (w/ the help of PTK library components, our 7 example apps required 10 to 40 lines of logic code); and the PTK facilitates code re-use ().

audiophidgetsIMphidgetsbusphidgets (LEDs)busmotors (non-phidget)audioorbcameraorbaudio+cameraorb+ticker

Notification maps determine the notification level of an event give some parameters you set. These parameters are outlined in blue here.

The MM uses an exact match notification map, which checks if the new events value is equal to these parameters.There is no value specified for interrupt and demand action, so it doesnt use those notification levels.The PTK library includes other notification maps. Threshold determinesPeripheral displays (PDs) enable you to monitor many sources of info at once while maintaining a calm environment But, a calm environment can be maintained only if they are designed to properly manage the attention they attractBecause managing human attention is currently ad hoc, PDs are hard to createThe PTK enables easier creation of Peripheral displays that manage human attention

Why should I care about peripheral displays?Provide awareness w/ minimal attention and they arent part of a users primary activityCurrently in a ubiquitous computing environment, if one user has many computers, the non-focal devices are not being used unless they provide peripheral info.In this way, peripheral displays are important to the ubicomp vision of one user making good use of many devices and to keeping that use calm and manageable.

Motion Monitor:Just like the Audio Monitor, except it uses motion to indicate how much activity is occurring in a remote place.Again, this is useful because it helps users feel connected to remote friends or colleaguesActivity is sensed by a camera and analyzed for motionSimilar to the Audio Monitor, the orb is white when there is no motion, blue when there is low motion, and red and green when there is high motion. It changes colors a little more rapidly as motion increases.The PTK supports the creation of pds by focusing on managing user attention.Managing user attention means finding a balance between info urgency and user attention (for now we are doing this independent of user context).To accomplish this, the PTK supports three issues specific to conveying information on the periphery of human attention: abstraction, notification levels, and transitions.We talked to designers, we looked at The PTK supports the creation of pds by focusing on managing user attention.Managing user attention means finding a balance between info urgency and user attention (for now we are doing this independent of user context).To accomplish this, the PTK supports three issues specific to conveying information on the periphery of human attention: abstraction, notification levels, and transitions.

A Toolkit for Managing User Attention in Peripheral Displays

Documents

peripheral display toolkit

attention levels

attention pds

successmore info

info urgencyliterature

key issues

uc berkeley

pds hard