Page 1
1 CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University.
Lecture 12: Beyond WIMP
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 2
WIMP interfaces
• WIMP = Windows, Icons, Menus, Pointers – they’re everywhere! – in fact, after circa 1983, the vast majority of
interfaces are based on the WIMP paradigm • e.g., think about Windows vs. Macintosh vs. UNIX-based
window managers
– some exceptions... • e.g., text terminals, game systems
– ... but generally, WIMPs rule!
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 3
Evolution of computers (& users)
Generation Hardwaretechnology
Operatingmode
Programminglanguages
Terminaltechnology User types Advertising
image User interface paradigm
11945-1955Pioneer
Vacuum tubes, hugemachines, muchcooling, short meantime between failures.
One user at atime "owns"machine (butfor a limitedtime only)
Machinelanguage001100111101
TTY,typewriter.Only used inthe computercenter.
Experts,pioneers
Computer ascalculator Programming
21955-1965Historical
Transistors; morereliable. Computersstart seeing useoutside the lab.
Batch("computer astemple" -makeofferings toget oraclereplies)
Assembler ADDA,B
Line-orientedterminals("glass-TTY")
Technocrats,professionalcomputerists
Computer asinformationprocessor
Command languages
31965-1980Traditional
Integrated circuits.Businesses can cost-justify buyingcomputers for manyneeds.
Timesharing(onlinetransactionprocessingsystems)
"High-level"languages,Fortran, Pascal
Full screenterminals,alphanumericcharacters only.Remote accesscommon.
Specializedgroups withoutcomputerknowledge(e.g. banktellers)
Mechanizationof white-collarlabor
Full-screen strictly hierarchicalmenus and form fill-in
41980-1995Modern
VLSI. Individuals canbuy their ownpersonal computer
Single userpersonalcomputers
Problemorientedlanguages,spreadsheets
Graphicaldisplays withfair resolution.Desktops andheavy portables.
Businessprofessionals,hobbyists
Personalproductivity(computer astool)
WIMP (Windows, Icons,Menus, and a Pointing device)
51996-?Future
Wafer-scaleintegration, computer-on-a-chip. Individualscan buy manycomputers.
Networkedsingle usersystems andembeddedsystems
Non-imperative,possiblygraphical
"Dynabook"[61],multimedia I/O,easily portable,with cellularmodem.
Everybody Computer asentertainment
Noncommand interfaces
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 4
Beyond WIMP
• WIMP has been around a while, and probably will remain on the scene for a long time.
• But let’s think forward. What’s the next step? – Nielsen: “virtual realities, head-mounted displays, sound
and speech, pen and gesture recognition, animation and multimedia, limited artificial intelligence, and highly portable computers with cellular or other wireless communication capabilities” (?!)
– increasing computing power makes this possible – but can all this be in a single interface?
maybe not... that’s yet another difference! • interfaces may become more task/user specific
Page 2
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 5
Functional vs. Object-Oriented
• “Old-school” interfaces are functional in requiring specification of entire function – uses a “verb-noun” syntax – e.g., “rm foo”, “emacs file.java”
• Current GUIs are object-oriented – uses a “noun-verb” syntax – e.g., select icon, drag to trash / select “Open”
• What seems to be coming on the horizon? – “syntax-free” interfaces – well, at least syntax-flexible
• can specify noun-verb, or verb-noun, or something radically different... just like communication w/ people
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 6
One step forward...
• Thought exercise – imagine you have access to a movie database
• database = info about films, actor/actresses, etc.
– how can you answer questions like... • In what films did Harrison Ford star between 1980-90? • In what films did Julia Roberts and Richard Gere costar? • What are the most popular Sci Fi movies of 1993?
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 7
One step forward...
• Ahlberg & Shneiderman’s FilmFinder
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 8
Non-command interfaces
• Idea: user and computer interact not through a dialogue of commands and responses, but through a complex, dynamic, continuous interaction
• Huh? What’s that? • Nielsen’s 12 “interaction characteristics” for
next-generation, non-command software – not all applications will incorporate all 12 – rather, it is expected that many applications will
incorporate a significant subset of the 12
Page 3
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 9
1. User focus
• Interaction feels like “using a computer”, not working on a task (according to Nielsen)
• With NC interfaces, focus = task... features come for free implicitly
• Example: Portholes system – update
every 5min – implicit
awareness
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 10
2. Computer’s role
• Old: “Do what you’re told” New: “Do what I mean!”
• Great idea... but why is this hard? • Example: Intelligent tutoring
– monitor what the student knows – interrupt with instruction when necessary
• Example: Model tracing / “Mind tracking” – infer student knowledge, or disabled user’s
intentions, or driver’s intentions...
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 11
3. Interface control
• Old: User controls computer New: Computer controls interaction
• Examples: warn user of incoming email, infer current writing task and provide template, etc.
• BUT this is very hard to do well – must avoid interrupting the user – guesses / inferences had better be right!
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 12
4. Syntax
• Old: Rigid interaction “syntax” New: No / little syntax
• Example: deleting files – way #1: select and delete (noun-verb) – way #2: say “remove all *.java files” (verb-noun) – can we integrate multiple methods?
• Example: writing math expressions – try not to require top-down or bottom-up
Page 4
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 13
5. Object visibility
• Old-school interfaces with “direct manipulation” require visible objects
• New interfaces could manipulate objects implicitly through higher-level interactions, or with hidden agents
• Might this be dangerous? – user doesn’t know about manipulation – can be good, can be bad
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 14
6. Interaction stream
• Old: Single-threaded input / output New: Multi-threaded, multimodal
• Example: “Put that there!” – point to display wall at object – say “Put that”... point to destination... “there”
• Example: eye-driven window interface File Edit Special
Zoo
FishBirdsdovehawk
Zoo
TrashDone
Birds
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 15
7. Bandwidth
• Old: Low input bandwidth (keys, mouse) New: Very high bandwidth
• Systems may incorporate motion tracking, virtual reality, speech, “peripheral” input
• Difficulties – requires lots of processing power, both for
accepting input and interpreting it – lags are unacceptable!
(e.g., motion sickness in virtual environments)
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 16
8. Tracking feedback
• Old: Feedback only after completed input New: Continuous feedback on-the-fly
• Example: Emacs search (sort of) • Example: movie database • Interface should react like the real world
– again, input-output lag is an issue – again, processing power is an issue
Page 5
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 17
9. Turn-taking
• Old: First user, then computer, then user... New: Continuous stream for both
• Closely related to feedback... computer is always responding, so there’s never a “turn”
• How does this map onto the real world? – when we interact with the inanimate world?
• e.g., walking through the park, playing basketball
– when we interact with the animate world? • e.g., talking to someone, interacting with a pet
– we sometimes take turns in the real world... why not in a user interface?
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 18
10. Interface locus
• Old: Computer on the desk New: Computers everywhere
• Ubiquitous computing from... – smaller, lighter “computers”
• e.g., PDAs, calculators, watches
– computers built into everyday objects • e.g., ovens, cars, shopping carts
– computers built into not-so-everyday objects • e.g., pet dog robots
• Good for awareness, “telepresence”; dangerous for privacy?
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 19
11. User programming
• That is, programming for end users • Old: (Usually) hard-core macro languages
New: Smooth adaptation of objects • Example: object-oriented customization
– “take your basic” <object> “but make the” <subobject> “behave like this...”
– if possible, can specify with state transitions • like storyboards, or like SILK’s behaviors
• BUT in the end, complex programs require complex languages – hard (for me) to envision huge successes here
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 20
12. Software packaging
• Old: Application-centered approach New: System-wide, OO approach
• Example: spell checkers – should be only one for your entire system – does Microsoft have this right??
• integrated across Word, Excel, PowerPoint, email • easier to do for a single vendor, and has the
unfortunate side effect of monopolization • open source, open standards — but can we really
arrive at a true standard?
Page 6
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 21
Sample noncommand interface domains
• Interface agents – provide active help, reminders, etc.
• Embedded help – actually show the process, guiding the user’s
“hand” to the right places
• Computer music – computer listens, plays along, harmonizes, etc.
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 22
• Gaze-based interfaces (GBIs) – users control the interface using
gaze / eye movements – typical focus on disabled users – gaze is often the only (primary) input
– keyboard, mouse
– speech, handwriting, gestures
– poking (CRL Kiosk), tickling (Tickle-Me Elmo) !!
HC“Eye”
• Human --> machine communication...
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 23
Gaze-Added Interfaces (GAIs)
• Users control the interface using gaze and/or other inputs
• Gaze added to basic/existing input instead of replacing it
• Users can… – employ only basic inputs – employ only gaze input – employ any combination of basic / gaze
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 24
File Edit Special
Zoo
Fish Birds dove hawk
Zoo
Trash Done
Birds
Case Study: IGO
Page 7
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 25
• Gaze focus: background highlight
• Gaze button: (keyboard) key • Gaze control analogous to mouse
– click for select – double-click for open – hold for drag
hawk
Gaze-Added Input
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 26
• All GBIs must interpret gaze – i.e., assign gaze to intended object
• Standard interpretation – assign gaze to underneath / nearest object
• Intelligent interpretation – assign gaze to most likely object – use probabilistic model of behavior
File Edit Help
Intelligent Interpretation
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 27
File Edit Help
Intelligent Interpretation
• Find object that maximizes Pr (gaze|object) • Pr (object|history)
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 28
Edit
Copy
Paste
Edit
Copy
Paste
Intelligent Interpretation
Page 8
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 29
Evaluation Study
• Training Stage (8 blocks of 10 trials) – gaze only or mouse only
• Free Stage (2 blocks of 10 trials) – gaze and/or mouse as desired
• Trial = one of five tasks (5-10 s) – select icon, close window, open window
(double-click), select menu (drag), move icon (drag)
• Ten users with no GBI experience
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 30
0123456789
10
Train1
Train2
Train3
Train4
Free1
Free2
B l o c k
GazeMouseFree
Task Times
• No difference between gaze and mouse
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 31
0102030405060708090
100
Train1
Train2
Train3
Train4
Free1
Free2
B l o c k
GazeMouseFree
Task Errors
• More errors with gaze than mouse • Errors in free stage –> gaze use
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 32
0102030405060708090
100
1 2 3 4 5 6 7 8 9 10U s e r
Free Stage Gaze Use
• Overall gaze use = 67% • Correlated to (gaze–mouse) times (R=.70)
Page 9
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 33
0102030405060708090
100
Select Close Open Menu MoveAct i on
Gaze Use by Action Type
• Less complex actions –> more gaze use
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 34
Intelligent Interpretation
• Comparison of correctness with... – intelligent interpretation – “no-context” interpretation:
assign gaze to nearest object – “basic” interpretation:
assign gaze to underneath object
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 35
Lessons Learned
• Gaze nicely complements other inputs – users quickly adapt to gaze input – users successfully interleave gaze, mouse
• Common difficulties – “leave before click” – gaze dragging – handling two “cursors”
• Intelligent interpretation helps – eye trackers will improve,
but variability will remain – better eye tracking –> greater usability
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 36
Thought question…
• Ok, let’s try to put all this to use. • We know how a typical web browser looks • Can we design a new one “beyond WIMP”?
Page 10
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 37
On Beyond GUI: Wearable, Heterogeneous, Adaptive
Steven Feiner Computer Graphics & User Interfaces Lab Department of Computer Science Columbia University New York, NY 10027
Supported in part by AFRL, NSF, ONR, and gifts from Alias Systems, IBM, Mitsubishi Electric Research Labs, and Microsoft
CHI 2006 Workshop: What is the Next Generation of Human-Computer Interaction? Montreal, Canada April 23, 2006
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 38
• Cursorless • Eyes-free
Wearable
Corners of watch bezel serve as
“tactile landmarks” to
guide user’s finger
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 39
• Cursorless • Eyes-free
Wearable
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 40
• Cursorless • Eyes-free
Wearable
Page 11
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 41
Heterogeneous
• Coordinated use of different displays and interaction devices
– Multiple displays • Projected desktop • Handheld • High-res monitor • See-through head-
worn – Multiple interaction
devices • Multi-touch table • Tracked hands/
heads • Speech
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 42
Heterogeneous
• Coordinated use of dimensionalities – “Pull” 2D image
into 3D model – “Push” 3D model
into 2D image
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 43
Adaptive
• Change user interface in response to environment – Modify
layout of virtual objects to avoid overlap in AR
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 44
Adaptive
• Change user interface in response to environment – Modify
transparency to reveal important content on desktop
Page 12
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 45
Three Themes for Future UIs
• Wearable – Cursorless Eyes free,
glanceable,…
• Heterogeneous – Mix of users, displays,
devices, dimensionalities,…
• Adaptive – Dynamically modified in
response to changes in environment, tasks,…