AUGMENTED LARGE SCALE COGNITION Stuart Card Palo Alto Research Center (PARC) (Visiting Professor, Stanford) Stanford Computer Forum Stanford University April 15, 2009
AUGMENTED LARGE SCALE COGNITION
Stuart Card Palo Alto Research Center (PARC) (Visiting Professor, Stanford)
Stanford Computer Forum Stanford University April 15, 2009
ENGINEERING HUMAN PROSTHESES
HOW DO WE DESIGN COGNITIVE PROSTHESES?
? Meg Stewart
Paul MacCready
Motivating Problem: Large-Scale Cognition
7.3 million pages/day
Comparisons
1999 2009 (EB) (EB)
Unique Info 0.6 36.0 Store in earth’s population
memory in 1 yr 0.1 0.1 Record all words in all
lives 3.6 3.6
Source: Berkeley SIMS + Computations
Man the informavore (George Miller, 1983)
Informavores Hunger for information about
the world Use information to adapt to the
world
COGNITION
Find . . . (Perceive, Learn, Remember) Think . . . (Decide) Do . . . (Create, Act)
LEVELS OF COGNITION
Immediate behavior Experiential cognition Routine cognitive skill
LEVELS OF COGNITION
Problem Solving Reflective cognition
LEVELS OF COGNITION
Social cognition People finding, thinking, and doing
together
TIME LEVELS OF BEHAVIOR
107 (months) SOCIAL Social Behavior 106 (weeks) 105 (days) 104 (hours) RATIONAL Problem Solving 103 102 (minutes) 101 COGNITIVE Immediate Behavior 100 (seconds) 10-1 10-2 BIOLOGICAL Automatic Behavior 10-3 (msec) 10-4
IMMEDIATE BEHAVIOR
107 (months) SOCIAL Social Behavior 106 (weeks) 105 (days) 104 (hours) RATIONAL Problem Solving 103 102 (minutes) 101 COGNITIVE Immediate Behavior 100 (seconds) 10-1 10-2 BIOLOGICAL 10-3 (msec) 10-4
COGNITIVE LEVEL (Immediate behavior) Minimize mental time and motion)
MODEL HUMAN PROCESSOR
Processors and Memories applied to human
Used for routine cognitive skill
EXAMPLE: ZERO-PARAMETER CALC
Problem: Inventor claims he invented 600 wpm typewriter. License and develop?
Solution 1: Half stroke: τM = 70 ms/char Whole stroke: τM + τM = 140 ms/char
but if between hands, overlap: τM = 70 ms = 131 words/min
EXAMPLE: ZERO-PARAMETER CALC
Solution 2: (range calculation) Half stroke: τM=70 [30~100] ms/char = 131 [308~92] words/min
Conclusion: Bogus claim. Throw him
out!
TASK ANALYSIS: GOMS (GOALS, OPERATORS, METHODS, SELECTION RULES)
GOAL: EDIT-MANUSCRIPT • repeat until done GOAL: EDIT-UNIT-TASK
GOAL: ACQUIRE-UNIT-TASK • if not remembered GET-NEXT-PAGE • if at end of page GET-NEXT-TASK • if an edit task found
GOAL: EXECUTE-UNIT-TASK GOAL: LOCATE-LINE • if task not on line
[select : USE-QS-METHOD USE-LF-METHOD]
GOAL: MODIFY-TEXT [select USE-S-COMMAND
USE-M-COMMAND]
task analysis
PREDICTS TIME WITHIN ABOUT 20%
SAE RECOMMENDED PRACTICE J2365
Predict task times for car navigation systems
Check compliance with SAE J2364 (15-Second Rule)
Note: To estimate times while driving, multiply by 1.3 to 1.5.
Based on GOMS and work by Paul Green at Univ. of Michigan Transportation Research Institute.
Dario Salvucci
SAE J2365 OPERATOR TIMES Time (s)
Code Name Young (18-30
Old (55-60)
Rn Reach near 0.31 0.53 Rf Reach far 0.45 0.77 C1 Cursor once 0.80 1.36 C2 Cursor 2 times or more 0.40 0.68 L1 Letter or space 1 1.00 1.70 L2 Letter or space 2 times or more 0.50 0.85 N1 Number once 0.90 1.53 N2 Number 2 times or more 0.45 0.77 E Enter 1.20 2.04 F Function keys or shift 1.20 2.04 M Mental 1.50 2.55 S Search 2.30 3.91 Rs Response time of system-scroll 0.00 0.00 Rm Response time of system-new menu 0.50 0.50 Paul Green UMITRI
COGNITIVE LEVEL (Immediate behavior) Minimize mental time and motion (Efficient communication) Input on output
INTERACTIVE COMPUTING
typewriter I/O Graphical CRT
Whirlwind (MIT)
DIRECT MANIPULATION
Sketchpad (Sutherland, 1963)
Input on Output
J. C. R. LICKLIDER
(Cognitive Prosthetic) HUMAN-MACHINE SYMBIOSIS: “The hope is that in not too many years, human brains and computing machines will be coupled together very tightly, and that the resulting partnership will think as no human brain ever thought.”
THREE THEMES FOR LARGE SCALE COGNITION
Efficient Communication Tight Coupling
Representation Shift
LOOK AT COMBINATORICS OF COGNITIVE LEVELS X THEMES
PROBLEM SOLVING
107 (months) SOCIAL Social Behavior 106 (weeks) 105 (days) 104 (hours) RATIONAL Problem Solving 103 102 (minutes) 101 COGNITIVE Immediate Behavior 100 (seconds) 10-1 10-2 BIOLOGICAL Automatic Behavior 10-3 (msec) 10-4
Meg Stewart
COGNITIVE LEVEL (Immediate behavior) Minimize mental time and motion (Efficient communication) Input on output (Representation shift) RATIONAL LEVEL (Problem solving) Maximize (information gain)/(time cost) (Efficient search)
IMMEDIATE BEHAVIOR
Routine cognitive skill Well-known path
Information Search
Problem solving Heuristic search Exponential if
don’t know what to do
PHASE TRANSITION IN NAVIGATION COSTS AS FUNCTION OF INFORMATION SCENT
Notes: Average branching factor = 10 Depth = 10
0 2 4 6 8 10 0
50
100
150
Depth
Num
ber o
f pag
es v
isite
d
.100
.125
.150
0 2 4 6 8 10 0
50
100
150
.100
.150
Probability of choosing wrong link (f)
0 0.05 0.1 0.15 0.2 0
20
40
60
80
100
f
Num
ber o
f Pag
es V
isite
d pe
r Lev
el
Linear Exponential
OPTIMALITY THEORY
Max Useful info
Time Max Energy
Time [ ] [ ]
Optimal Foraging Theory Information Foraging Theory
Information Foraging Theory:
patchWithinpatchBetweenWB TTGain
TTGR
−− +=
+=
COGNITIVE LEVEL (Immediate behavior) Minimize mental time and motion (Efficient communication) Input on output (Representation shift) RATIONAL LEVEL (Problem solving) Maximize (information gain)/(time cost) (Efficient search) Information scent (by design) (Efficient semantic
search)
WITHIN-PATCH ENRICHMENT: INFORMATION SCENT
Tokyo
San Francisco
New York
perception of value and cost of a path to a source based on proximal cues
Boosting Information Scent
IMPORTANCE FOR WEB DESIGN
Jarad Spool, UIE
COGNITIVE LEVEL (Immediate behavior) Minimize mental time and motion (Efficient communication) Input on output (Representation shift) RATIONAL LEVEL (Problem solving) Maximize (information gain)/(time cost) (Efficient search) Information scent (by design) (Efficient semantic
search) Information scent (by machine) (Semantic coupling)
RELEVANCE-ENHANCED THUMBNAILS Emphasize text that
is relevant to query Text callouts
Enlarge text that might be helpful in assessing page Enlarge headers
020406080100120140160180
Picture Homepage E-commerce Side-effects
Tota
l Sea
rch
Tim
e (s
)
Text Plain Enhanced
Allison Woodruff
MACHINE MODELING OF INFORMATION SCENT
cell
patient
dose
beam
new
medical
treatments
procedures
Information Goal Link Text
PREDICTION OF LINK CHOICE
R2 = 0.72
0
5
10
15
20
25
30
35
0 5 10 15 20 25 30 35 Observed frequency
Pre
dict
ed fr
eque
ncy
R2 = 0.90
0
10
20
30
40
50
0 10 20 30 40 50 Observed frequency
Pre
dict
ed fr
eque
ncy
(a) ParcWeb (b) Yahoo
Piroli, PARC
BLOODHOUND PROJECT
Starting Point: www.xerox.com Task: look for “high end copiers”
OUTPUT usability metrics
INPUT
Chi, et al
Smart Book Semantic Index and Scent Highlighting Aids the analyst in finding the most
relevant information quickly.
Information Scent
COGNITIVE LEVEL (Immediate behavior) Minimize mental time and motion (Efficient communication) Input on output (Representation shift) RATIONAL LEVEL (Problem solving) Maximize (information gain)/(time cost) (Efficient search) Information scent (by design) (Efficient semantic
search) Information scent (by machine) (Semantic coupling) Information visualization (Visual representation
shift)
VISUALIZATION Re-represent to see patterns (eg.,
amphidromic points).
Ray
MACRO-MICRO READING: 6M POINTS
VISUALIZATION REFERENCE MODEL
Human InteractionHuman Interaction
VisualVisualMappingsMappings
VisualVisualStructuresStructures
ViewViewTransformationsTransformations
Visual FormVisual Form
ViewsViews
Raw Data: idiosyncratic formatsData Tables: relations (cases by variables) + meta-dataVisual Structures: spatial substrates + marks + graphical propertiesViews: graphical parameters (position, scaling, clipping, …)
RawRawDataData
DataDataTransformationsTransformations
DataData
DataDataTablesTables
TaskTask
COGNITIVE LEVEL (Immediate behavior) Minimize mental time and motion (Efficient communication) Input on output (Representation shift) RATIONAL LEVEL (Problem solving) Maximize (information gain)/(time cost) (Efficient search) Information scent (by design) (Efficient semantic
search) Information scent (by machine) (Semantic coupling) Information visualization (Visual representation
shift) Input on data (Data coupling)
INTERACTIVE
Human InteractionHuman Interaction
VisualVisualMappingsMappings
VisualVisualStructuresStructures
ViewViewTransformationsTransformations
Visual FormVisual Form
ViewsViews
Raw Data: idiosyncratic formatsData Tables: relations (cases by variables) + meta-dataVisual Structures: spatial substrates + marks + graphical propertiesViews: graphical parameters (position, scaling, clipping, …)
RawRawDataData
DataDataTransformationsTransformations
DataData
DataDataTablesTables
TaskTask
Dynamic Queries Magic Lens Overview + Detail Linking & Brushing Extraction & Comparison Attribute Explorer
DYNAMIC QUERIES
Home Finder (U. Maryland)
COGNITIVE LEVEL (Immediate behavior) Minimize mental time and motion (Efficient communication) Input on output (Representation shift) RATIONAL LEVEL (Problem solving) Maximize (information gain)/(time cost) (Efficient search) Information scent (by design) (Efficient semantic
search) Information scent (by machine) (Semantic coupling) Information visualization (Visual representation
shift) Input on data (Data coupling) Attention-reactive displays (Implicit coupling)
ATTENTION REACTIVE
Human InteractionHuman Interaction
VisualVisualMappingsMappings
VisualVisualStructuresStructures
ViewViewTransformationsTransformations
Visual FormVisual Form
ViewsViews
Raw Data: idiosyncratic formatsData Tables: relations (cases by variables) + meta-dataVisual Structures: spatial substrates + marks + graphical propertiesViews: graphical parameters (position, scaling, clipping, …)
RawRawDataData
DataDataTransformationsTransformations
DataData
DataDataTablesTables
TaskTask
ATTENTION-REACTIVE
Degree-of-Interest Trees
COGNITIVE LEVEL (Immediate behavior) Minimize mental time and motion (Efficient communication) Input on output (Representation shift) RATIONAL LEVEL (Problem solving) Maximize (information gain)/(time cost) (Efficient search) Information scent (by design) (Efficient semantic
search) Information scent (by machine) (Semantic coupling) Information visualization (Visual representation
shift) Input on data (Data coupling) Attention-reactive displays (Implicit coupling) Sensemaking tools (Representation shift)
SENSE MAKING TASKS Characteristics
Massive amounts of data Ill-structured task Organization,
interpretation, insight needed
Output, decision, solution required
Examples Understanding a health
problem and making a medical decision
Buying a new laptop Weather forecasting Producing an intelligence
report
IMPORTANCE OF SENSE MAKING
75% of “significant tasks” on the Web are more than simple “finding” of information (Morrison et al., 2001) Understanding a topic (e.g., about health) Comparing/choosing products
Information retrieval does not support these tasks (Bhavnani et al., 2002) E.g., Estimated that one must visit 25 Web
pages in order to read about 12 basic concepts about skin cancer
SENSEMAKING
SHOEBOX
EVIDENCE FILE
Search & Filter
Read & Extract
Schematize
Build Case
Tell Story
Search for Information
Search for Relations
Search for Evidence
Search for Support
Reevaluate
TIME or EFFORT
STRU
CTUR
E SCHEMAS
HYPOTHESES
PRESENTATION
EXTERNAL DATA
SOURCES
SENSEMAKING
SHOEBOX
EVIDENCE FILE
Search & Filter
Read & Extract
Schematize
Build Case
Tell Story
Search for Information
Search for Relations
Search for Evidence
Search for Support
Reevaluate
TIME or EFFORT
STRU
CTUR
E SCHEMAS
HYPOTHESES
PRESENTATION
EXTERNAL DATA
SOURCES
Sensemaking Loop
Foraging Loop
Entity Workspace Notebook
Drag-and-drop interface for capturing knowledge
Snap-together knowledge
Captures the user’s degree of interest
Controls automatic highlighting
AUGMENTED SOCIAL COGNITION
107 (months) SOCIAL Social Behavior 106 (weeks) 105 (days) 104 (hours) RATIONAL Problem Solving 103 102 (minutes) 101 COGNITIVE Immediate Behavior 100 (seconds) 10-1 10-2 BIOLOGICAL Automatic Behavior 10-3 (msec) 10-4
Newell
Meg Stewart
COGNITIVE LEVEL (Immediate behavior) Minimize mental time and motion (Efficient communication) Input on output (Representation shift) RATIONAL LEVEL (Problem solving) Maximize (information gain)/(time cost) (Efficient search) Information scent (by design) (Efficient semantic
search) Information scent (by machine) (Semantic coupling) Information visualization (Visual representation
shift) Input on data (Data coupling) Attention-reactive displays (Implicit coupling) Sensemaking tools (Representation shift) SOCIAL LEVEL (Social cognition) Social sensemaking (Social representation
shift)
TAGGING WORKS IF MANY PEOPLE
1 User
1 User: 6 tags Many users: 100
tags Need ~ 20
Furnas
Info
rmat
ion
Con
nect
ivity
Social Connectivity
Nova Spivack
Ben Wattenberg
Viégas & Wattenberg
MANY EYES
Social Amplification
Rep
rese
ntat
ion
Am
plifi
catio
n
COGNITIVE LEVEL (Immediate behavior) Minimize mental time and motion (Efficient communication) Input on output (Representation shift) RATIONAL LEVEL (Problem solving) Maximize (information gain)/(time cost) (Efficient search) Information scent (by design) (Efficient semantic
search) Information scent (by machine) (Semantic coupling) Information visualization (Visual representation
shift) Input on data (Data coupling) Attention-reactive displays (Implicit coupling) Sensemaking tools (Representation shift) SOCIAL LEVEL (Social cognition) Social sensemaking (Social representation
shift) Input on social context (Social representation
shift)
WikiDashboard Social Transparency: Make socially significant information visible.
Bongwon Suh & Ed Chi
lowering system costs increases sharing
Ma.gnolia
Ma.gnolia
Google Reader
Media Wiki Google Notebook
Google Notebook
intended recipient:
COGNITIVE LEVEL (Immediate behavior) Minimize mental time and motion (Efficient communication) Input on output (Representation shift) RATIONAL LEVEL (Problem solving) Maximize (information gain)/(time cost) (Efficient search) Information scent (by design) (Efficient semantic
search) Information scent (by machine) (Semantic coupling) Information visualization (Visual representation
shift) Input on data (Data coupling) Attention-reactive displays (Implicit coupling) Sensemaking tools (Representation shift) SOCIAL LEVEL (Social cognition) Social sensemaking (Social representation
shift) Input on social context (Social representation
shift)
Efficient Communication Tight Coupling
Representation Shift
-- Minimize mental time and motion -- Maximize (information gain/(time cost) -- Information scent (by design)
-- Input on data -- Implicit coupling -- Information scent (by machine)
-- Input on output -- Information visualization -- Sensemaking tools -- Social sensemaking