Physical User Interfaces What they are and how to build them Saul Greenberg University of Calgary.

Physical User InterfacesWhat they are and how to build them

Saul GreenbergUniversity of Calgary

Physical User Interfaces

Special purpose computer-controlled devices that can be situated in a real-world setting.

• typically designed for particular contexts and uses.

Physical User Interfaces

Foundations for• Ubiquitous Computing• Tangible Media• Context aware computing

Styles of use• foreground interaction• ambient displays• information appliance• collaborative interaction• physical controls

Ubiquitous Computing

• A less-traveled path I call the invisible; its highest ideal is to make a computer so imbedded, so fitting, so natural, that we use it without even thinking about it.

• Provide hundreds of wireless computing devices per person per office, of all scales … It is invisible, everywhere computing that does not live on a personal device of any sort, but is in the woodwork everywhere.

Source: Mark Weiser’s UbiqCom web site

Mark WeiserXerox PARC

Ubiquitous Computing

Invisible Everywhere Computing

• invisible: o tiny, embedded, attachable…

• everywhere: o wireless, dynamically configurable,

remote access, adapting…

• aka Pervasive Computing


Mark WeiserXerox PARC

Ubicomp is Situated Computing

Makes use of simple shared context

• space • time• proximity• affordances

Participation in the context• is physical• is out here with us• is in many small and large

places, including trivial ones


Ubicomp Technology Trends


Ubicomp Technology Trends

Processors: • cheap, small, dedicated, microprocessors

Low Power• small batteries, solar (?)

Wireless• 802.11, Bluetooth, infrared, mobile telephony, …

Displays • very small (inches) to very large (walls)

Peripherals• sensors, actuators, motors, …

Run-time systems• Linux on a chip, Windows CE, downloadable executables…


ParcTab

Mobile hardware • infrared• room-sized cells• location information

A portable GUI device• small case with belt clip, ½ size of PDAs• touch sensitive 128x64 pixels display • 3 finger-operated mechanical buttons (chorded)• piezo-electric speaker• low power needs (~ 1 week between charges )• can be used in either hand by rotating display


Tangible User Interfaces

From ‘painted bits’ to ‘tangible bits’• genre change• give physical form to digital information• physical objects, surfaces, and spaces become

tangible embodiments of digital information.

seamlessly couple the dual worlds of bits and atoms • input: grasp and manipulate• output: change of physical properties of object

Source: Tangible Media Group web site

Hiroshi IshiiMIT

Context-Aware Computing

Context as information• … characterizes a situation of a person, place or object relevant

to the interaction between a user and an applicationo location o identity o state and activites of people, groups o state of computational and physical objects

Context-aware computing• uses contextual information to

o selectively present information and serviceso automatically execute a serviceo attach context information for later retrieval

Source: Dey, Abowd and Salber, HCI Journal 2001

Styles of use

Foreground interactions

Conscious intentional interactions• graspable objects • augmented surfaces• link between physical and virtual actions

• exploit human senses of touch and kinesthesia.

MusicalBottles - controlling sound, light

InteractionOne opens and closes bottles to control digital contents e.g., opening it tells a story

Music bottlesmovement and uncorking of the bottles controls the different sound tracks and the patterns of colored light rear-projected onto the table’s translucent surface.

Source: Tangible Media Group web site

MusicalBottles - controlling sound, light

Ishii, MITSource: Tangible Media Group web site

IAMASCOPE

Wooden Mirror - wooden pixels, image

Ishii, MITDaniel Rozin, NYU

Wooden Mirror - wooden pixels, image

Ishii, MIT Daniel Rozin, NYU

Triangles – building a digital story

When triangles connect together, they trigger digital events.

These events influence the progress of a non-linear story, or allow users to organize media elements in order to create their own story space.

Extracted from Tangible Media Group web site

Triangles – building a digital story


Ambient displays

Background information • communicate digitally-mediated senses of activity and presence

at the periphery of human awareness. • ambient light sound, airflow, water movement, object motion… • peripheral displays

Extracted from Mark Weiser’s UbiqCom web site

Extracted from Mark Weiser’s UbiqCom web site

Pinwheels - ambient digital wind


Pinwheels - ambient digital wind


Information Perculator

bubbles of digital patterns

Heiner, Hudson & Tanaka

Information Perculator

bubbles of digital patterns

Heiner, Hudson & Tanaka

Ambient Light Display

light reflection from water onto ceiling

Ambient Room

ACM CHI’98, Tangible Media Group

Personal Ambient Display

Small, physical devices worn to display information to a person in a subtle, persistent, and private manner.

Ambient information is displayed solely through tactile modalities such as heating and cooling, movement and vibration, and change of shape.


Physical controls

Physically-mediated computer controlled interactivity

Marble Answering Machine

Incoming voice messages are physically instantiated as marbles.

The user can grasp the message (marble) and drop it into an indentation in the machine to play the message.

The user can also place the marble onto an augmented telephone, thus dialing the caller automatically.

Durrell Bishop

TouchCounters

Computational tags track the usage of physical objects.

TouchCounters sense activity through magnetic, acceleration, and infrared sensors, and indicate their status on bright LED displays.

TouchCounters can be networked to a web server that generates use histograms for each object.


TouchCounters


Tagged Objects

ACM CHI’99, Xerox Parc

mediaBlocks


SenseTable

A system for tracking the positions and states of multiple objects wirelessly on a flat surface.

Objects can be equipped with various controls -- dials or buttons -- which can be monitored in real-time.

When coupled with a projector, the system can display information about the objects on or near the objects themselves.


HandScape

A vectorizing digital tape measure for digitizing field measurements, and visualizing the volume of the resulting vectors with computer graphics.

Using embedded orientation-sensing hardware, it captures relevant vectors on each linear measurements and transmits this data wirelessly to a remote computer in real-time.


Manipulative User Interfaces

Customizable Physical Interfaces

ACM UIST 2002, Greenberg & Boyle

Collaborative interactions

Physical objects connecting collaborators• collaborative interaction• awareness

Adapted from Tangible Media Group web site

PingPong Plus

features a "reactive table" that incorporates sensing, sound, and projection technologies.

Projectors display patterns of light and shadow on the table; bouncing balls leave images of rippling water; and the rhythm of play drives accompanying music and visuals.


PingPong Plus


InTouch – collaborative haptics

Force-feedback technology is employed to create the illusion that people, separated by distance, are interacting with a shared physical object.

When one of the rollers is rotated, the corresponding roller on the other distant object rotates in the same way. Two people separated by distance can then play…


InTouch – collaborative haptics


Bench

…two cold steel benches located in different cities.

When a person sits on one of these benches, a corresponding position on the other bench warms, and a bi-directional sound channel is opened.

At the other location, after feeling the bench for "body heat," another person can decide to make contact by sitting near the warmth.

Initially the sound channel is distorted, but as the second party lingers, the audio channel clears.

--summarized by Ishii and Ullmer

Anthony Dunne and Fiona Raby at the RCA

Physical but Digital Surrogates

Rotating figurine servo motor

Tippable figurine light sensors

Proximity detector ultrasonic sensor

Hydra unitvideo, camera, speakers, microphone

Physical but Digital Surrogates

Privacy preserving media space

Roomware i-land

Computer-augmented room elements• like doors, walls, furniture (e.g. tables and chairs) with integrated information

and communication technology.

From the GMD Darmstadt web site on I-Land

Roomware i-land

Dynawall


Roomware i-land

CommChair


Roomware i-land

ConnecTableBy moving multiple

ConnecTables together, they can be arranged to form a large display area. Integrated sensors measure the distance between the ConnecTables and initiate the automatic coupling of the displays


Roomware i-land

InteracTable•touch-sensitive plasma-display (PDP) is integrated into the table top•Border for leaning…


Roomware i-land

Phillips – Intelligent Information Surfaces

From the Philips Lime Video CD

Tokens

Phillips – Intelligent Information Surfaces

From the Philips Lime Video CD

Designing out of the box

The problem:• programming / designing with physical devices is hard

o circuit design (electrical engineering)o microprocessor interface to digital/analog deviceso ‘wire’ interface (serial, USB, wireless, IR…)o wire protocolo connection/disconnection/intermittent connectivityo software to use deviceso maintenance and extensibility

• simple things take a long time to do• most people don’t bother

Solution 1: Interdisciplinary team

Works, but…• Still takes time• When one of the team

leaves, knowledge islost

• systems hard tomaintain &extend

Hydra unitvideo, camera, speakers, microphone

Rotating figurine servo motor

Tippable figurine light sensors

Proximity detector ultrasonic sensor

Solution 2: Hack existing devices

Programmable Embodied Agents (Kaminsky et al)

• hacked Microsoft Actimates

http://www.microsoft.com/products/hardware/actimates/pc/

Solution 2: Hack existing devices

Programmable Embodied Agents (Kaminsky et al)

• Arm position -> quantity• Squeezing hand/leg -> counting• Movement-> task progress

Proxy for other person squeeze hand, other’s hand goes up)

Event monitoring Signal document is printing, then complete Barney Email biff

Solution 3: Phidgets

Physical Widgets• simple, easy to program devices and component-based software

with well-defined API• building blocks for physical interfaces

• analogous to GUI widgets

Phidget Examples

PhidgetServo:• Control 1 or 2 servo motors

PhidgetPowerbar• Control power state of outlets on a power bar

PhidgetInterfaceKit• 8 simple input and outputs plus 2 sensors• A constructor kit

PhidgetProximitySensor• Returns how close something is to it

PhidgetMotionDetector• Periodically returns the amount of motion in a space

Related areas

Mobile ComputingAugmented RealityContext-aware computingReactive EnvironmentsUbiquitous MediaCooperative Buildings

Digital Inputs - switches

Rocker Toggle Push button Push-Pull

Rotary Slide Tactile Keylock

Analog inputs: Environmental Sensors

Distance

Light MotionPressureTemperature

Voltage Weight

rangefinder

Distance

proximity

Analog inputs: Input Sensors

Force Mini-joystick

Potentiometers

multi-turnsingle-turn

Capacitive

encoder

Accelerometer

slider

Analog inputs: Input Sensors

Bend

Force

Tilt

Camera

Custom Input: Identification

RFID Tags and Antenna Bar Code Scanner

Custom inputs

Camera

Digital Outputs – low power

Lights

LEDslamps

Relays Solenoids

http://www.supertrip.co.za/new_page_7.htm

Output: Motors

Position: 0-180o Servo

Rotate by steps: +/- xo

Stepper

SpeedDC Motor

Output: Displays

Text LCD

Numeric-alpha

Graphics (not yet)

Approaches: PIC Micro-controller

Single programmable chip computer with:• CPU, RAM, ROM, I/O, serial/parallel ports, A/D and D/A converters

Need to know:• basic circuit design• basic electronics

o resistor, capacitor, diodes, transistors…

• low level programming• PIC hardware details• …

Powerful, flexible, but• learning curve• excessive time in low level details• complex things are hard• serial

e.g. www.voti.nl/swp

http://www.voti.nl/swp/psu-big.gif

Approaches: Basic Stamp

Pre-built boards• Pic microcontroler• pre-wired circuits and connectors• boards designed for different uses

Still need to know • electronic components • electronic circuitry • PBasic language: stamp-specific

instruction set

Still flexible, but• learning curve still there• time in low level details• complex things are hard• serial

Approaches: HW as SW components

Phidgets and Making Things• dedicated devices• some plug and play electronics• under direct computer control• well-defined component-based software

o Interface via APIs, objects, and/or widgets

Need to know • conceptual knowledge of device• high level programming language• software component documentation

High design flexibility, low electronic flexibility• very low learning curve• design by combining and varying• time in conceptual design, not electronics

Approaches: Programmable Brick

Lego Mindstorms• robotics (downloadable code)• computer control as well• proprietary RCX microcontroller• infrared communication• 3 input ports

o light/touch sensors…• 3 output ports

o motors, lights…• Lego building blocks• children’s programming language for downloading but

o well-defined SDKo 3rd party access from standard languages

Need to know • SDK / language

High design potential for robotics• limited input/output (3+3)• expensive for basic set, plus add-ons

e.g. http://www.crynwr.com/lego-robotics/

www.mastincrosbie.com/mark/lego/images/grabberarm1.jpg

e.g. http://www.crynwr.com/lego-robotics/

Physical User Interfaces What they are and how to build them Saul Greenberg University of Calgary.

Documents

physical user interfaceswhat

small inches

large wallsperipheralssensors

large places

later retrieval source

groups state of computational

dual worlds of bits

belt clip