Multi Touch User Interfaces - · PDF fileMulti Touch User Interfaces Semir Elezovic Computer Science Jacobs University Bremen Campus Ring 1 28759 Bremen Germany Type: Guided Research

Multi Touch User Interfaces

Semir Elezovic

Computer ScienceJacobs University BremenCampus Ring 128759 BremenGermany

Type: Guided Research Final ReportDate: May 9, 2008Supervisor: Prof. Dr. Jurgen Schonwalder

Contents

1 Introduction 3

2 Multitouch Technologies and Multitouch Interaction Techniques 52.1 Multitouch Technologies . . . . . . . . . . . . . . . . . . . . . . 6

2.1.1 The keyboard . . . . . . . . . . . . . . . . . . . . . . . . 62.1.2 Touchpads . . . . . . . . . . . . . . . . . . . . . . . . . 62.1.3 Camera based multitouch technologies . . . . . . . . . . 7

2.2 Multitouch Interaction Techniques . . . . . . . . . . . . . . . . . 92.2.1 Gesture based interaction . . . . . . . . . . . . . . . . . . 92.2.2 Bimanual vs Unimanual Interaction . . . . . . . . . . . . 112.2.3 Precise Selection and Multi Cursor Techniques . . . . . . 122.2.4 Pressure sensitivity based interaction . . . . . . . . . . . 12

3 The Wiimote Solution 143.1 The world of HID . . . . . . . . . . . . . . . . . . . . . . . . . . 143.2 The Wiimote’s IR camera . . . . . . . . . . . . . . . . . . . . . . 183.3 Infrared Pen . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193.4 Calibration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193.5 Right click . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203.6 GlovePie . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

4 Conclusion 22

2

1 Introduction

From the first days of computers when we had to remember long tedious commandsin order to do simple tasks like copying and deleting, through the days of the dom-ination of Windows, Icons, Menus and Pointing (WIMP), to today’s TouchscreenUser Interfaces (TUI), efforts have been made to improve human-computer interac-tion by designing more intuitive and natural ways of understanding electronic mes-sages. TUI, as one of the most promising post-WIMP ways of human-computerinteraction, is a computer-pointing technology offering users reality-based inter-action based upon touch. Multi Touch User Interfaces offer additional level ofinteraction, by giving the user the ability to interact with multiple points of interestat a time. Multi-Touch technology also implies multiple users, which can be ofgreat use in interaction scenarios where multiple users jointly work on a project,offering new ways of interaction and collaboration. Using a Nintendo Wii remote(Wiimote), which can track sources of Infrared Light, and aiming it to a projectionscreen, we are going to create an interactive whiteboard. Infrared LEDs can beused to serve as pens and since the Wiimote can track up to 4 points, we can havemultiple pens interacting with our low-cost multi touch interactive whiteboard.

Multi Touch User Interface is a multifunctional gestural interface using hardwareand software to recognize, track and interpret multiple simultaneous touches on atouch screen. Touch being one of the most intuitive things in the world, a multitouch interface can lead human computer interaction beyond simple pointing andclicking, button pushing, dragging and dropping that dominated our interaction inthe past. Registering multiple simultaneous points of contact, multi touch userinterfaces can allow multiple users, and can demonstrate their real power whenoperating on larger screens [1]. Interactive whiteboards and digital walls wouldbe just an example of the new interaction and collaboration methods of allowingmultiple, simultaneous users to interact in an intuitive fashion without interferingwith each other[1][2].

Multi Touch User Interfaces are becoming a kind of chameleon providing a singledevice that can transform itself into whatever interface is appropriate for the taskgiven at hand. Instead of having additional physical buttons on the device, thiskind of interface will just draw a picture on the same area of the desired buttonsand give you the desired action by interacting with them.

In this project, I am going to present a low-cost technique to create a multi touchinteractive whiteboard with the help of a Nintendo Wii Remote. The Wiimote hasa built-in camera in the front with an InfraRed pass filter.

It is really easy and cheap to create an InfraRed pen, and since the Wiimote cameracan track up to 4 moving objects, we are going to imitate an interactive whiteboard which can allow 4 simultaneous users, or 4 pens in our case. We also need aprojector which projects the computer’s desktop onto our whiteboard’s surface and

3

thus enables the users to also interact with the computer using our IR pens. A 4point calibration is made to denote the working area of our interactive whiteboard,giving us a lot of freedom with respect to the size of the area on which we are canoperate.

Moreover, we will analyze the evolution of multitouch technologies, the technol-ogy shifts that eliminate traditional tradeoffs, the advantages and drawbacks thatarise with introducing new features, the way multiple points are being processed tooffer gesture-based interaction, the odds of consumers embracing multi touch andthe odds of multi touch replacing the domination of WIMP.”iSuppli predicts that the multi-touch market will nearly triple, increasing from112.9 million this year to 433.1 million by 2012”[3].

4

2 Multitouch Technologies and Multitouch Interaction Tech-niques

When analyzing multitouch technologies and multitouch interaction techniques itis very important to discuss some relevant criteria and common disctinctions onwhich we base our analysis:

• Touch screen or Touch pad: there is a lot of confusion with these two. Notanything that responds to touch can be called a touch screen. With touchscreens, the touch technology is superimposed over a display, and the useris interacting directly with the object on the location he is touching. If thetouch surface is not overlaid on a visual display then it will be regarded astouch pad[4].

• Horizontal or Vertical orientation: when operating on a horizontal touchsurface, resting the hand or the finger on the surface, will affect the inter-action. With a pressure-sensitive surface, foreign objects create fallacioustouch points causing malfunctioning of single touch systems. In verticallyoriented touch surfaces users do not tend to rest their hands on the screen.With the wiimote interactive whiteboard, since the wiimote IR camera istracking only IR light sources, the orientation doesn’t matter. There are nolimits to choosing how the surface is mounted.

• Size: the size can determine how many points/persons can operate on thetouch surface, and also which gestures would be suitable for the device. Thewiimote interactive whiteboard from this aspect, in ideal circumstances isvery suitable since it can be recalibrated to change the size of the interactivesurface.

• Multipoint or multitouch: with new technologies arising, it is really impor-tant to distinguish between multipoint or multitouch systems. The maindistinguishing feature is the variety of gestures and interaction techniquesthat the new technology brings and how characteristic it is to the technol-ogy. There are new technologies that bring novelties in terms of sensing thepoints of contact, but do not offer innovative interaction techniques, and justoffer another way of using already known interaction techniques (clicking,double clicking, dragging and dropping etc.).

• Multiperson or multitouch: When designing interaction techniques, it makesa difference whether two points are sensed from fingers of the same hand(one person interacting with multiple points of interest like holding the shiftkey and typing or performing gesture based interaction like rotation andzooming in), or from fingers of separate hands (the case with multiple usersworking on the same touch surface).Technology like Diamond Touch allows information to be determined inde-

5

pendently for each user. Each touch can be identified with a particular userand can broaden interaction techniques with multiple users. Diamond touchtransmits signals through antennas contained in the table, and using capac-itive coupling, these signals are coupled to the users and the chairs to thereceivers, and in that way it is possible to identify the points that each user isinteracting at[2]. Later, the computer processes the touch points’ locationsand gives corresponding responses.

• Semi-transparent surface or diffusing projection surface: In camera basedmultitouch systems, using a diffusing projection surface only detects objectsthat are on the surface or very near to it. Using a semi-transparent surface,like the TouchLight multitouch system, the cameras enable to resolve objectsthat are beyond the surface. This offers many new possibilities like facerecognition to identify the users[5].

2.1 Multitouch Technologies

2.1.1 The keyboard

They can be regarded as multitouch systems in a way. Touch typists are performingmultitouch interaction (pressing another key before releasing the previous one).Keystrokes are also a proof of this mechanical multitouch system: Ctrl + Alt +Del, is multiple points of interaction.

2.1.2 Touchpads

The 1985 multitouch three dimensional touch sensitive tablet[6] is among the firstmultitouch technologies to have appeared. The touchpad also approximates degreeof contact sensing by monitoring the increase in capacitance as the fingertip flattensagainst its surfaces. The hardware is designed following the requirements of therecursive area subdivision algorithm (to avoid scanning all pixels) and it consists ofa sensor matrix board which is based on the technique of capacitance measurementbetween finger tip and metal plate. The resolution can be further improved thanthe physical resolution of the sensor matrix by interpolating data from neighboringgroups of sensors.

Another touch pad based on capacitive coupling is the Diamond Touch system[2].It is composed of a grid of row and column antennas with unique signals so that thesystem can detect which contact point is by which user. When the user interactswith the surface the antennas couple with the users, who are capacitively coupledwith a receiver throught their chairs.

Advantages of touchpad:

6

• They have no need of intermediate device such as stylus. No fear of losingthe necessary stylus

• The Diamound Touch can identify which user is touching the surface.

• They are very suitable for hostile environments and no restrictions on inter-acting with more points of interest

2.1.3 Camera based multitouch technologies

Camera based multitouch systems can be either direct, when the camera is di-rected towards the user’s hands, or indirect where the camera is directed towardsa touch surface, such that the touch surface is rear projected, and the user is infront of the touch surface. One of the most prominent indirect camera based mul-titouch technologies that detect multiple points on rear projected surfaces is JeffHan’s multitouch technology[1]. It is based on frustrated total internal reflection,a phenomenon used to transport light efficiently with very little loss. Total internalreflection occurs when a light ray is travelling inside a higher-index medium andstrikes surface boundary against lower-index medium. When a finger touches thetouch surface, the light is frustrated, scattered downward and picked up by an IRcamera.

The advantages of Han’s multitouch technology is that it provides full touch infor-mation without occlusion or ambiguity issues. It also has astounding interactiontechniques: it can accomodate multiple people interacting on the same surface, it ismultitouch, gesture-based (it offers to pull a virtual keyboard which can be rescaledto match the user’s hand size), it can be mounted both horizontally and vertically.It uses a diffusing projection surface, same like HoloWall[7].

An improvement over the projection surface used is TouchLight[5], which uses asemi-transparent projection surface allowing it to see beyond the touch surface.

This can be used to allow face recognition to identify the users interacting with thetouch surface. Also by not using a diffusing projection surface, it permits a high

7

resolution image of objects placed on the display. It computes depth informationby relating binocular disparity, the difference in images of an object seen by leftand right eye resulting from the eyes’ horizontal separation. There are certaindisadvantages of indirect camera based multitouch technologies:

• It requires a lot of space behind the interaction surface because it is rearprojected by a camera

• It is not that suitable for harsh lightning environments

8

2.2 Multitouch Interaction Techniques

With the rapid technological growth, new applications have emerged demandingnew ways of interaction. In real life, interaction with objects is often fluid andtransparent and we are making full use of our dexterity. However, in the digitalworld it is a pity to be confined to interact with using only a single mouse clickwhile having powerful computers that can cope with multiple sources of input. Toserve as an example, imagine trying to tie your shoes, eat, write with only onefinger. In this section we are going to analyze different ways of multitouch tech-nologies utilizizng the multiple points of contact. We are going to explore the multitouch interaction techniques that the multitouch systems offer by transforming thesensed points to the correct software parameters depending on the mode of inter-action.

2.2.1 Gesture based interaction

Human communication can be performed in many modalities, including speech,gestures, facial and bodily expressions. Gesture has been used in places as posture,but there is a tendency of refering gesture as something dynamic[8]. To go evenfurther, gesture is a hand movement ( for example a down then right movementmeans go forward in an internet browser) or a hand posture (for example, in camerabased multitouch systems such as Touchlight[5] if the user is detected holding hisindex finger might mean that he wants to invoke the virtual keyboard) expressingan intended action. Single gestures may be used to invoke actions like next page,previous page, or simply put such gestures are used to invoke simple keyboardkeys or key strokes. Charade is a very good example of a gesture-only interface.The user is standing in front of a presentation system and is using a DataGlove inorder to issue dynamic hand gestures. The gesture set consists of 16 gestures andgestures are interpreted by a set start posture, end posture and the dynamic motionbetween the start and the end postures[9].

9

Multitouch applications like RoomPlanner[10](using Diamond Touch and two userssitting accross each other) fully exploit whole hand, multi finger and gesture basedinteraction by offering a rich gesture set to enable as intuitive an interaction aspossible.

The majority of multi touch technologies use gestural interaction style where anadditional sensed point would mean activating a certain mode of interaction ratherthan a parallel source of input. Exploiting the multiple sensed points, we can havecompound gestures, for example two fingers can perform a rotation of a photo.Microsoft’s Surface Computer[11] is one of the most prominent provider of mul-titouch technologies fully exploiting gestural interaction style. The gesture basedinteraction is carried out by a gesture analyzing software which acts as a layerbetween the application and the multitouch hardware. The application receivesgesture data and the objects affected by the gestures as input. The basic gesturesare: rotate, select, scale, and move.

The gesture analyzing software must keep track of the application’s affected ob-jects whenever a gesture is performed. One thing in common between the gestureanalyzing software and the application run on the multitouch system is position. Abounding rectangular box must be aligned with the object that is being affected atthe current time. The gesture analyzing software must analyze the gesture, and per-form the correct behaviour on the application’s object. Selection can be performedby drawing a line around the objects which we want to include. Multi finger se-lection can be performed by two fingers in the same manner by drawing lines eachwith relatively same starting and ending point. Rotation gesture can be performedby two fingers rotating either clockwise or counterclockwise. The running appli-cation responds with updating the affected object with the angle rotated.

A condition that must be satisfied for the rotation gesture is for both of the fingersto have different x and y heading. That can be checked with the following piece ofcode:

if(((finger1.x * finger2.x)<0)&&((finger1.y * finger2.y)<0))

10

A scaling gesture can be performed by placing two fingers at an object in oppositedirections and either separating them which will result in the scaling up, or bringingthem together which will result in scaling down. The scaling factor is determinedby how far away the fingers get separated. Once the scaling gesture is performed,the application is scaling the affected object by the scaling factor. A conditionthat must be satisfied for the scaling gesture is that the fingers must be in oppositedirection in either direction:

if(((finger1.x * finger2.x)<0) || ((finger1.y * finger2.y)<0))

A move gesture is performed by grabbing and dragging an object. That way theapplication changes the point of origin of the affected object. A condition that mustbe satisfied for the move gesture is that every finger must be in the same direction

if(((finger1.x * finger2.x)>0) || ((finger1.y * finger2.y)>0))

2.2.2 Bimanual vs Unimanual Interaction

Depending on the tasks carried out on the multitouch system, bimanual interactioncan offer some advantages over unimanual interaction. Two handed interactiontechniques closely resemble the multitouch interfaces philosophy, since it attemptsto increase parallelism in parameter input. Studies have shown that for scrollingand GUI manipulation, bimanual interaction increases speed of task completion.This is mainly due to the way users brainstorm performing a task by using twohands instead of one. Unimanual interaction is more beneficent when performingsimultaneous rotation and moving of objects.

An experiment was carried out by Leganchuk[12] with participants being askedto use an ellipse to enclose triangles, square and triangles. Users were using theconventional one handed technique modeled according to the latest techniques andmethods used in traditional graphical user interfaces, and also bimanually interact-ing by stretching with two hands the encompassing figure. Using the latter tech-nique the user was quicker and more precise in drawing the encompassing figure.

11

2.2.3 Precise Selection and Multi Cursor Techniques

When disputing about multi-touch interaction completely replacing WIMP, one hasto bear in mind the tasks that are best suited for multi-touch interaction. It is truethat direct touch interaction reflects the way we interact with real objects, but themouse is a lot more precise in interacting with small user interface elements thanthe finger. One might argue that arm fatigue, high error rates, smaller precisionmight be enough of a reason for the mouse to be dominant a lot longer. Multipoint touchpads can be used to control a mediating representation, for examplethe cursor, which controls objects. In this way the hand is not directly mapped tothe object being interacted, but it takes intermediary roles, in this case the cursor.With multiple cursors, the user can at the same time focus on the main target andperform rotation, scaling and moving operations.

Another interaction technique to avoid the lack of precision of the finger is a preciseselection technology described by Wilson and Benco[13].

This technique offers pixel-accurate targeting by using the distance between twocontact points to adjust the control-display ratio between a touchscreen and a cur-sor. Another feature is the ”hover state” simulation on devices that are unable tosense proximity.

2.2.4 Pressure sensitivity based interaction

Touch surfaces that give degree of contact independently for each sensed pointin a continous manner by monitoring the increase in capacitance as the fingertipflattens against its surfaces offer rich interaction. This combined with detectingwhich points are pressed simultaneously can find a place in many applications. Oneexample for this multiple point sensing with pressure sensitivity is a simulation of

12

a piano keyboard with polyphonic melodies. Pressure sensitivity can also be usedto control the thickness of the brushes in paint programs as described by K.C Smithin the 1985 paper[6].

13

3 The Wiimote Solution

The Wiimote is a revolutionary game controller used for Nintendo’s Wii gameconsole. Its motion tracking capability leaves places for reuse of its hardware andsoftware for purposes other than those the manufacturer planned. Due to the factthat it talks Bluetooth it can wirelessly establish a two-way communication withother Bluetooth devices, including the PC [14][15]. For the purpose of this project,we created a low cost multitouch interactive whiteboard and tested it succesfully indifferent interaction scenarios. The prototype succesfully realizes the design goalsplanned:

• The interactive whiteboard enables interaction with any software loaded onthe connected PC. The infrared pen should be used as a mouse to manipu-late and click objects on the computer’s desktop. The wiimote interactivewhiteboard prototype also provides right click in the PDA right click fash-ion, namely it performs a right click when left click is kept on the same placefor some ammount of time

• The interactive whiteboard should be able to utilize software designed tomaximize the ability of multi touch boards: sample multitoich programs,capturing notes, Google Earth, drawing programs

The benefits of this multi touch board in some aspects even exceed those of com-mercial multitouch boards. The main benefit is not restricting ourselves to thesize of the interactive whiteboard (as far as the Wiimote’s camera is positionedclose and well enough so that it can track the IR pens). We can also mount ourinteractive whiteboard on different places: our desk, our wall. These two aspectscombined give a lot of freedom on choosing interaction scenarios. The multi touchinteractive whiteboard prototype posesses the following characteristics:

• Multipoint: up to 4 users (each using one pen) can be accommodated

• Resizable: the operating touch surface can be resized (re-calibration needed)

• Durable: enables normal use without frequent re-calibration

In the following subsections, I will describe in more details the whole processbehind the wiimote solution as a cheap substitute for multitouch technologies.

3.1 The world of HID

The wiimote is built on a Broadcom BCM2042 bluetooth System-on-a-chip, andcan communicate with devices which follow the Bluetooth Human Interface De-vice (HID) standard. The Bluetooth HID standard closely follows the USB HIDstandard, which defines the configuration and communication protocols for Hu-man Interface Devices such as keyboards, mice, joysticks that humans use to di-

14

rectly enter data into the computer. When the Wiimote is paired with the computervia bluetooth it can be regarded as a HID-compliant device. Information about aHID device is stored in segments of the device’s ROM (read-only memory). Thesesegments are called descriptors. An interface descriptor can identify a device asbelonging to one of a finite number of classes.The USB HID[16] class requires that every device describes how it will communi-cate with the host device. During enumeration the device describes how its reportsare to be structured so that the host device can properly prepare to receive this in-formation. The reports are unidirectional and they can be input reports (from theWiimote to the host), and output reports (from the host to the Wiimote). To get theHID descriptor block from the Wiimote, the SDP protocol [17] is used, which candiscover which services are available and their respective features[18].

The SDP protocol uses a request/response model where each transaction consistsof one request protocol data unit (PDU) and one response PDU. Every PDU con-sists of a PDU header followed by PDU-specific parameters, among which is theTransactionID parameter, that is used to match the response PDUs to the requestPDUs. A service consists of a list of service attributes. A service attribute has twocomponents: an attribute ID which is a 16 bit unsigned integer and an attributevalue. The attribute value’s role is determined by the attribute ID and service classof the service record in which the attribute belongs.Here are some important service attributes that the wiimote returns when queriedwith the SDP protocol:

Attribute Identifier : 0x1 - ServiceClassIDListData SequenceUUID16 : 0x1124 - HumanInterfaceDeviceService (HID)

From here we can see that the attributes are specified by the HumanInterfaceDe-viceService (HID) service class. Each service class has a universally unique iden-tifier UUID which is contained in the attribute value for the ServiceClassIDListattribute and in this it is: 0x1124.

Attribute Identifier : 0x206 - DescriptorListData SequenceData SequenceInteger : 0x22Data : 05 01 09 05 a1 01 85 10 15 00 26 ff 00 75 0895 01 06 00 ff 09 01 91 00 85 11 95 01 09 01 91 0085 12 95 02 09 01 91 00 85 13 95 01 09 01 91 00 8514 95 01 09 01 91 00 85 15 95 01 09 01 91 00 85 1695 15 09 01 91 00 85 17 95 06 09 01 91 00 85 18 9515 09 01 91 00 85 19 95 01 09 01 91 00 85 1a 95 0109 01 91 00 85 20 95 06 09 01 81 00 85 21 95 15 0901 81 00 85 22 95 04 09 01 81 00 85 30 95 02 09 0181 00 85 31 95 05 09 01 81 00 85 32 95 0a 09 01 8100 85 33 95 11 09 01 81 00 85 34 95 15 09 01 81 00

15

85 35 95 15 09 01 81 00 85 36 95 15 09 01 81 00 8537 95 15 09 01 81 00 85 3d 95 15 09 01 81 00 85 3e95 15 09 01 81 00 85 3f 95 15 09 01 81 00 c0

This is the HID descriptor block of all the available input and output HID reports.Each report has Report ID and Report Size[19]. There is no information on thedata units within the reports, only the report IDs and the respective report countsare given. They are extracted in the following way:

85 20 Report ID=2095 06 Report count=6

There are two well-known easy-to-use APIs for the Wiimote that handle all theHID communication and can be used in any software applications. One of them is alibrary written by Brian Peek[20], on which Johny Chung Lee’s application and thejava implementation are based, and the other one is GlovePIE[21], an applicationspecifically designed to emulate Joystick and Keyboard input from a variety ofdevices. We will use Peek’s library to explain the underlying principles of playingwith the HID reports, since the we are using software built on top of it.

The way communication is established between the wiimote and bluetooth in thewiimote library written by Peek, is by getting a list of devices which are part of theHID class, and comparing Vendor ID and Product ID of each device till a matchis found with the already known Wiimote’s Vendor ID and Product ID. Then aFileStream is created to carry out the communication. Here is a list of the inputand output reports specified:

public enum InputReport : byte{Status = 0x20,//report count = 6ReadData = 0x21,//report count = 21Buttons = 0x30,//report count = 3ButtonsAccel = 0x31,//report count = 5IRAccel = 0x33,//report count = 17ButtonsExtension = 0x34,//report count = 21ExtensionAccel = 0x35,//report count = 21IRExtensionAccel = 0x37,//report count = 21

};private enum OutputReport : byte{

None = 0x00,

16

LEDs = 0x11,//report count = 1Type = 0x12,//report count = 2IR = 0x13,//report count = 1Status = 0x15,//controller status report count = 1WriteMemory = 0x16,//report count = 21ReadMemory = 0x17,//report count = 6IR2 = 0x1a,//report count = 1

};

For our project, we need to initialize the IR camera to start tracking for IR spots.The reports regarding the IR camera are both output i.e. they are sent from thehost to the Wiimote. There are methods implemented to enable and disable the IRcamera. The IR camera should be disabled when not used in order to save batterylife:

private void EnableIR(IRMode mode){...

}private void DisableIR(){

...}

These are the reports for the Wiimote’s IR camera use:

Table 1: IR Camera ReportsReport ID Report Size Use

0x13 1 IR Camera Enable 10x1a 1 IR Camera Enable 2

So, the computer that communicates with the Wiimote should perform the actions:Enable IR Camera and Enable IR Camera 2 to initialize the IR Camera. In order todo that, bit 2 on output reports 0x13 and 0x1a has to be set:

Table 2: Enable IR cameraBit 7 Bit 6 Bit 5 Bit 4 Bit 3 Bit 2 Bit 1 Bit 1

? ? ? ? ? IRPCK ? RUMBLE

17

Here is the table for the Enable IR camera 2:

Table 3: Enable IR camera 2Bit 7 Bit 6 Bit 5 Bit 4 Bit 3 Bit 2 Bit 1 Bit 1

? ? ? ? ? IREN ? RUMBLE

Bit 2 is set by sending 0x04 in the first byte of the payload

13 041a 04

There are additional HID reports that need to be sent and handled in order to handlesensitivity and write to registers.

3.2 The Wiimote’s IR camera

The Wiimote includes a 128x96 monochrome camera with an IR-pass filter on thefront. If the IR filter is removed, the Wiimote will track any bright object. Theinfrared camera contains a built-in image processor capable of tracking up to 4moving objects. The Wii Operating System and many Nintendo Wii games use theIR camera in combination with a sensor bar (which is only two clusters of IR lightemitting diodes) to provide yaw - the horizontal angle of rotation of the Wiimote tothe sensor bar. Without the sensor bar, only pitch and roll can be measured reliably.That is because orientation is estimated by measuring the direction of gravity, butthe direction doesn’t change when the wiimote is rotated in the horizontal plane. Inthat way, yaw combined with pitch and roll, complete orientation of the Wiimotecan be achieved[22].

The built-in processor analyzes the raw camera image, reports up to 4 bright spotsit detects on the image, and computes their (x, y) positions and approximate radiuson the camera’s image plane. These values can be queried by the host computer.As a result it provides an 1024x768 resolution for the tracked points by using an8x subpixel analysis. All these values are included in the Bluetooth’s report stream[18][23].

The IR Camera tracks up to four IR hotspots it detects in its image, with no par-ticular order. When it detects a hotspot it assigns it to the first empty slot. Oncean object goes out of the frame, the slot it was in is emptied. The IR camera cangive the illusion that it tracks more than 4 spots due to the fast switching betweenpoints. The infrared camera offers a 1024x768 resolution for the tracked points,and it returns data for the computer X (ranging from 0-1023) and Y (ranging from0-767) coordinates, plus additional data depending on the mode of the report: ba-sic, extended or full.

18

3.3 Infrared Pen

This will be the infrared light source that the Wiimote infrared camera is goingto track. The IR pen is very easy to construct and very cheap. It consists of apen housing, IR LED (1.5 V), battery (1.5 V) and a switch. We can operate withthe connected computer’s desktop in the following manner: when the IR pen isswitched on and the IR LED is activated, then the mouse is moved to where the IRpen is pointing and a left click is performed. The interactive whiteboard softwareis dependent on the reflection of the LED off of the wall/screen to pick up the pen’sposition. In case we want to perform a right click, we can hold the IR pen onthe same position for some amount of time. This is similar to the point and clickmechanism used in Personal Digital Assistants. Another fact is that the wiimote’sfield of vision is 45 degrees. The wiimote has to be positioned in a way that it candetect all four points required for calibration.

3.4 Calibration

When starting the software, we are required to perform calibration to get the rightscreen position whenever we are pointing to the area we operate on. It is necessaryto transform the corrdinates of the wiimote camera to the screen coordinates. Forthat it requires exactly 4 points to get the perspective transformation from onequadrilateral (the 4 IR points that the wiimote sees when calibrating) to the otherquadrilateral (4 points of the intended area of interaction). To get the corresponding2D (x,y) point of a 3D point’s (the IR source) projection onto the camera image’splane, it is required that the 3D point in global coordinates is transformed into thewiimote’s IR camera’s local coordinate system, and then projected onto the imageplane from the camera’s focus point taken to be at the local origin.

Another useful thing is to show the user to what extent the wiimote’s IR camerais precise at detecting the IR sources. The percentage is taken of the area of thequadrilateral formed by the actual coordinates of the points where the IR pen waspointed from the area of the quadrilateral formed by the ideal coordinates.

float UpdateTrackingUtilization(){float idealArea = (1 - 2*calibrationMargin) * 1024 *(1 - 2*calibrationMargin) * 768;float actualArea = 0.5f * Math.Abs((srcX[1] - srcX[2]) *(srcY[0] - srcY[3]) - (srcX[0] - srcX[3]) * (srcY[1] - srcY[2]));float util = (actualArea / idealArea)*100;String utilstatus = "Tracking Utilization: " + util.ToString("f0")+"%";BeginInvoke((MethodInvoker)delegate() { lblTrackingUtil.Text = utilstatus; });return util;

}

19

Calibration gives us a lot of freedom with respect to the size of the area on whichwe can operate.

3.5 Right click

Right click is performed in the following manner:First we get the right click delay time in miliseconds. The method trigger is calledafter the method process(Point), and what it does is it checks whether there is alightStartPoint detected, and compares the difference between the current sys-tem time and the system time when an IR light was detected to the right click delaytime to see whether enough time has passed so that it can activate the right mouseclick. In the end, it just returns a boolean telling whether the right mouse click wasactivated or not.

public boolean trigger(){if (lightStartPoint != null && System.currentTimeMillis() -lightStartTime > prefs.getRightClickDelay()){if (!active) {active = true;Mouse.LEFT_BUTTON.setPressed(false);activate();

}return true;

}return false;

}

The method process(Point p) is responsible with saving the current cursor posi-tion and time whenever an IR light has been detected.

public void process(Point p){if (p != null){if (!active && (lightStartPoint == null ||lightStartPoint.distance(p) > PIXEL_MOVE_TOLERANCE)){lightStartPoint = p;lightStartTime = System.currentTimeMillis();

}}...

}

20

3.6 GlovePie

GlovePie offers another easy implementation of the interactive multitouch white-board prototype. GlovePie is a programmable input emulation device scriptinglanguage and offers a lot of effortlessness with receiving and processing data fromthe wiimote.

Following the same manner, a left click is performed wherever an IR light sourceappears. The way calibration is done in one GlovePie implementation is by deter-mining the parameters that affect the transformation of the wiimote’s coordinatesto the interaction area’s coordinates. For each of the IR points this formula applies:

xs =a1 · x + b1 · y + c1

a3 · x + b3 · y + 1(1)

ys =a2 · x + b2 · y + c2

a3 · x + b3 · y + 1(2)

This way, we have two formulas for each IR point or 8 formulas in total, and exactly8 unknowns (a1, a2, a3, b1, b2, b3, c1, c2 are the unknown parameters). Once theparameters are known, the correct screen position can be found whenever the IRpoint is triggered at the interaction area.

21

4 Conclusion

Multitouch interfaces reflect the way we interact with real life objects. It is veryintuitive and very useful in varios interaction scenarios. They are also not com-pletely throwing away the traditional interaction ways: ex. Jeff Han’s perceptivepixel technology is not throwing away the keyboard: he is offering a virtual key-board, whose size can be accomodated to the user’s preferences. However, mul-titouch technologies are met with big challenges: people are already used to theWIMP way, and they cost a lot.

The wiimote solution to the low cost interactive whiteboard is very appealing andgives satisfactory results when considered with the above mentioned criteria: itssize can vary, it can be mounted both vertically and horizontally, it is multitouch (4points of contact), can successfully imitate multiperson, multipoint and multitouchsystems with some restrictions, and is very cheap to build.

22

References

[1] Jefferson Y. Han. Low-Cost Multi-Touch Sensing through Frustrated TotalInternal Refl ection. Technical report, New York University.

[2] Paul Dietz and Darren Leigh. DiamondTouch: A Multi-User Touch Technol-ogy. Technical report, MITSUBISHI ELECTRIC RESEARCH LABORA-TORIES, October 2003.

[3] Jessica Boether Mark Chisholm Shawn Dubravac Tim Herbert Steve KoenigCarolyn Slater Matt Swanston Gary Shapiro, Sean Wargo. 5 TechnologyTrends to watch. Technical report, CEA - Consumers Electronics Associa-tion.

[4] Bill Buxton. Case study 1: Touch tablets. Technical report.

[5] Andrew Wilson. A Demonstration of TouchLight, an Imaging Touch Screenand Display for Gesture-Based Interaction. Technical report, Microsoft Re-search.

[6] K.C. Smith SK. Lee, W. Buxon. A multi touch three dimensional touch sen-sitive tablet. Technical report, Computer Systems Research Institute, Univer-sity of Toronto.

[7] N. Matsushita and Rekimoto. HoloWall. Technical report.

[8] David J. Struman. Whole Hand Input. Technical report, Massachusetts Insti-tute of Technology.

[9] Mark Billinghurst. Gesture based interaction. Technical report, MicrosoftResearch.

[10] Ravin Balakrishnan Mike Wu. Multi-Finger and Whole Hand Gestural Inter-action Techniques for Multi-User Tabletop Displays. Technical report, Uni-versity of Toronto.

[11] Microsoft. Microsoft’s Surface Computer,http://www.microsoft.com/Surface/index.html.

[12] Shumin Zhai Andrea Leganchuk and William Buxton. Manual and CognitiveBenefits of Two-Handed Input: An Experimental Study. Technical report,University of Toronto.

[13] Andrew Wilson Hrvoje Benco. Precise Selection Techniques. Technical re-port, Microsoft Research, Columbia University, New York.

[14] Torben Schou and Henry J. Gardner. A Wii Remote, a Game Engine, FiveSensor Bars and a Virtual Reality Theatre. Technical report, Department ofComputer Science, FEIT.

23

[15] Wii homebrew. http://en.wikipedia.org/wiki/Wii homebrew.

[16] USB Implementers’ Forum. Device Class Definition for Human InterfaceDevices (HID). Technical report.

[17] Service Discovery Protocol (SDP). http://www.palowireless.com/infotooth/tutorial/sdp.asp.

[18] Wiili. http://www.wiili.org/index.php/Main Page.

[19] Craig Ranta and Steve McGowan. HUMAN INTERFACE DEVICE (HID)PROFILE. Technical report, May 2005.

[20] Brian Peek. Managed Library for Nintendo’s Wiimote,http://blogs.msdn.com/coding4fun/archive/2007/03/14/1879033.aspx.

[21] Carl Kenner. GlovePIE.

[22] NASA Ames Research Center. VMS Six Degrees of Freedom,http://www.simlabs.arc.nasa.gov/vms/dof.html.

[23] Wii Remote. http://en.wikipedia.org/wiki/Wii Remote.

24