9/20/20151 Interaction Devices CIS 577 Bruce R. Maxim UM-Dearborn.

04/19/23 1

Interaction Devices

CIS 577

Bruce R. Maxim

UM-Dearborn

04/19/23 2

User Communication

• Keyboard remains the user’s primary input device• Most GUI systems expect pointing devices as well

– Light pen (direct)– Touch screen (direct)– Mouse – Trackball – Joystick – Graphics tablet – Touch pad – Foot controls (novel)– Gloves (novel)– Eye movement detectors (novel)

04/19/23 3

Pointing Device Tasks

• Selection • Positioning objects • Orienting objects on screen • Path tracing • Quantify • Text entry/editing (not easily done)

04/19/23 4

Direct vs Indirect

• Light pens and touch screens allow the fastest user responses, but are the least accurate (finger vs pixel level resolution)

• Graphics tablets with a stylus allows good resolution and long periods of use without switching to the key board (with good software design – often menu driven)

04/19/23 5

Comparison

• Keyboard– Fast control, require memorization and recall

• Mouse– Best choice for pointing to objects in arbitrary

positions, can be as fast as pointing with a finger

• Joystick– Poor cursor control devices, actually slower than

mouse (only use where they are game realistic)

04/19/23 6

Speech Technology

• Discrete word recognition • Continuous speech recognition • Speech store and forward • Speech generation

04/19/23 7

Discrete Word Recognition

• 90 to 98% reliability for small vocabulary • Usually requires speaker dependent training • Most people would rather type than dictate

04/19/23 8

When should you use it?

• Speakers hands are busy • Mobility required • Speakers eyes are occupied • Harsh or cramped conditions prevent use of

key board

04/19/23 9

Speech Store and Forward

• Voice mail type technology • Video games • Low cost • Resource intensive

04/19/23 10

When to use computer generated speech?

• Message is simple • Message is short • Message will not be referred to later • Message deals with events in time • Message requires immediate response • Visual communications channels are overloaded • Environment lighting is bad • User must move around • User subjected to high G forces or lack of oxygen

04/19/23 11

Display Technologies• CRT

– raster scan cathode ray tube

• LCD– flicker free, lightweight, limited viewing angles

• Plasma– early touch screens

• LED– some resolution limitations

• Projectors– room lighting and resolution restrictions

04/19/23 12

Video Display Features

• Rapid operation • Reasonable size (24 by 80 or 66 by 166 chars)• Reasonable resolution (768 by 1024 minimum)• Quiet operation • No waste paper • Low cost (< $100)• Reliable • Good for highlighting • Good for graphics and animation

04/19/23 13

Image Technologies

• Scanners (< $100)• Digital cameras (< $200)• Digital video (< $200)• 12 inch video disk • CD ROM (600 MB data, 1 hour music)• DVD (3 GB data, 2 hours video)• Motion Capture

04/19/23 14

Hardcopy Devices

• Printers – ink jet, laser, thermal, dot matrix

• Plotters– seem to be vanishing (slow, high resolution)

• Photographic printers– ink jet technology, special paper

04/19/23 15

Color• Can soothe or strike the eye • Adds accents to an uninteresting display • Facilitates subtle discriminations in complex

displays • Emphasizes the logical organization of

information • Draws attention to warnings • Can evoke strong emotional reactions of joy,

excitement, fear, or anger

04/19/23 16

Color Display Problems

• Colors can't be represented perfectly on digital computer systems

• People don’t have the same perceptions of colors

• Color blindness• Eye fatigue can be caused by using bad color

combinations

04/19/23 17

Color Description Schemes

• Computer Industry Approach (RBG)– Red, Green, Blue (0 – 255), additive model

• Ink jet Approach (CMY or CMYK)– Cyan, Magenta, Yellow, subtractive model

• Broadcast TV (YIQ)– Y = luminence, I, Q = chromacity

• Psychological Dimensions (HSB or HSV) – Brightness (amount of light energy)– Hue (predominant wave length)– Saturation (color purity)

RGB Model

RGB Model

RGB Model

• Note that for the "complementary" colors the sum of the values equals white light (1, 1, 1).

• For example:red (1, 0, 0) + cyan (0, 1, 1) = white (1, 1, 1)

green (0, 1, 0) + magenta (1, 0, 1) = white (1, 1, 1)

blue (0, 0, 1) + yellow (1, 1, 0) = white (1, 1, 1)

CMY Model

CMY Model

CMY Model

• Cyan = Green + Blue, so light reflected from a cyan pigment has no red component, i.e., the red is absorbed (or subtracted) by cyan.

• Magenta subtracts Green• Yellow subtracts Blue. • Printers usually use four colors: Cyan, Yellow,

Magenta and Black.• Cyan, Yellow, and Magenta together produce

a dark gray rather than a true Black.

CMY Conversions

• RGB to CMY

C = 1 – R

M = 1 – G

Y = 1 – B

• CMY to RGB

R = 1 – C

G = 1 – MB = 1 – Y

• For printing (CKYK)

K = min(C,M,Y)

C = C – K

M = M – K

Y = Y – K

• Where K = Black

RGB to YIQ

|Y| |0.299 0.587 0.114| |R|

|I| = |0.596 -0.275 –0.321| * |G|

|Q | |0.212 –0.528 0.311| |B|

• For YIQ to RGB use inverse matrix

HSV Model

HSV Model

• Pure Red is H=0, S = 1, V = 1• White is S = 0, V = 1 (H is irrelevant)• Black is V = 0 (both H and S are irrelevant)• Adding white is done by decreasing S and

leaving V unchanged• Changing shades is done by decreasing V

and keeping S unchanged• Create tones by decreasing both S and V

Using Color Effectively - 1

• Limit total number of colors• Recognize power of color to speed up or slow

down a task• Ensure color coding supports user task• Have color coding appear with minimal user

effort• Place color coding under user control• Design for good monochrome display first• Use color to help formatting

04/19/23 29

Using Color Effectively - 2

• Use consistent color coding• Be alert to user expectations for certain

colors (e.g. red means stop or danger)• Use color changes to indicate state changes• Use color in graphic displays to increase

information density• Be alert to bad color pairings (e.g. saturated

blue and red)• Consider needs of color impaired users

04/19/23 30

Direct-Input

1. Create main DirectInput interface using CreateIDirectIntput8( )

2. (optional) Query for device GUID’s for all devices (mouse, keyboard, joystick)

3. Create each device using CreateDevice( ) please note: GUID_SysKeyboard and GUID_SysMouse are built in

4. Set cooperation level for each device using SetCooperativeLevel( )

5. Set data format for each device using SetDatFormat( )

04/19/23 31

Direct-Input

6. Set properties of each device using IDirectIntputDevice8::SetProperty( )

7. Acquire each device using IDirectIntputDevice8::Acquire( )

8. (optional) Poll devices using IDirectIntputDevice8::Poll( )

9. Read device using IDirectIntputDevice8::GetStateDevice( )

04/19/23 32

Input

• DirectX can request input in two ways– Immediate input (default)– Buffered input

• LaMothe only uses immediate input• DirectX can cut Windows out of the message

loop altogether, so be careful when you set each device cooperation level

04/19/23 33

LaMothe Example

04/19/23 34

Keyboard Global Declarations

#include <ddraw.h> // directX includes

#include <dinput.h>

#include "T3DLIB1.H"

// directinput globals

LPDIRECTINPUT8 lpdi = NULL; // dinput object

LPDIRECTINPUTDEVICE8 lpdikey = NULL; // dinput keyboard

LPDIRECTINPUTDEVICE8 lpdimouse = NULL; // dinput mouse

LPDIRECTINPUTDEVICE8 lpdijoy = NULL; // dinput joystick

GUID joystickGUID; // guid for main joystick

char joyname[80]; // name of joystick

// these contain the target records for all di input packets

UCHAR keyboard_state[256]; // contains keyboard state table

DIMOUSESTATE mouse_state; // contains state of mouse

DIJOYSTATE joy_state; // contains state of joystick

04/19/23 35

Game_Init( )

// initialize directdrawDDraw_Init(SCREEN_WIDTH, SCREEN_HEIGHT, SCREEN_BPP);// first create the direct input objectif (DirectInput8Create(main_instance,DIRECTINPUT_VERSION, IID_IDirectInput8, (void **)&lpdi,NULL)!=DI_OK) return(0);// create a keyboard device //////////////////////////////////if (lpdi->CreateDevice(GUID_SysKeyboard, &lpdikey, NULL)!=DI_OK)

return(0);// set cooperation levelif (lpdikey->SetCooperativeLevel(main_window_handle, DISCL_NONEXCLUSIVE | DISCL_BACKGROUND)!=DI_OK) return(0);// set data formatif (lpdikey->SetDataFormat(&c_dfDIKeyboard)!=DI_OK) return(0);// acquire the keyboardif (lpdikey->Acquire()!=DI_OK) return(0);

04/19/23 36

Game_Shutdown( )

// kill the reactor

Destroy_Bitmap(&reactor);

// kill skelaton

Destroy_BOB(&skelaton);

// release keyboard

lpdikey->Unacquire();

lpdikey->Release();

lpdi->Release();

// shutdonw directdraw

DDraw_Shutdown();

04/19/23 37

Game_Main( )

// get player input

// get the keyboard data

lpdikey->GetDeviceState(256, (LPVOID)keyboard_state);

// reset motion flag

player_moving = 0;

// test direction of motion

if (keyboard_state[DIK_RIGHT] && keyboard_state[DIK_UP])

{

// move skelaton

skelaton.x += 2;

skelaton.y -= 2;

dx=2; dy =- 2;

// set motion flag

player_moving = 1;

// check animation needs to change

if (skelaton.curr_animation != SKELATON_NEAST)

Set_Animation_BOB(&skelaton,SKELATON_NEAST);

} // end if

…

04/19/23 38

Mouse

• Detects changes in X, y position (Mickeys)• Specifies “absolute” position• Can specify velcoity range• Good for selecting objects, moving objects, or

changing head position (view)• Hard to center if used as joystick replacement• Has small number of buttons (2 or 3

04/19/23 39

LaMothe Example

04/19/23 40

Mouse Global Declarations

#define BUTTON_SPRAY 0 // defines for each button#define BUTTON_PENCIL 1#define BUTTON_ERASE 2#define BUTTON_EXIT 3

// directinput globalsLPDIRECTINPUT8 lpdi = NULL; // dinput objectLPDIRECTINPUTDEVICE8 lpdikey = NULL; // dinput keyboardLPDIRECTINPUTDEVICE8 lpdimouse = NULL; // dinput mouseLPDIRECTINPUTDEVICE8 lpdijoy = NULL; // dinput joystick GUID joystickGUID; // guid for main joystickchar joyname[80]; // name of joystick // these contain the target records for all di input packetsUCHAR keyboard_state[256]; // contains keyboard state tableDIMOUSESTATE mouse_state; // contains state of mouseDIJOYSTATE joy_state; // contains state of joystick

04/19/23 41

Game_Init( )

// first create the direct input object

DirectInput8Create(main_instance,DIRECTINPUT_VERSION,IID_IDirectInput8, (void **)&lpdi,NULL);

// create a mouse device /////////////////////////////////////

lpdi->CreateDevice(GUID_SysMouse, &lpdimouse, NULL);

// set cooperation level

lpdimouse->SetCooperativeLevel(main_window_handle, DISCL_NONEXCLUSIVE | DISCL_BACKGROUND);

// set data format

lpdimouse->SetDataFormat(&c_dfDIMouse);

// acquire the mouse

lpdimouse->Acquire();

04/19/23 42

Game_Init( )

// set the global mouse positionmouse_x = screen_height/2;mouse_y = screen_height/2; // load the master bitmap in with all the graphicsLoad_Bitmap_File(&bitmap8bit, "PAINT.BMP");Set_Palette(bitmap8bit.palette); // make sure all the surfaces are clean before startingDDraw_Fill_Surface(lpddsback, 0);DDraw_Fill_Surface(lpddsprimary, 0); // create the pointer bobCreate_BOB(&pointer,mouse_x,mouse_y,32,34,1, BOB_ATTR_VISIBLE |

BOB_ATTR_SINGLE_FRAME,DDSCAPS_SYSTEMMEMORY);  // load the image for the pointer inLoad_Frame_BOB(&pointer,&bitmap8bit,0,0,2,BITMAP_EXTRACT_MODE_CELL);

04/19/23 43

Game_Init( )

// set clipping rectangle to screen extents so mouse cursor

// doens't mess up at edges

RECT screen_rect = {0,0,screen_width,screen_height};

lpddclipper = DDraw_Attach_Clipper(lpddsback,1,&screen_rect);

// hide the mouse

ShowCursor(FALSE);

04/19/23 44

Game_Shutdown( )

// first unacquire the mouse

lpdimouse->Unacquire();

// now release the mouse

lpdimouse->Release();

// release directinput

lpdi->Release();

04/19/23 45

Game_Main( )

// move the mouse cursormouse_x+=(mouse_state.lX);mouse_y+=(mouse_state.lY); // test bounds // first x boundariesif (mouse_x >= screen_width) mouse_x = screen_width-1;elseif (mouse_x < 0) mouse_x = 0; // now the y boundariesif (mouse_y >= screen_height) mouse_y= screen_height-1;elseif (mouse_y < 0) mouse_y = 0;

04/19/23 46

Game_Main( )

// position the pointer bob to the mouse coords

pointer.x = mouse_x - 16;

pointer.y = mouse_y - 16;

// test what the user is doing with the mouse

if ((mouse_x > 3) && (mouse_x < 500-3) &&

(mouse_y > 3) && (mouse_y < SCREEN_HEIGHT-3))

{

// mouse is within canvas region

// if left button is down then draw

if (mouse_state.rgbButtons[0])

{

…

};

04/19/23 47

Game Pads

• Simplified keyboard• Buttons have good stimulus/response

compatibility• Lets you use two hand• Good for multiplayer games• Can require weird combinations of buttons• Don’t get to rely on fastest finger (index)• Doesn’t support variable input

04/19/23 48

Joysticks

• Lots of devices look like joysticks(digital – 8 positions, analog x,y deflection)

• Allow variable input plus buttons• Great for specifying changes• Can be used to specify “rate in a direction”• Not good for specifying absolute positions (need

to “hold position steady)• Not good for many arcade games

04/19/23 49

Using Joysticks & Game Pads

1. Scan for all devices and record GUID’s

2. Create joystick device using one GUID

3. Use interface from step 2 and create another interface (release the first)

4. Set cooperation level

5. Set data format and properties

6. Acquire joystick

7. Read joystick using Poll and GetDevice

04/19/23 50

LaMothe Example

04/19/23 51

Joystick Global Declarations

// directinput globals

LPDIRECTINPUT8 lpdi = NULL; // dinput object

LPDIRECTINPUTDEVICE8 lpdikey = NULL; // dinput keyboard

LPDIRECTINPUTDEVICE8 lpdimouse = NULL; // dinput mouse

LPDIRECTINPUTDEVICE8 lpdijoy = NULL; // dinput joystick

GUID joystickGUID; // guid for main joystick

char joyname[80]; // name of joystick

// these contain the target records for all di input packets

UCHAR keyboard_state[256]; // contains keyboard state table

DIMOUSESTATE mouse_state; // contains state of mouse

DIJOYSTATE joy_state; // contains state of joystick

04/19/23 52

DI_Enum_Joysticks( )

// this function enumerates the joysticks, but

// stops at the first one and returns the

// instance guid of it, so we can create it

*(GUID*)guid_ptr = lpddi->guidInstance;

// copy name into global

strcpy(joyname, (char *)lpddi->tszProductName);

// stop enumeration after one iteration

return(DIENUM_STOP);

04/19/23 53

Game_Init( )

// joystick creation section ////////////////////////////////

// first create the direct input object

if (DirectInput8Create(main_instance,DIRECTINPUT_VERSION, IID_IDirectInput8, (void **)&lpdi,NULL)!=DI_OK)

return(0);

// first find the f***ing GUID of your particular joystick

lpdi->EnumDevices(DI8DEVCLASS_GAMECTRL, DI_Enum_Joysticks,

&joystickGUID, DIEDFL_ATTACHEDONLY);

if (lpdi->CreateDevice(joystickGUID, &lpdijoy, NULL)!=DI_OK)

return(0);

// set cooperation level

if (lpdijoy->SetCooperativeLevel(main_window_handle,

DISCL_NONEXCLUSIVE | DISCL_BACKGROUND)!=DI_OK)

return(0);

04/19/23 54

Game_Init( )

// set data formatif (lpdijoy->SetDataFormat(&c_dfDIJoystick)!=DI_OK) return(0); // set the range of the joystickDIPROPRANGE joy_axis_range; // first x axisjoy_axis_range.lMin = -24;joy_axis_range.lMax = 24;

joy_axis_range.diph.dwSize = sizeof(DIPROPRANGE); joy_axis_range.diph.dwHeaderSize = sizeof(DIPROPHEADER); joy_axis_range.diph.dwObj = DIJOFS_X;joy_axis_range.diph.dwHow = DIPH_BYOFFSET;

lpdijoy->SetProperty(DIPROP_RANGE,&joy_axis_range.diph);

04/19/23 55

Game_Init( )

// now y-axisjoy_axis_range.lMin = -24;joy_axis_range.lMax = 24;joy_axis_range.diph.dwSize = sizeof(DIPROPRANGE); joy_axis_range.diph.dwHeaderSize = sizeof(DIPROPHEADER); joy_axis_range.diph.dwObj = DIJOFS_Y;joy_axis_range.diph.dwHow = DIPH_BYOFFSET; lpdijoy->SetProperty(DIPROP_RANGE,&joy_axis_range.diph); // and now the dead band (zone where movements are ignored)DIPROPDWORD dead_band; // here's our property word dead_band.diph.dwSize = sizeof(dead_band);dead_band.diph.dwHeaderSize = sizeof(dead_band.diph);dead_band.diph.dwObj = DIJOFS_X;dead_band.diph.dwHow = DIPH_BYOFFSET;

04/19/23 56

Game_Init( )

// 10% will be used on both sides of the range +/-dead_band.dwData = 1000; // finally set the propertylpdijoy->SetProperty(DIPROP_DEADZONE,&dead_band.diph); dead_band.diph.dwSize = sizeof(dead_band);dead_band.diph.dwHeaderSize = sizeof(dead_band.diph);dead_band.diph.dwObj = DIJOFS_Y;dead_band.diph.dwHow = DIPH_BYOFFSET;// 10% will be used on both sides of the range +/-dead_band.dwData = 1000; // finally set the propertylpdijoy->SetProperty(DIPROP_DEADZONE,&dead_band.diph);// acquire the joystickif (lpdijoy->Acquire()!=DI_OK) return(0);

04/19/23 57

Game_Shutdown( )

// release joystick

lpdijoy->Unacquire();

lpdijoy->Release();

lpdi->Release();

// shutdonw directdraw

DDraw_Shutdown();

04/19/23 58

Game_Main( )

// get joystick datalpdijoy->Poll(); // this is needed for joysticks onlylpdijoy->GetDeviceState(sizeof(DIJOYSTATE), (LPVOID)&joy_state); // lock the back bufferDDraw_Lock_Back_Surface(); // draw the background reactor imageDraw_Bitmap(&playfield, back_buffer, back_lpitch, 0); // unlock the back bufferDDraw_Unlock_Back_Surface(); // is the player moving?blaster.x+=joy_state.lX;blaster.y+=joy_state.lY;

04/19/23 59

Game_Main( )

// is player firing?

if (joy_state.rgbButtons[0])

Start_Missile();

// display joystick and buttons 0-7

sprintf(buffer,"Joystick Stats: X-Axis=%d, Y-Axis=%d, buttons(%d,%d,%d,%d,%d,%d,%d,%d)", joy_state.lX,joy_state.lY,

joy_state.rgbButtons[0], joy_state.rgbButtons[1], joy_state.rgbButtons[2], joy_state.rgbButtons[3],

joy_state.rgbButtons[4], joy_state.rgbButtons[5],

joy_state.rgbButtons[6], joy_state.rgbButtons[7]);

// print out name of joystick

sprintf(buffer, "Joystick Name & Vendor: %s",joyname);

Draw_Text_GDI(buffer,0,SCREEN_HEIGHT-40,RGB(255,255,50),lpddsback);

04/19/23 60

Force Feedback

• Gameplay can influence input device by producing some resistive force

• Requires bi-directional communication between interaction device and computer(e.g. joysticks, steering wheels, yokes, etc.)

• Examples– Engine-hum (sine-wave)– Machine gun (square-wave vibrations)

04/19/23 61

Force Feedback

• Lags > 25ms between visual event and feedback are noticable

• Designers need to anticipate effects of conditions (e.g. wind) and surface texture (e.g. bumps and grit) in planning game play

• Can be mesmerizing to players (adds to realism)

• LaMothe has example, can’t run on my laptop (my FFB device is not USB)