Top Banner
GESTURAL COMMUNICATIONS WITH ACCELEROMETER-BASED INPUT DEVICES AND MULTI-MODAL DISPLAYS ul D. Varcholik TIVE Laboratory stitute for Simulation and Training iversity of Central Florida [email protected] James L. Merlo LTC, US Army US Military Academy West Point [email protected]
15

Paul D. Varcholik ACTIVE Laboratory Institute for Simulation and Training University of Central Florida [email protected] James L. Merlo LTC, US Army.

Dec 16, 2015

Download

Documents

Nickolas Black
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: Paul D. Varcholik ACTIVE Laboratory Institute for Simulation and Training University of Central Florida pvarchol@ist.ucf.edu James L. Merlo LTC, US Army.

GESTURAL COMMUNICATIONS WITH ACCELEROMETER-BASED INPUT DEVICES AND MULTI-MODAL DISPLAYS

Paul D. VarcholikACTIVE LaboratoryInstitute for Simulation and TrainingUniversity of Central [email protected]

James L. MerloLTC, US ArmyUS Military AcademyWest [email protected]

Page 2: Paul D. Varcholik ACTIVE Laboratory Institute for Simulation and Training University of Central Florida pvarchol@ist.ucf.edu James L. Merlo LTC, US Army.

Gestural Communication

Page 3: Paul D. Varcholik ACTIVE Laboratory Institute for Simulation and Training University of Central Florida pvarchol@ist.ucf.edu James L. Merlo LTC, US Army.

Problem Statement Reliable communications between

military personnel is critical

Hand gestures are used when vocal means are inadequate

Line of sight issues may cause visual signals to be unreliable

Page 4: Paul D. Varcholik ACTIVE Laboratory Institute for Simulation and Training University of Central Florida pvarchol@ist.ucf.edu James L. Merlo LTC, US Army.

Our Research Purpose

To determine the usefulness of a computer mediated gesturing recognition system for non-visual communication

ScopeProvide a proof of concept which would lay the groundwork for future research

Research Questions Have we developed a recognition system capable of

accurately converting and transmitting a visual communication mode into a non-visual form?

Do computer-mediated gestures provide a viable form of non-visual communication?

Page 5: Paul D. Varcholik ACTIVE Laboratory Institute for Simulation and Training University of Central Florida pvarchol@ist.ucf.edu James L. Merlo LTC, US Army.

Input Device

Nintendo Wiimote

Nintendo Wiimote 3-axis accelerometer Wireless (Bluetooth) Inexpensive COTS 100Hz Sampling Rate

Page 6: Paul D. Varcholik ACTIVE Laboratory Institute for Simulation and Training University of Central Florida pvarchol@ist.ucf.edu James L. Merlo LTC, US Army.

Output Devices

Tactile Belt

Auditory (headphones) Tactile Display

Wireless (Bluetooth) 1.2 lbs (w/o battery) Elastic belt 8 tactors at 45-degree

increments

Page 7: Paul D. Varcholik ACTIVE Laboratory Institute for Simulation and Training University of Central Florida pvarchol@ist.ucf.edu James L. Merlo LTC, US Army.

Tactile Patterns Emulating Standard Army Hand Signals (FM 21-60)

TACTONS

Page 8: Paul D. Varcholik ACTIVE Laboratory Institute for Simulation and Training University of Central Florida pvarchol@ist.ucf.edu James L. Merlo LTC, US Army.

Gesture Recognition Machine Learning Algorithms (3 implemented for

evaluation) Linear Classifier AdaBoost Artificial Neural Network (evolved w/ NEAT)

29 Features Based on work by Rubine (1991) on 2D symbol

recognition Example features:

○ Bounding Volume Length○ Min, Max, Median, Mean (X, Y, Z)○ Starting Angle, Total Angle Traversed, Total Gesture Distance

Page 9: Paul D. Varcholik ACTIVE Laboratory Institute for Simulation and Training University of Central Florida pvarchol@ist.ucf.edu James L. Merlo LTC, US Army.

Training & Visualization UI Arbitrary gesture set Left-hand, right-hand,

both-hands 3D animated soldier Text label display Sound display Tactile display Wiimote visualization Data serialization UI Independent of

recognition API

Page 10: Paul D. Varcholik ACTIVE Laboratory Institute for Simulation and Training University of Central Florida pvarchol@ist.ucf.edu James L. Merlo LTC, US Army.

Experiments & Results

Several experiments run to dateDifferent algorithms and gesture setsAccuracy > 94%Classification time < 10ms / gestureLinear classifier best performer (for training

time and classification considered together)AdaBoost (highest accuracy, but slower

training time than linear classifier)ANN w/ NEAT (worst performer – requires

more training data)

Page 11: Paul D. Varcholik ACTIVE Laboratory Institute for Simulation and Training University of Central Florida pvarchol@ist.ucf.edu James L. Merlo LTC, US Army.

Discussion Proved Concept

System capable of accurately converting and transmitting a visual communication mode into a non-visual form.

Wiimote is a convenient and inexpensive device for experimentation. Technology transfers to more robust hardware (e.g. instrumented glove).

Wiimote produces some ambiguous data (e.g. static poses). Additional attachment (e.g. gyroscopes) required for more accuracy.

Experiments indicate promising form of communication – more experiments are needed.

Page 12: Paul D. Varcholik ACTIVE Laboratory Institute for Simulation and Training University of Central Florida pvarchol@ist.ucf.edu James L. Merlo LTC, US Army.

Future Work Determine the maximum number of

gestures that can be accurately recognized Gesture rejection Dynamic mapping between gesture, sound,

and tactile sequence Scenario development for realistic

experimentation (establishing context) Transmitting signal data via RF (currently

sent to local device or via UDP/IP)

Page 13: Paul D. Varcholik ACTIVE Laboratory Institute for Simulation and Training University of Central Florida pvarchol@ist.ucf.edu James L. Merlo LTC, US Army.

USMA Collaboration

CDT Robert Darket, CDT Zachary Schaeffer (Principal Investigators)

Application: TrainingCollect exemplar gestures from SMEsValidate less-experienced soldier’s gestures

against exemplars

Page 14: Paul D. Varcholik ACTIVE Laboratory Institute for Simulation and Training University of Central Florida pvarchol@ist.ucf.edu James L. Merlo LTC, US Army.

Video Demonstration

Page 15: Paul D. Varcholik ACTIVE Laboratory Institute for Simulation and Training University of Central Florida pvarchol@ist.ucf.edu James L. Merlo LTC, US Army.

Questions?

Gestural communications with accelerometer-based input devices and multi-modal displaysPaul D. Varcholik

ACTIVE LaboratoryInstitute for Simulation and TrainingUniversity of Central [email protected]

James L. MerloLTC, US ArmyUS Military AcademyWest [email protected]