Human Interaction with Robots Working in Complex … · Human Interaction with Robots Working in Complex and Hazardous Environments ... TR Control Computer assisted teleops

Human Interaction with RobotsWorking in Complex and Hazardous

Environments

Bill Hamel, Professor & HeadIEEE FellowRAS Vice President for Publication Activities

Mechanical, Aerospace, & BiomedicalEngineering Department

University of Tennessee

Colloquium on Robotics and AutomationUniversita` degli Studi di Napoli Federico II

Napoli 18 December 2006 2

Human Interaction with Robots Working inComplex and Hazardous Environments

Preliminaries Complex and Hazardous? Robots? �Human Interaction? Needs for Performance Improvements?

Modern Telerobot? System Concept & Architecture Where should research foci be?

Current Research…Telerobotics Experimental System Results Future Directions

Closing Remarks



Complex and Hazardous? Remote operations in

hazardous and uncertainenvironments…unstructured Nuclear, Space, Underwater,

Medical Surgery… Remote work systems that

involve: Remote sensing Manipulators Mobility Use of tools

Human-in-the-loopoperations/control



Robot?…Telerobotics Telerobots?

Telerobot = Teleoperator U Robot tR = Robot U Teleoperator Tr = Teleoperator U Robot tR ≠ Tr Manual control U Autonomous

operations Human interaction Computer assisted

teleoperation Selective autonomy



teleaction

telepresence

Physical/Temporal/HazardsBarriers

Tasks

Manipulators

MobileTransporter

Power

Tools

Human/MachineInterface

Sensors Signals

Human Interaction…



Decades of Refinements…



Needs… Reality:

Too expensive Too slow Too complicated

Improve remote workefficiency Best: direct/teleaction ≥ 10 Worst: direct/teleaction ~

100’s Open engineering

Power supply andtransmission; cable handling

3D remote viewing andsensing



Telerobot Functional Architecture

Mobility

Manipulation

Tooling

In Situ ModelBuilder

Interactive TaskPlanner

CooperativeAssists Planner

Control

HMIAutonomousOperations

HMITeleoperations

Fault Detection& Recovery

Task Space

RTSATR Control

Computer assisted teleops



Critical Performance Issues

Manual teleoperation

REMOTE OPERATIONSTASK SEQUENCE

Subtaski

manual

Subtaski+1 Subtaski+2

automatic

Subtaski+3 Subtaski+4

manual

Build in situ

3D modelPlan

subtaskExecute

automaticallyVerifyresults

Select and configureassistance strategy

Execute manuallywith assistance

HUMAN-MACHINE COOPERATIVE TELEROBOTIC PROCESS

Manual teleoperation

ROBOT TASK SCENE ANALYZER PROCESS

Seamless Transfer Essential



Telerobot Control StructureSystem ON

IDLE

MANUAL_TELEOP AUTOEXEC

ESTOP

ERROR ERROR

OK OK

d

dtJu

!

!�!

K m

B m

Bs-1

�!Bs

-1Ks

!!

+

+

+

+

+

+

+

+

-

-

-

Ju-1

�

Cartesian Controller

TrajectoryGenerator

RTSA3D Task Geometry

& Action Points

Task Planner& Assist Functions



Research Test Bed

Dual remote manipulators…

host computer(LINUX)

Schilling Titan IImanipulator

pan-tilt stereosensor head

laser rangecamera

D&Dmockup

mastercontroller

developmentcomputer

(Windows NT)

image processingcomputer

(Windows NT)

camera controlhardware

manipulatorcommunication

hardware

LOCAL AREA NETWORK



Human Interactive Task Space Modeling andPlanning

Max use of a prioriknowledgeParallel Execution

Manual Foreground.AutonomousBackground.

Operator ControlWhere, what, andhow.Acceptance of results.

Simplified GUI5 pop-up windowmenusColor overlays

ROI1

ROI2

ROI3

ROI5

ROI4

User points

camera

OOIs

OOIs

OOIs

ROI4

ROI2

ROI1

Confirm

window

Okay?

No

Yes

Place

part

QueueSelect

ROI

PANORAMIC VIEW

AUTOSCAN

(BACKGROUND)

MANUAL

(FOREGROUND)



Laser pointing designatorLaser pointing designator

± 10 mm in robotworkspace



REMOTE OPERATIONS TASK SEQUENCE

SUBTASKi:

Manual

SUBTASKi+1:

Auto

SUBTASKi+3:

Auto

SUBTASKi+2: Man-

ual

SUBTASKi+4:

Auto

RTSA

BUILD IN SITU

3D Model

PLAN SUB-

TASK

EXE-

CUTEVERIFY

RESULTS

TELEROBOTIC SUBTASK

SEQUENCE i+3

8-10 min 4 min 18 min

32 minversus 45-90

min Teleop



Recent activities… 7 dof “haptic” controller

Barrett TechnologiesWAM

Large-scale multi-fingered end effector Barrett Technologies

Wraptor Reduced tool

specializations Controller integration

Tooling:behavior-basedgrasping

Test & evaluation



Continuing Research

Abort Plan Sequence

Original Plan Sequence

Re-Plan Sequence

Return to original

Retrace/restart

original

Start re-plan

Fault state/step

“Drift” state/step

subtasks

Mobility

Manipulation

Tooling

In Situ ModelBuilder

Interactive TaskPlanner

CooperativeAssists Planner

Control

HMIAutonomousOperations

HMITeleoperations

Fault Detection& Recovery

Task Space

RTSA

TR Control

Computer assisted teleops

Tool-based Control

OperationalSpace FaultDetection Intelligent/interactive

State Management



Higher-fidelity telepresence andteleaction…needs, directions… Perception/Sensing

3D viewing 3D graphics augmentation Natural kinesthetic/tactile feedback

Intelligence/Control Human relationships

• Interactivity reduced through increasing autonomy• More cooperative• Humans move up in hierarchy

Machine learning• By observation – symbiosis

Action/Manipulation Multi-fingered end effectors

Human Machine Interface Algorithmic complexity VR-based interfaces



Summary “True” Telerobots are being realized. There is hope for improving the

effectiveness of remote operations incomplex environments.

Ongoing research will continue to drivethis trend.

Thank you for your attention!Thank you for your attention!

Human Interaction with Robots Working in Complex … · Human Interaction with Robots Working in Complex and Hazardous Environments ... TR Control Computer assisted teleops

Documents