Outline

Omni-directional Vision and 3D Robot Animation Based Teleoperation of Hydraulically Actuated Hexpod Robot

COMET-IV

Hiroshi OHROKU

Control and Robotics Lab.Control and Robotics Lab.CHIBA UniversityCHIBA University

Background

http://www.yomiuri.co.jp/feature/20080614-2892868/news/20080616-OYT1T00506.htm

Assistance during hazardous operations

• Disaster-relief work• Construction• Mine detection and clearance etc.

http://headlines.yahoo.co.jp/hl

The Expectation of robots is increasing!

Background

Utility of multi-legged robot

• Disaster site• Construction site• Mine field

Rough terrain

• Multi-legged robots have high stability and mobility in rough terrain.

Sites of hazardous operations

• Multi-legged robots can enter areas where it is difficult for wheeled robots and crawler type robots to enter.

Development of COMET-IV

COMET-III

COMET-IV

COMET-II

COMET-I

1.1. Fully autonomous locomotion on outdoor rough terrainFully autonomous locomotion on outdoor rough terrain

2.2. Assistance during various hazardous operations

Development goalDevelopment goal

Large-scaled legged Robots (Past)Large-scaled legged Robots (Past)

Walking Forest Machine

MECANT

ASV

TITAN-XI

About autonomous system

Fully autonomous system Half autonomous system

＜ Intelligent system ＞　 - Task management & Planning　 - Environment recognition　 - Self-Localization

Upper-layer system

＜ Robust controller ＞　 - Force & Position control

Lower-layer system

＜ Robust controller ＞　 - Force & Position control

Lower-layer system

　 - Image of real environment　 - Sensor data

Teleoperation system

　 - Operation　 - Supervision

It is very difficult to implement fully autonomous robot in the stricken area ・・・

Tele-robotics researches(1)

http://pc.watch.impress.co.jp/docs/2002/1219/hrp.htm

Master-slave system+

3D Animation+

Open Dynamic Engine （ ODE ）

http://const.tokyu.com/topics/1998/topics_03.html

Master-slave system+

ＦＥＳ（ Functional Electrical Stimulation ）

based bilateral control

Tele-robotics researches(2)

K.Saitoh, T.Machida, K.Kiyokawa, H.Takemura : A Mobile Robot Control Interface Using Omnidirectional Images and 3D Geometric Models, Technical report of IEICE. Multimedia and virtual environment 105-256, 7/12(2005)

Focus of my researchThe research on teleoperation system to assist operator for

controlled large-scale legged type robot is little

In rough terrain operation, the change of ・・・・ Body height

Attitude

Leg movement

The teleoperation assistant system is indispensable for controlling legged robot in outdoor environment

Development of theTeleoperation assistant system

■Omni-directional vision

■3D Animation of robot movement

Development TasksDevelopment Tasks Autonomous Navigation SystemAutonomous Navigation System

Teleoperation Assistant SystemTeleoperation Assistant System

1. Walking speed: max 1 km/h

2. Vertical step: max 1 m

Design specification in locomotion

3. Gradient: max 20 deg

4. Omni-directional walking

Control SystemControl System

• Gait Planning

• Foot Trajectory Tracking

• Force Control

• Attitude Control

Hardware SpecificationsHardware Specifications

L×W×H 2.5×3.3×2.8 （ m ）

Weight 2200 （ kg ）

Power Source 　(Max. Output)

Gasoline Engine ×2 　(25.0ps/3600rpm)

Supply Pressure 22 （ Mpa ）

Supply Flow 78 × 2 　（ L/min ）

Leg 1

Leg 2

Leg 3

Leg 4

Leg 5

Leg 6

W:3.3 m

H:2.8 m

L:2.5 m

Hardware SpecificationsHardware Specifications

LRF（ Laser Range Finder ）

Stereo vision

Omni-directionalVision sensor

Coordinate System of Leg

Link Length (m) Range of Motion [deg]

Shoulder L1 0 -180° θ≦ 1 180°≦

Thigh L2 1.13 50° θ≦ 2 142°≦

Shank L3 0.77 36.6° θ≦ 3 150°≦

Foot L4 0.39 -47.9° θ≦ 4 103°≦

θ 2

θ 3

θ 4

θ 1

Z

YX

L2

L3

L4

L1

Basic configuration of hydraulic control system

COMET-IV

LRF : Laser Range Finder, SVC : Stereo Vision Camera, ODC : Omni-Directional CameraWCM : Wireless Client Module, WM : Wireless Modem (for sensor data), AP : Access Point

Foot

Thigh

ShankShoulder

: Potentiometer: Pressure Sensor

× 6Hydraulic Cylinders

HydraulicMotor

Proportional Solenoid Valve

Valve Controller

D/A Board

Locomotion Computer

Navigation Computer

LRF SVC ODC GPS

WMWCM

TeleoperationComputer

Wireless NetworkA/D

Board

Attitude Sensor

Azimuth Sensor

WMAP

COMET-IV


Foot

Thigh

ShankShoulder



HydraulicMotor

Foot

Thigh

ShankShoulder



HydraulicMotor


Valve Controller

D/A Board

Locomotion Computer

Navigation Computer

LRF SVC ODC GPS

WMWCM


Wireless NetworkWireless NetworkA/D

Board

Attitude Sensor

Azimuth Sensor

WMAP

COMET-IV


Foot

Thigh

ShankShoulder



HydraulicMotor

Foot

Thigh

ShankShoulder



HydraulicMotor


Valve Controller

D/A Board

Locomotion Computer

Navigation Computer

LRF SVC ODC GPS

WMWCM



Board

Attitude Sensor

Azimuth Sensor

WMAP

COMET-IV


Foot

Thigh

ShankShoulder



HydraulicMotor

Foot

Thigh

ShankShoulder



HydraulicMotor


Valve Controller

D/A Board

Locomotion Computer

Navigation Computer

LRF SVC ODC GPS

WMWCM



Board

Attitude Sensor

Azimuth Sensor

WMAP

Omni-directional gait

In this system, we applied the trajectory ・・・

“The Standard Circular Gait” [ ＊ ]

“Low impact foot trajectory”[ ＊＊ ]

If it is possible to change the body trajectory arbitrarily,we can apply various navigation strategies !!

Strategy

The ability to change the body trajectory arbitrarily at any step during walking is needed.

+

[ ＊ ]S.Hirose, H.Kikuchi, Y.Umetani: 　 The Standard Circular Gait of the Quadruped Walking Vehicle, 　 Journal of Robotics Society of Japan, 2-6, 41/52 (1984)

[ ＊＊ ]Y.Sakakibara, K.Kan, Y.Hosoda, M.Hattori, M.Fujie :　　 Low Impact Foot Trajectory for aQuadruped Walking Machine, 　　 Journal of the Robotics Society of Japan, 8-6,22/31(1990)

Omni-directional gait

Circular gait

Rotation center

Crab gait

Crab gait

Circular gait

Circular gait includes all gaitCircular gait includes all gait[*][*]

Definition of coordinate system

Xc

Yc

Xs_1

Ys_1

<FRONT>

<REAR>

<L

EF

T>

<R

IGH

T>

Leg1

Leg2

Leg3

Leg4

Leg5

Leg6

Os_1, Zs_1

Oc, Zc

Xt

Yt

Ot

Rotation center coordinate system

Body coordinate system

Shouldercoordinate

system

RRctct

θθctct

： Crab angle

Condition of circular angle setting

btbct rR )cos1(2 btbct rR )cos1(2

2

ct

-2 -1 0 1 2-2.5

-2

-1.5

-1

-0.5

0

0.5

1

1.5

2

2.5

X [m]

Y [m

]

dL

rb

-2 -1 0 1 2-2.5

-2

-1.5

-1

-0.5

0

0.5

1

1.5

2

2.5

X [m]

Y [m

]

dL

rb

Condition:

Arbitrary crab angle walking is achieved by setting as follows.

　→ ∞

Circular gait ≒ Crab gait

ctR 2

ct

Teleoperation assistant system

Instruction information

Log

Omni-directional image

Generated image

Receivedsensor data

Map

3D Animation of robot

Peripheral Device

Interface USB1.1

Size （ mm ）

210 × 199 × 240

Buttons 11

Axes 3

■Omni-directional vision sensor

　　 - Digital Video Camera ： DCR-HC48 （ SONY ）　　 - Hyperbolic Mirror

■Joystick: Cyborg Evo Force （ Saitek ）

Diameter of mirror 82 mm

Angle of elevation 15°

Angle of depression 50°

Instruction information

Action Action value

Cycle_Time Cycle time

θtb Traverse angle of the body center at one cycle [Ot]

Rct Position of rotation center [polar display of Oc]

θct

Action value shows the basic movement action.　■ Stand up

　■ Sit down

　■ Walking start

　■ Walking stop


LocomotionComputer

UDP Socket（ User Datagram Protocol ）

Coordinate system of JoystickZ

zrot _

Z

zrot _X

Y

X

Y

X

Y

X

Y

The eight directions of basic locomotion angle β can be selected using the joystick.

Gait parameters Setting

10125.0

125.010

tb

tb

2

ct2

tbctR

cos12

1

The range of the traverse angle

＜ Crab gait ＞＜ Circular gait ＞

　→ ∞ctR

is set to the minimal valuetb The circular angle of body center derived from rot_z which is set to tb

ct

10125.0

125.010

tb

tb

2

Configuration of omni-directional vision sensor

P（X,Y,Z）

Z

Y

X

OM

Mirror focus

OC Center of camera lens

O

x

y

p（x,y）Image plane

c

b

f

a 26.2 [mm]

b 35.4 [mm]

c 44.1 [mm]

Diameter of mirror 55.0 [mm]

Angle of elevation 15.0 [deg]

Angle of depression 50.0 [deg]

Performance of Ambient Environmental Image(1)

180/

180/

tilt

pan

i

i

y

x

fp

vz

vy

vx

cos

sinsin

sincos

0

cos

sin

sin

cossin

coscos

Central axis of a mirror fits an optical axis of a camera is assumed. The direction where the angle of pan and tilt is set to 0 degree to make X axisand the origin of projection center coordination is (0, 0).

（ 7 ）

The spherical coordinates was derived by using pan, tilt angle and focal length which is direction projection plane as against the sphere.

（ 8 ）

222iiii vzvyvxr （ 9 ）

i

iis

i

iis

r

vz

vx

vy

arctan

arctan

_

_

（ 10 ）


The coordinates that derived from Eq. (7) to Eq. (10) is transformed to rectangular coordinates.

isis

isisis

isisis

z

y

x

__

___

___

cos

sinsin

sincos

（ 11 ）

Mirror parameter is in Semi-major axis and mirror parameter b is in Semi-minor axis. Semi-latus rectum is derived by Eq. (12).

a

b2

（ 12 ）

2

2

1b

a （ 13 ）


Parameters l and m is used in spherical projection was derived from Eq. (14) to Eq. (15) including.

21

2

l （ 14 ）

21

22

m （ 15 ）

lz

lms

（ 16 ）

On the other hand, projection coordinates of hyperboloidal is calculated as Eq. (16) and (17).

isi

isi

yshy

xshx

_

_

*

*

（ 17 ）


Finally, the pixel coordinate system is determined by using matrix K via Eq. (18). Furthermore projection coordinates is calculated by Eq. (19). The values (358.1, 215.1) in the matrix K are XY coordinates of the center point in omni-directional image.

100

1.2154.7880

6.35805.884

K （ 18 ）

11i

i

i

i

hy

hx

Kpy

px

（ 19 ）

Texture mappingTime consuming factor on the system processing unit still become an issues because previous mentioned calculation method does similar process to all image pixels.

Consequentially, it is difficult to present the smooth image operator.

Therefore, the load of the calculation processing is reduced by using texture mapping.

Texture mapping

Fig.11-(a)

(0,1) (1,1)

(0,0) (1,0)X

Y

T[0]T[1]

T[2] T[3]

Fig.12

P[0]

(-100,100) (100,100)

(-100,-100) (100,-100)

X

Y

P[1]

P[3] P[2]

P[3]

(-1,1) (1,1)

(-1,-1) (1,-1)

X

Y

P[2]

P[0] P[1]

Fig.11-(b)

The corresponding points in omni-directional image as against lattice points were derived from Eq1 to Eq3.

②

①

Texture mapping

vertex[0] = P[0] texcoor[0] = T[0]vertex[1] = P[1] texcoor[1] = T[1]vertex[2] = P[2] texcoor[2] = T[2]vertex[3] = P[3] texcoor[3] = T[3]

③

Mapping image

Fig.12 Calculated points on texture coordinate system

Robot animation using 3D geometric models and sensor data

The 3D COMET-IV online model is designed to predict the real-time movement of COMET-IV on the reality environment.

Each sensor data is used for the robot 3D animation movement reference which is transmitted from target computer unit on COMET-IV to

teleoperation computer via wireless serial MODEM with data transfer rate 57600bps at 10Hz.

LocomotionComputer

Wireless serial Modem(for sensor data)

Wireless serial Modem(for sensor data)

2.4GHz

Coordinate system of 3D robot

Table.6 and Fig. 13 shows the parameters used for robot 3D animation in virtual environment and coordinate system of 3D robot respectively.

X

Z

Y

Roll

PitchYaw

Initialization of 3D Animation

For initialization, received azimuth angle, XY value of GPS, and roll and pitch angle of the body are set up as default values.

The robot is rotated using azimuth angle ψ including magnetic variation Δd with Y axis.

The homogeneous transformation

[1] Each leg (Foot - Shank - Thigh - Shoulder)

After initialization, each part is expressed as follows.

ifootifoot PTransZRotPTransT _4_1 , （ 20 ）

iskisk PTransZRotPTransT _3_2 ,90 （ 21 ）

ithith PTransZRotPTransT _2_3 ,90 （ 22 ）

ishoisho PTransYRotPTransT _1_4 , （ 23 ）

[2] Robot body

YRotXRotZRotT offset ,,,5 （ 24 ）

offsetoffset GXGYTransT ,0,6 （ 25 ）

Obstacle avoidance walking test

Operator

Goal

Obstacle(2)

Obstacle(1)

■Controller is PID position control

■Cycle time 16 [s]

■Omni-directional gait （ crab gait ）

■Tripod walking

Robot Setting:

■The number of lattices was set as 16×12

■virtual obstacle object on the screen

System Setting:

Experimental Results

Fig.15 Walking trajectory acquired by GPS Fig.16 Foot trajectory of walking test (Leg1)

Experimental Photos and Images(1)

110 s 205 s 365 s110 s 205 s 365 s

Fig.17 Photos of the obstacle avoidance walking test

Fig.18 Images of 3D animation

Experimental Photos and Images(2)

(a)’

(a)

(b)’

(b)

(c)’

(c)

Fig.19 Omni-directional images and generated images

Conclusion

• Application of omni-directional gait

• Implementation of Teleoperation assistant system

-Outdoor obstacle avoidance walking experiment indicates the effectiveness of proposed system.

-Teleoperation assistant system is that applied with the omni-directional vision sensor and robot 3D animation was successfully implemented on COMET-IV system.

However the problem when network entered the state of a high load in the video data transfer that influenced the communications between teleoperation computer and locomotion computer are still remained.

Future WorkFuture Work

• The network of the video data transfer and communications will be separated.

• We will improve the accuracy of self-localization and

apply the force control to COMET-IV in order to

realize steady walking in rough terrain.

Outline

Documents

ambient environmental image

3d geometric models

video data transfer

teleoperation assistant system

position control lower

standard circular gait

directional vision sensor

rough terrain