1 Vision Based Following of Locally Linear Structures using an Unmanned Aerial Vehicle Sivakumar Rathinam, Zu Whan Kim, Raja Sengupta Center for Collaborative.

1

Vision Based Following of Locally Linear Structures using an Unmanned Aerial Vehicle

Sivakumar Rathinam, Zu Whan Kim,

Raja Sengupta

Center for Collaborative Control of Unmanned VehiclesUniversity of California, Berkeley

2

Motivation

Aim: Enable UAV use for infrastructure monitoring• Traffic monitoring, aqueduct inspection, pipeline monitoring ….

3

Our Technology

Keep the vehicle over the structure using with vision in the loop

Complement GPS waypoint navigation• Waypoint navigation to get the vehicle over the

structure• Lock it on using vision for accurate imaging

4

Forest Fire Monitoring

Inter-Office Cargo Delivery

Motivation

Unmanned Aerial Vehicles for Traffic Surveillance • The Ohio Department of Transportation, The Florida Department of

Transportation, The Georgia Department of Transportation

Lane changes, Average inter-vehicle distances, Heavy vehicle counts, Accidents, Vehicle trajectories, Type of vehicles etc.

The road should be in view.

Coifman et. al, Surface Transportation Surveillance from Unmanned Aerial Vehicles

“The turning radius of the fixed wing UAV is such that changing directions at waypoints can take some time and space until the vehicle regains its course. When traversing roadway links of lengths less than 400 ft, large portions of the links went unobserved.”

5

Motivation

Hanshin Expressway, Japan 1995 Alaska pipeline

The visual feedback compensates GPS inaccuracies and tracks the structure even it is shifted from the assumed location.

6

Generalization: Vision Based Following of Locally Linear Structures(Closed Loop on the California Aqueduct, June 2005)

7

The average error of the position of the vehicle from the curve was 10 meters over a length of 700 meters of the canal.

Algorithm ran at 5 Hz

Results Tracking the California Aqueduct

8

Current UAV Platform Configuration

Wing-Mounted Camera allowing for vision-based control, surveillance, and obstacle avoidance

Ground-to-Air UHF Antenna for ground operator interface

GPS Antenna for navigation

802.11b Antenna for A-2-A comm.

Payload Tray for on-board computations and devices

Payload Switch Access Door for enabling / disabling on-board devices

9

Current Payload Configuration

Off-the-shelf PC-104 with custom Vibration Isolation

Orinoco 802.11b Card and Amplifier for A-2-A comm.

Analog Video Transmitter for surveillance purposes

Printed Circuit Board for Power and Signal Distribution among devices.

Umbilical Cord Mass Disconnect for single point attachment of electronics to aircraft.

Keyboard, Mouse, Monitor Mass Disconnect for access to PC-104 through trap door while on the ground.

10

Problem

Follow a given curved structure based on visual feedback.

Overhead View

11

Following a Structure using Visual Feedback

1. Structure detection

a. Learn the structure from a one example

b. Real time structure detection of the structure

c. Curve fitting

2. Tracking

a. Transformation of image to ground coordinates

b. Control the vehicle to follow the structure

Hardware in the loop setup and evaluation

Experiments

12

Basic Detection Idea

Locally linear: Structure should look approximately linear in each image

13

1a. Learning the Structure from One Example

Rectify image-Finding the vanishing point

14

1a. Learning the Structure from One Example

mean

variance

Road Template

•Mean intensity will show high variation at the boundary•The variance in between the boundary points should be low•Done off-line•Can be automated or manual

15

1b. Real Time Detection in each Image

Road Template

For every 4th horizontal scan line pick several boundary hypotheses-Each hypothesis is a pair of features (high local intensity gradient)-Score each hypothesis for match quality with learnt profile -Keep the best three hypotheses for each scan line

16

1b. Real Time Detection in each Image

Road Template

For every 4th horizontal scan line pick several boundary hypotheses-Each hypothesis is a pair of features (high local intensity gradient)-Score each hypothesis for match quality with learnt profile -Keep the best three hypotheses for each scan line

Corr(Ih’(p),L)

17

1c. Curve Fitting

Road Template

RANSAC for Curve Fitting

Pick four scan lines at randomand four center hypothesesi.e., one from each line

Fit a cubic splineScore the cubic spline

Pick the spline with the best score

Set of supporting scan linematches

18

Cal Road Detection on MLB Video(No Control)

Generic corridor detection by one-dimensional learning•Roads•Aqueducts•Perimeters•Pipelines•Power Lines

19

2a. Transforming Image to Ground Coordinates

Height is measured by the pressure sensors.

Use accelerometers and the gyros in the avionics package to calculate the transformation

• Roll and pitch

Internal calibration parameters

Z

Y

X

Coordinates

20

2b. Controlling the Vehicle to Follow the Structure

Find a connecting contour that joins the current position to the desired curve and follow that path

• Position and slope at the origin and the look ahead distance (Soatto 2000)

21

Literature review – Vision Based Road Following Systems

VITS (1988) • Tracked roads at 13 miles/hr

Dickmanns (1992)• Tracked roads in autobahn at speeds up to 62 miles/hr

Taylor et.al (1999)• Tracked roads at speeds up to 75 miles/hr

Eric Frew et.al (2003)• Unmanned Aerial Vehicle • Tracked roads at around 44 miles/hr

22

More dangerous stuff……Obstacle Avoidance

Experiment flown on a Sig Rascal airframe with a Piccolo avionics package and vision processing on an onboard PC104.

An 8.5 foot diameter balloon was used as the obstacle (distance currently calculated using GPS).

23

Flight Demonstration

24

Related Work

Vision-based obstacle avoidance has been studied primarily in the context of mobile ground robots.• Lenser ’03, Ohya ’00, Lorigo ‘97,

Vision based navigation of UAVs• Saripalli ’02, Shakernia ’02, Furst ’98 – Landing with known markings• Sinopoli ’01, Doherty ‘00 – Visual landmark navigation (terrain avoidance)

for helicopter• Ettinger ’02, Pipitone ’01, Kim ’03 – Pose estimation for aircraft

Obstacle/Collision Avoidance for UAVs• Mitchell ‘01 – Aircraft avoiding known aircraft• Sigurd ’03 – Aircraft with magnetic sensors• Sastry ‘03 – Helicopters avoiding known helicopters/obstacles• How ’02 – MILP for Obstacle Avoidance

Vision based obstacle avoidance• Barrows ’03 – Biomimetic reactive control

25

Related Research

Ground robots• Fixed baseline stereo – JPL, many others• Monocular map construction – Lenser (CMU), Kim (Berkeley)• Cooperative stereo - CMU

Optical Flow• Helicopter ground following – Srinivasan/Chahl (Australia)• Corridor following - USC helicopter• Micro UAV obstacle avoidance – Centeye

UAV depth map construction • Lidar – CMU Helicopter Project, Sastry (Berkeley Helicopter Project).• Vision + high precision IMU – Bhanu (joint with Honeywell)

Stereo Vision• GT Helicopter

26

Requires DepthTypically use Stereo Vision

Given the image coordinates of a feature in one image• if one can find the image coordinates of the feature in

the other image (feature matching), and• if one knows the rotation and translation of the two

image planes then one knows the world coordinates of the feature (Ego-motion Estimation)

27

Problem with Depth Estimation by Stereo Vision

ZZ+ Z-0

z

Increased accuracy requires increased camera separation

28

Accurate Depth Estimation is a Problem

Range error due to pixel errors is . fB

Z

dp

dZ

2

29

Approach

UAVs flying at low altitudes must autonomously avoid obstacles

Strategy• Segment the image into sky and non-sky

Non-sky in the middle OBSTACLE

• Strategy 1 Aim at the sky

• Strategy 2 If it looms faster than a threshold and is in the middle AVOID

Else do NOTHING

30

Tailored to………………..

For most UAV applications (>50 m), the obstacles of concern will be large objects such as towers, buildings or large trees

For these cases, the problem of obstacle detection is different from that of ground vehicles in environments cluttered with many obstacles.

VS

31

Segmentation at Moffet Airfield

Results for multiple regions found (only largest regions shown, dark blue represents all small regions)

32

Sky Segmentation

33

Flight Demonstration

-500

-450

-400

-350

-300

-250

-200

-150

-100

-50

0

-100 0 100 200 300 400

Balloon

y po

siti

on (

m)

x position (m)

direction of flight

autonomous control started

avoidance with GPS

-500

-450

-400

-350

-300

-250

-200

-150

-100

-50

0

-100 0 100 200 300 400

Balloon

y po

siti

on (

m)

x position (m)

direction of flight

autonomous control started

avoidance with GPS

34

Vision Processing

Classification: balloon/horizon correctly found in ~ 90% of images

Time results: ~2Hz (120ms SVM, 200-600 ms horizon)

35

Flying Low

Helicopter pilots fly low

FAA requires see and avoid

Find the freeway and follow it

36

Used Sectionals to build a Manhattan model at 300 feet (approx.)

Simulation testing of Control

37

Cal UAV: Target CapabilitiesObstacle Avoidance

Simulation testing of Control• Flight through Manhattan model (300+ feet)

38

End 2