Robotic Cameras and Sensor Networks for High Resolution Environment Monitoring Ken Goldberg and Dezhen Song Alpha Lab, IEOR and EECS University of California,

Robotic Cameras and Sensor Networks for High Resolution Environment Monitoring

Ken Goldberg and Dezhen Song

Alpha Lab, IEOR and EECS

University of California, Berkeley

NetworkedRobots

internettele-robot:

RoboMotes: Gaurav S. Sukhatme, USC

Smart Dust: Kris Pister, UCB (Image: Kenn Brown)

NetworkedCameras

Security Applications

Banks, Airports, Freeways, Sports Events, Concerts, Hospitals, Schools, Warehouses, Stores, Playgrounds, Casinos, Prisons, etc.

Conventional Security Cameras • Immobile or Repetitive Sweep• Low resolution

New Video Cameras:Omnidirectional vs. Robotic

• Fixed lens with mirror• 6M Pixel CCD• $ 20.0 K• 1M Pixel / Steradian

• Pan, Tilt, Zoom (21x)• 0.37M Pixel CCD• $ 1.2 K• 500M Pixel / Steradian

Where to look?

Sensornet detects activity

• “Motecams”

• Other sensors:audio, pressure switches,

light beams, IR, etc

• Generate bounding boxes

and motion vectors

• Transmit to PZT camera

Activity localization

Viewpoint Selection Problem

Given n bounding boxes, find optimal frame

Related Work• Facilities Location

– Megiddo and Supowit [84]– Eppstein [97]– Halperin et al. [02]

• Rectangle Fitting – Grossi and Italiano [99,00]– Agarwal and Erickson [99]– Mount et al [96]

• Similarity Measures – Kavraki [98]– Broder et al [98, 00]– Veltkamp and Hagedoorn [00]

Problem Definition

Requested frames: i=[xi, yi, zi], i=1,…,n

Problem Definition• Assumptions

– Camera has fixed aspect ratio: 4 x 3– Candidate frame = [x, y, z] t

– (x, y) R2 (continuous set)– z Z (discrete set)

(x, y)3z

4z

Problem Definition• “Satisfaction” for user i: 0 Si 1

Si = 0 Si = 1

= i = i

• Symmetric Difference

• Intersection-Over-Union

SDArea

AreaIOU

i

i

1)(

)(

)(

)()(

i

ii

Area

AreaAreaSD

Similarity Metrics

Nonlinear functions of (x,y)

• Intersection over Maximum:

),(

)(

),max(

)1,)/min(()/(),(

i

i

i

i

biiii

Max

Area

aa

p

zzaps

Requested frame i , Area= ai

Candidate frame

Area = a

Satisfaction Metrics

pi

Intersection over Maximum: si( ,i)

si = 0.20 0.21 0.53

Requested frame i

Candidate frame

),(),( yxpyxs iii

),( yxpi

Requested frame i

Candidate frame (x,y)

)1,)/min(()/(),( biiii zzaps

(for fixed z)

Satisfaction Function

– si(x,y) is a plateau

• One top plane• Four side planes• Quadratic surfaces at corners• Critical boundaries: 4 horizontal, 4 vertical

Objective Function• Global Satisfaction:

n

iii

n

i

biii

yxpyxS

zzapS

1

1

),(),(

)1,)/min(()/()(

for fixed z

Find * = arg max S()

S(x,y) is non-differentiable, non-convex, butpiecewise linear along axis-parallel lines.

Properties of Global Satisfaction

Approximation Algorithm

spacing zoom :

spacing lattice :

zd

dx

y

d

Compute S(x,y) at lattice of sample points:

Approximation Algorithm

– Run Time: – O(w h m n / d2)

* : Optimal frame

: Optimal at lattice ~

: Smallest frame at lattice that encloses *

)ˆ()~

()( * sss

)(

)ˆ(

)(

)~

(1

**

s

s

s

s

ddz

z

z

min

min...

Exact Algorithm• Virtual corner: Intersection between boundaries

– Self intersection:– Frame intersection:

y

x

Exact Algorithm

• Claim: An optimal point occurs at a virtual corner. Proof:– Along vertical boundary, S(y) is a 1D piecewise

linear function: extrema must occur at boundaries

Exact Algorithm

Exact Algorithm:

Check all virtual corners(mn2) virtual corners(n) time to evaluate S for each (mn3) total runtime

Improved Exact Algorithm

• Sweep horizontally: solve at each vertical – Sort critical points along y axis: O(n log n)– 1D problem at each vertical boundary O(nm) – O(n) 1D problems– O(n2m) total runtime

O(n) 1D problems

Examples

Summary• Networked robots

• High res. security cameras

• Omnidirectional vs. PTZ

• Viewpoint Selection Problem

• O(n2m) algorithm

Future Work

• Continuous zoom (m=)• Multiple outputs:

– p cameras – p views from one camera

• “Temporal” version: fairness– Integrate si over time: minimize accumulated

dissatisfaction for any user

• Network / Client Variability: load balancing• Obstacle Avoidance

[email protected]

Related Work• Facility Location Problems

– Megiddo and Supowit [84]– Eppstein [97]– Halperin et al. [02]

• Rectangle Fitting, Range Search, Range Sum, and Dominance Sum– Friesen and Chan [93] – Kapelio et al [95]– Mount et al [96]– Grossi and Italiano [99,00]– Agarwal and Erickson [99]– Zhang [02]

Related Work

• Similarity Measures – Kavraki [98]– Broder et al [98, 00]– Veltkamp and Hagedoorn [00]

• Frame selection algorithms – Song, Goldberg et al [02, 03, 04], – Har-peled et al. [03]

Problem Definition• Assumptions

– Camera has fixed aspect ratio: 4 x 3– Candidate frame c = [x, y, z] t

– (x, y) R2 (continuous set)– Resolution z Z

• Z = 10 means a pixel in the image = 10×10m2 area • Bigger z = larger frame = lower resolution

(x, y)3z

4z

Problem Definition

Requests: ri=[xli, yt

i, xri, yb

i, zi], i=1,…,n

(xli, yt

i) (xri, yb

i)

Optimization Problem

n

iii

zyxcrcsS

1],,[

),(max

User i’s satisfaction

Total satisfaction

Problem Definition• “Satisfaction” for user i: 0 Si 1

Si = 0 Si = 1

= c ri c = ri

• Measure user i’s satisfaction:

)1),/min(()/(

1,)(

)(min

)(

)(),(

zzap

cResolution

rResolution

rArea

rcAreacrs

iii

i

i

ii

Coverage-Resolution Ratio Metrics

Requested frame ri

Area= ai

Candidate frame c

Area = a

pi

Comparison with Similarity Metrics

• Symmetric Difference

• Intersection-Over-Union

SDcrArea

crAreaIOU

i

i

1)(

)(

)(

)()(

crArea

crAreacrAreaSD

i

ii

Nonlinear functions of (x,y), Does not measure resolution difference

Optimization Problem

n

iii

zyxcrcs

1],,[

),(max

),(),( yxpyxs iii

),( yxpi

Requested Frame ri Candidate

Frame c

)1,/min()/(),( zzapcrs iiii

(for fixed z)

Objective Function Properties

• si(x,y) is a plateau

• One top plane• Four side planes• Quadratic surfaces at corners• Critical boundaries: 4 horizontal, 4 vertical

Objective Function for Fixed Resolution

4z x

y

3z

4(zi-z)

Objective Function• Total satisfaction:

n

iii

n

iiii

yxpyxS

zzapcS

1

1

),(),(

)1),/min(()/()(

for fixed z

Frame selection problem: Find c* = arg max S(c)

S(x,y) is non-differentiable, non-convex, non-concave, but piecewise linear along axis-parallel lines.

Objective Function Properties

4z x

y

3z

4(zi-z)

3z y

si

3z

(z/zi)2

3(zi-z)

x

si

4z 4z

(z/zi)2

4(zi-z)

Plateau Vertex Definition• Intersection between boundaries

– Self intersection:– Plateau intersection:

y

x

Plateau Vertex Optimality Condition

• Claim 1: An optimal point occurs at a plateau vertex in the objective space for a fixed Resolution. Proof:– Along vertical boundary, S(y) is a 1D piecewise

linear function: extrema must occur at x boundaries

y

S(y)

Fixed Resolution Exact Algorithm

Brute force Exact Algorithm:

Check all plateau vertices (n2) plateau vertices(n) time to evaluate S for each (n3) total runtime

Improved Fixed Resolution Algorithm

• Sweep horizontally: solve at each vertical – Sort critical points along y axis: O(n log n)– 1D problem at each vertical boundary O(n) – O(n) 1D problems– O(n2m) total runtime

for m zoom levels

O(n) 1D problems

y

S(y)

x

y

A New Architecture

Activity and Video databaseActivity localization

Activities

Frame selection

Active surveillance

Control commands

Videos

Software diagram

TCP/IP

TCP/IP

Activity & video database

Core (with shared memory segments)

RPC module

RPC module

RPC module

Communication

Console/Log

Activity server

Activity generation

Motescam

Wireless Camera control

Calibration

Panoramic image generation

Video server

Panasonic HCM 280 Camera

Visual C++

NesC + Tiny OS

Gnu C++MySQL

Database: indexing video data

• Activity Index– Timestamp

– Speed (Or other sensor data)

– Range

• Query video data using activity – Show video clips of moving objects with speed faster than

1 meter per second in zone 1 in last 10 days

– Show video clips of zone 1 when CO2 concentration exceeded the threshold in Jan. 2004 (Assuming CO2 sensor is used in detecting activity)

Activity and Video database

Camera control