CBWare - Distributed Middleware for Autonomous Ground Vehicles

CBWare - Distributed Middleware for Autonomous Ground

Vehicles

Master’s Thesis DefenseBy

Vidhyalakshmi VenkitakrishnanAdvisor: Dr. Arun Lakhotia

Center for Advanced Computer StudiesUniversity of Louisiana at Lafayette

October 18, 2006

Presentation Outline

Part I - Motivation and Research Contributions

Part II - Background and Related Work

Part III - CBWare and Evaluation Results

Part IV - Conclusions and Future Work

Middleware in AGV’s - Motivation

I. Real-time Information Exchange

Processes are computationally intensive and complex

Need for distributing components on multiple machines to achieve fairness and efficiency

Middleware – Key to information exchange among distributed components

CajunBot

II. Sensor Data Fusion

LIDAR

(Range, Theta)

(x, y, z)Global (x, y, z)

INS

Sensor Data Fusion – contd.

LIDAR Queue

INS Queue

Fusing most recent reading from queues – not consistent

Both sources generate data at different frequencies

CajunBot on rough terrainDARPA GC 2005

Sensor Data in Rough Terrain

t1

III. Log Server

Need data from processes for post-analysis

Collect data to tune parameters

CajunBot – NQE 2005 Z spike - wall of obstacles detected from log data

IV. Remote Real-time Monitoring

Why Real-time monitoring?

Visual/graphical views of the world seen by system

Internal states of Obstacle Detection

Visualize live data during field testing

Debug problems with processes in real-time

Research Contributions

Real-time Information Exchange in distributed system

Support for fusion of consistent data

Log server

Real-time monitoring capability

Middleware Background

What is Middleware?

Layer of software that connects different application components.

Supports complex, distributed applications

Hides heterogeneity of underlying platforms

Types of Middleware

Publish/Subscribe

Client/Server

Remote Procedure Calls

Other IPC Mechanisms: Shared Memory, Message Queues, UNIX IPC

Publish/Subscribe Model

Powerful abstraction for distributed systems

Message-based anonymous communication

Publishers and Subscribers are decoupled

Good solution for scalability

Examples: CMU-IPC, NDDS, IBM-Gryphon, Siena

Publisher SubscriberPublish/Subscribe

MiddlewarePublisher

publishes data

Subscriber subscribes to data

Middleware

delivers data

Client/Server Model

Clients send request to the server

Server processes requests and responds back

Clients blocks until response from server

Client 1

Client 2

Client 3Server

Request

Response

Remote Procedure Calls

Protocol to extend Local Procedure

Involves two independent processes, which may reside on different machines Caller (Process A on Host A) issues procedure call to Callee

(Process B on Host B) with list of argument values Caller is suspended Callee executes procedure, returns values to Caller Caller resumes execution

Examples: Java RMI, CORBA, Microsoft DCOM

Robotic Middlewares

IPT (1996) Object-oriented, message passing toolkit Unmanned Ground Vehicle Client/Server Model

MIRO (2002) Middleware for Mobile Robots Based on CORBA

Broker (2005) IPC Toolkit for Multi-Robot Systems Works on Publish/Subscribe Model

CBWare Architecture

Works on Publish/Subscribe Model

• Two types of Interfaces CBQueues CBPackets

• Log server

• Remote monitoring

Dedicated Logging Machine

Separate Machine – Why? Hard disk – Most sensitive part of a machine Bumps – Common reason for hard disk

crash/failure Disk crash – Affects autonomous operations

Provision to log only sample of data to disk

Sampling data for remote monitoring

Why sample data sent over wireless network? Wireless network cannot handle all the data produced

Onboard network – 1 GB LAN Ethernet switch Wireless network – 802.11g Wireless

Bandwidth Ethernet LAN: 1000 Mbps 802.11g wireless: 54 Mbps

CBQueues

Interprocess communication on a single machine

Interface to read/write messages

Built using POSIX Shared memory

Shared Memory Model

Area of memory shared by multiple processes.

Shared Memory area -indistinguishable to a process from unshared memory

Code Code

Private data Private data

Shared data Shared data

Why Shared Memory Model?

Fastest form of IPC available

Negligible communication overhead

Need to deliver high bandwidth sensor data in a timely manner

CajunBot’s Shared Memory Model

Queues for every message type in shared memory area

Messages in every queue temporally ordered Crucial for Interpolation support

Single Writer Restriction

Only one writer for each message queue, no limit on the number of readers

Enables temporal ordering of data in distributed queues

Multiple producers for same message type Separate queue maintained for each producer

Data Fusion and Interpolation

INS generates data at 100 Hz Produces data at 10 ms interval

LIDAR LIDAR generates data at 75 Hz Scans at 13 ms interval

Most recent INS data may be up to 10 ms old when LIDAR scan is read

Stabilization of sensors

In CajunBot, no stabilization of sensors Fusing most recent data can give erroneous results 0.5 degree inaccuracy of angular difference -> 2 feet

displacement of global point from correct location Leverage rough terrain to increase visibility

In vehicles with sensor stabilization, can fuse most recent data Stabilization dampens rotating movements Sensors wont experience significantly different

orientations in the 10 ms period

CBPackets

Distributed interprocess communication across machines

UDP Broadcast

Support for multiple readers and writers

Why UDP Protocol?

Real-time applications Deliver messages in time

Small Packet Header overhead TCP Header = 20 bytes, UDP Header = 8 bytes

For a 20 byte message, compare: 40 bytes Vs 28 bytes

Example:Case 1: Message Size = 12 bytesEfficiency using TCP = (12 / 12+20) = 0.37 = 37%Efficiency using UDP = (12/ 12+8) = 0.6 = 60%

Case 2: Message Size = 1000 bytesEfficiency using TCP = (1000 / 1000+20) = 0.98 = 98%Efficiency using UDP = (1000 / 1000+8) = 0.992 = 99%

Why UDP Protocol? – contd.

Broadcast/Multicast capabilities Send messages to all processes at once

Connectionless nature Useful for remote monitoring No problems if wireless network fails

Data Marshaling

Why data marshaling? Distributed System – Machines with heterogeneous

architectures/OS follow different byte-ordering, alignment strategies

Data converted to neutral format for transmission over the network

Examples of Data Encoding standards: XDR, NDR, CDR

XDR used by CBWare

XDR Translation

Encode: Convert data from native format to XDR format

Decode: Convert data from XDR format to native format

Every message marshaled by CBWare before sending over network

Sender ReceiverXDR FormatEncode data Decode data

Monitoring Process Status

Processes on multiple machines generate status/warning/error messages

Monitoring messages on each machine Tedious task when system scales to higher level

Cbmesg: Interface for logging and real-time monitoring of process status information

Replication of Shared Memory

Combination of CBQueues and CBPackets

Subscriber replicates local shared memory Distribute queues to multiple machines Easy transfer of programs Scalability of computational power

CBWare Network Evaluation

• CBWare evaluation metrics End-to-End Latency (msec) Packet Rate Bandwidth (bytes/second) Packet Order

• Experimental setup• Dell Poweredge servers• 1 GB Ethernet LAN switch

CBWare Network Evaluation Results

End-to-End Latency Delays ranging from 0.4 msec to 6 msec depending on packet

size Transmission delay = Packet reception time – Packet sent time

CBWare Network Evaluation Results – contd.

Bandwidth and Packet Rate Packet Rate = (Bandwidth/Message Size) Maximum rate ranging from 1200 packets/second to 90

packets/second

Packet order Measurement based on timestamp No out of order packets on CBWare network No network congestion

Network Monitoring Tools used: Ethereal, tcpdump

Conclusion

CBWare – Middleware for Autonomous Ground Vehicle

Publish/Subscribe Model

Distributed Information exchange - Shared Memory and UDP Broadcast

Support for fusion of sensor data

Log server

Sampling for remote real-time monitoring

CBWare’s comparison with other Middlewares

CBWare CMU-IPC NML NDDS Broker IPT

Data interpolation &

Fusion

Yes No No No No No

Sampling data for Remote Monitoring

Yes No No No No No

Protocol Shared Memory +

UDP

TCP

UDP

Shared Memory + UDP

UDP UDP TCP Sockets

Languages C++ C++

Java

LISP

C++

Java

C++

Java

C++

Java

Python Perl

C++

Pub/Sub Models

Yes Yes Yes Yes Yes No

Future Work

Fault tolerance

Compression techniques for larger messages

Port CBWare to other Operating Systems

Questions

Extra slides

Distributed Computing System of CajunBot

Onboard Computing system of CajunBot Four machines NTP Separate networks

Why XDR?

XDR format of data is same on all machines Any machine can decode data encoded by

any machine Easily add machines with different

architectures

Log Control

Remote control of central log server (cb_logd) TCP Connection Communication using predefined ASCII Protocol

Three types of control signals Enable logging data Disable logging data Change log directory on log server

CBQueues Utilities

Remove corrupted shared memory: cb_qclean Clean shared memory area

Read/Write to shared memory: cb_qtest Testing and debugging problems with shared memory

CBWare’s Publish/Subscribe Components

Publisher: cb_publisher Handles socket operations Constructs CBPackets Marshal data Broadcast on network

Central Log Server: cb_logd Logs data received from broadcast network to disk Runs on separate machine to prevent disk failure affecting

autonomous operations Provision to log only a sample of data to disk Samples data at prescribed interval and broadcasts data on

wireless network for remote monitoring

CBWare’s Publish/Subscribe Components – contd.

Subscriber: cb_subscriber Runs in two modes

Mode 1: Replicate “shared memory” queues across machines Mode 2: Real-time monitoring on remote laptop

Easy transfer of programs to multiple machines

Easy scalability of computational power

CBQueues Interface

Read interfaces Most recent message Next message Check for new message

Write interface

CBPacket format CBPacket = CBHeader + Data

CBHeader = <Channel, Timestamp, Size, Checksum, Encoding Format>

Maximum transmission size upto 65507 bytes

CajunBot on flat terrain DARPA GC 2005

Sensor data in Flat Terrain

1 2 3 4 5 6 7 8 9 10 11

LIDAR Data

INS Data

Time Instance (t)

0

10

0

10

Pitch of the vehicle

(degrees)

Pitch of the sensor

(degrees)

t1