M.ENG RESEARCH PROJECT APRIL 2015 1 Abstract— A prototype autonomous airship and control system was developed and tested. Despite being among the earliest types of unmanned aerial vehicles to receive research attention, airships remain attractive as a platform because they can attain longer flight times and are inherently more stable in air than equivalent helicopters or multirotors. Previous autonomous blimps have required a connection to a computer on the ground, or carefully calibrated multi-camera rigs in order to calculate their pose relative to their target. This project instead utilizes currently available low-cost lightweight hardware and open- source software to perform all processing on board the blimp. The controller, running on a Linux microcomputer, combines pose estimates from a downward looking camera, motion vector estimates from video and an eleven degree-of-freedom inertial measurement unit to control the vehicle altitude, heading and speed in station-keeping and follow-me tests. The system was tuned and evaluated using indoor test flights. Index Terms— Autonomous airship, blimp, pose estimation, unmanned aerial vehicle, visual servoing. I. INTRODUCTION NMANNED AERIAL VEHICLES, popularly referred to as ‘drones’ or UAVs, first emerged in the 1950s, but in the last 15 years have become a topic of growing research, commercial and public interest. Applications of UAV technology are widespread and ever increasing, from military and surveillance uses to agriculture and package delivery. Two different perspectives on civilian UAV usage are provided by [1] and [2]; numerous other technology reviews have been written exploring uses in specific fields [3]. Several factors have combined to drive this expansion. Key technologies (including System-on-a-Chip (SoC) processors, lightweight lithium polymer batteries and microelectronic sensors) have matured and become mass-market as a result of widespread adoption of smartphone-like devices. These components are now able to meet the twin requirements for UAV technologies: real-time performance and small physical size and weight [4], [5]. Building UAVs was initially only feasible for well-resourced organisations but today the open-source, crowd-funding and hobbyist movements have lowered the barrier to entry. Smaller companies, researchers and individuals can now share their systems, designs, research and code with the community [6]. Capitalising on the now widespread availability of UAV components and associated computing platforms, this project set out to deliver a blimp system suitable for developing visual controls, with applications in wide area persistent surveillance and mapping missions. The indoor prototype, presented in Figure 1, has a flight duration of four hours and uses imagery and an Inertial Measurement Unit (IMU) to station-keep relative to an image target beneath. Major components are all commercially available items, resulting in a low cost, lightweight system. Figure 1 – Prototype blimp system in flight in the lab. Blimp is 3.5m long, flying around 4m above ground In the rest of this section, prior work in UAV airships is briefly reviewed and the objectives for this project are outlined. The main sections of this paper then present the blimp, review a dynamic model suitable for closed-loop control and outline the autopilot software. The chosen control techniques are evaluated on the blimp platform before applications and areas for future work are suggested. A. Airships and blimps Airships are traditionally defined as craft whose main source of lift is buoyancy from an envelope of lighter than air gas, i.e. Helium. Blimps are airships without a rigid skeleton. An ideal blimp is neutrally buoyant in air, only requiring powered thrust to move or change direction. As a result, compared to multirotor drones of equivalent payloads, an ideal blimp has far smaller inflight power consumption. Untethered Autonomous Flight Using Visual and Inertial Sensing on an Indoor Blimp Andrew Mathieson supervised by Dr Toby Breckon U
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
M.ENG RESEARCH PROJECT APRIL 2015 1
Abstract— A prototype autonomous airship and control system
was developed and tested. Despite being among the earliest types
of unmanned aerial vehicles to receive research attention,
airships remain attractive as a platform because they can attain
longer flight times and are inherently more stable in air than
equivalent helicopters or multirotors. Previous autonomous
blimps have required a connection to a computer on the ground,
or carefully calibrated multi-camera rigs in order to calculate
their pose relative to their target. This project instead utilizes
currently available low-cost lightweight hardware and open-
source software to perform all processing on board the blimp.
The controller, running on a Linux microcomputer, combines
pose estimates from a downward looking camera, motion vector
estimates from video and an eleven degree-of-freedom inertial
measurement unit to control the vehicle altitude, heading and
speed in station-keeping and follow-me tests.
The system was tuned and evaluated using indoor test flights.
Index Terms— Autonomous airship, blimp, pose estimation,
unmanned aerial vehicle, visual servoing.
I. INTRODUCTION
NMANNED AERIAL VEHICLES, popularly referred to as
‘drones’ or UAVs, first emerged in the 1950s, but in the
last 15 years have become a topic of growing research,
commercial and public interest.
Applications of UAV technology are widespread and ever
increasing, from military and surveillance uses to agriculture
and package delivery. Two different perspectives on civilian
UAV usage are provided by [1] and [2]; numerous other
technology reviews have been written exploring uses in
specific fields [3].
Several factors have combined to drive this expansion. Key
with 3 orthogonal axes), barometric pressure and temperature.
The pressure sensor gives a direct reading the height above
sea level of the blimp, using the ISA barometric formula. The
nominal resolution of this sensor is ±0.1 m, although in test
conditions this variability was found to be ±0.25 m. Tests on
the bench indicated that height readings are normally
distributed about the true height, so in flight, variability of
height estimates is smoothed by taking the mean of the most
recent ten samples. Flight altitude 𝑍 is calculated by
subtracting the current height above sea level from the height
above sea level at takeoff.
Similarly, with the magnetometer giving a measurement of
𝑦𝑎𝑤 relative to the North direction and the accelerometer
giving measurements for 𝑝𝑖𝑡𝑐ℎ and 𝑟𝑜𝑙𝑙 relative to gravity, the
current flight pose relative to the start is also calculated by the
IMU software [24]. Calculating positional coordinates by
double-integrating IMU accelerations is generally considered
to be ineffective because measurement noise is greatly
exacerbated by the integration process, so the IMU is currently
only used for rotational pose estimation.
C. Velocity estimation using H.264
The final source of pose information is the optic flow in the
scene, which in steady flight is correlated to the motion of the
camera over the scene. Unlike the direct pose estimation, this
method does not require a specific target in the scene, instead
it can work using features in the natural environment.
The camera used natively records video in H.264 (MPEG
AVC) format. An intermediate step in the H.264 encoding
process is to calculate the optic flow in each 16x16 pixel
macroblock. In common with many modern SoCs, this
encoding process is implemented using the graphic processing
unit instead of the CPU, happily, the camera drivers allow
access to these intermediates, permitting analysis of the
motion domain as well as the image domain.
Each time the camera encodes a video frame, the motion
field is copied to memory using a HSV colour space to
represent the direction of the motion field, sum-of-absolute-
differences measure and magnitude of the motion field at each
pixel, in a similar manner as used in [25]. In this application it
was found that the GPU can work about five times faster than
the CPU so to avoid unnecessary computation HSV motion
arrays are only processed when camera pose data is also ready.
The aim of the motion processing is to reduce the incoming
motion field into four estimates: the direction and magnitude
of the foreground and background motion.
To achieve station keeping relative to a target in the
foreground, the controller tries to minimise the motion relative
to the foreground, and keep the motion relative to the
background constant. In the case of a stationary target, the
background motion should additionally be zero. The MPEG
block-matcher that achieves this motion encoding is not as
robust as a specialised optic flow routine, so the produced
motion fields tend to be very noisy. Furthermore, the level of
motion detected is not always independent of scene features.
This is demonstrated in Figure 7, where the brightest colours
indicate largest motion and the mapping from motion direction
to hue is given in the key. The ground truth here is the camera
moving forwards over the chessboard. In the HSV array, the
predominant hue is red, corresponding to a direction of around
0° as expected. However, in some macroblocks the motion has
been incorrectly classified the motion incorrectly (green).
Figure 7 – Captured image with located chessboard axes overlaid;
Motion HSV representation of same frame, strongest motion is
detected in the chessboard area and along the bottom edge; Key.
In many frames the signal-to-noise ratio is high so a
clustering approach would frequently misclassify noisy
background pixels as foreground and vice versa. Instead, the
components of the motion field are considered, and the noise
is modelled as Gaussian white noise present at a low level in
every macroblock.
Otsu’s thresholding [26] is used to binarise the magnitude
of the motion field, removing small-magnitude noise from the
frame, leaving the larger magnitude motion which must be
classified as either foreground or background. A histogram of
the directions of these motions with bin widths corresponding
to 5 degree intervals is computed, and the modal class is taken
to be the average direction of the foreground’s motion. For the
HSV motion array shown above, this method estimated the
foreground motion direction to be at 353 ° which is in good
agreement with the ground truth.
The performance of this motion segmenter implementation
on the blimp is evaluated over a larger number of tests in
Table 4. In general, performance is satisfactory, although
better performance is seen when the blimp is moving relative
to a stationary target than when both the target and the
background are experiencing large magnitude motion. In the
final column of Table 4, if the target has already been located,
a region-of-interest around the target is used to denote the
foreground. In the problematic moving background case, this
method helps, however it relies on the target being in view.
M.ENG RESEARCH PROJECT APRIL 2015 8
Scenario
Ground truth
motion
direction
Number of frames
where motion
estimate is within
±10° of ground truth
F
ore
-
gro
un
d
Ba
ck-
gro
un
d
Fo
re-
gro
un
d
Ba
ck-
gro
un
d
Usin
g
RO
I
Flight over
stationary target 0° None
86/
177
91/
177
38/
41
Flight over
moving target None 180°
53
/124
66
/124
20/
53
Table 4 – Experimental verification of H.264 motion direction
segmentation, acquired in the lab.
A weakness of this current implementation of motion
direction estimation is that it is not able to correctly detect or
compensate for out-of-plane motion. The motion classifier
presented in [25] solves this by compared the incoming data to
a large library of pre-computed motion fields.
D. Sensor Fusion
Once all sources of pose estimates have been resolved into
the ego frame, the simplest way to combine them into a single
best guess is to take a weighted average. Here, the weights
awarded to each component were optimised experimentally to
minimise the disagreement between the different pose sources
over a large number of samples in controlled conditions.
In autopilot mode sensor fusion runs at 5 Hz, limited by the
vision system refresh rate. Improved techniques for combining
sensor data have been widely explored in the literature, where
Kalman Filtering is a commonly used technique. In [26], two
filter formulations are combined to fuse IMU and chessboard
derived pose information. An opportunity for future work
would be to apply that work to this blimp, and extend it by
incorporating the motion directions estimates and their
estimated uncertainties into state transitions. Such a ‘motion
augmented’ fusion method should give more reliable pose
estimates than the current system.
E. De-coupled control
The system now has an estimate of its current pose which is
good to within measurement error. The final step in realising
autonomous station-keeping is to transform the error between
the current pose estimate and the target pose into the actuator
movements required to get to the target.
In Section III the dynamics of the blimp were considered,
and the simplifying assumptions permitted by low speed flight
were explored. In particular, coupling between degrees of
freedom due to aerodynamics and moment effects was
neglected. Furthermore, motion in several degrees of freedom,
notable 𝑟𝑜𝑙𝑙 and 𝑝𝑖𝑡𝑐ℎ, was also neglected due to the flight
characteristics of the large blimp envelope and concentrated
payload mass. These simplifications mean that a separate
controller can be implemented for each actuatable degree of
freedom. Following the model of [9], higher-level controllers
may then be implemented to adjust the setpoints of these
controllers in accordance with mission requirements.
F. 𝑍 controller
Regulates the steady-state height of the blimp. Primary
input source is the barometric pressure sensor, supplemented
by camera pose distance-to-image information when
chessboard target is in view. Output is the vertical component
of the common mode thrust required to balance the weight of
the blimp with the buoyancy.
Good control of the flying height is required to ensure that
the planar motion detection works properly. This is primarily
provided by the neutral buoyancy of the envelope; using the 𝑍
controller at a low thrust level (10%) is sufficient to control
height and also helps improve responsiveness of the other
controllers, as the motors are already spinning continuously.
G. 𝑦𝑎𝑤 controller
Regulates the heading of the blimp. Primary input source is
the sensor fusion angle provided by the IMU and vision
system. When the chessboard target is in view, stable heading
hold relative to the chessboard with no operator intervention
was recorded. Without the vision system periodically
correcting the target heading, stability was limited by the
tendency for the IMU 𝑦𝑎𝑤 measurement to drift. Once the
blimp had lost the original target heading, oscillation between
±60° of the target direction was observed.
The 𝑦𝑎𝑤 controller was tested and tuned with the blimp
tethered at a fixed height above the chessboard target, but
allowing it to yaw freely. With autopilot engaged, the target
was displaced by angles between 0° and 60°. The system step
response was calculated from the recorded log data. In the
next section the tuning of this controller is outlined.
H. 𝑋 controller, 𝑋 velocity controller
Regulates the position of the blimp relative to the target, or,
in the case of a moving target, tries to match the speed of the
target. Primary input source for 𝑋 controller is the location of
the target centre relative to the image. If H.264 motion
directions are available, these are used to determine whether to
speed up or slow down. As there are no absolute or inertial
sources of reliable 𝑋 pose information, this controller can only
function when the target is in view. Output is the horizontal
component of the common-mode thrust.
The 𝑋 controller was tested by ballasting the envelope to
neutral buoyancy and flying over a chessboard moving
forwards below. In Figure 8 the 𝑋 position of the target points
(whose location in the first frame are shown in the inset)
during one of these tests is plotted. The controller is able to
keep the position of each target point in the frame within
±25 pixels of the starting value, which, given the properties of
the camera and distance to the target, equates to~ ±90 mm in
the real world. The lines do not remain parallel at all times
because the blimp does not move purely in the 𝑋 direction; in
particular, if 𝑍 and 𝑦𝑎𝑤 fluctuations were not also controlled,
the accuracy of the 𝑋 controller was reduced to ~±200 mm.
The large inertia of the blimp envelope when flying forward
made this controller more susceptible to overshoot than the
other controllers.
M.ENG RESEARCH PROJECT APRIL 2015 9
Figure 8 – Position of target points during 𝑋 controller test
VI. RESULTS & DISCUSSION
Each controller has a proportional and integral gain. As in
the conventional PI controller, proportional gain 𝐾𝑝 controls
the magnitude of the thrust output and the gains have been
selected to provide a sub-critical response for good stability.
Multirotors require fast responses to errors for aerobatic flight,
however in a blimp the low speed of flight and large inertia
means that there is no great advantage to trading off stability
for a quicker response.
Unlike in a conventional controller which is evaluated at a
fixed update frequency, this application requires that the
controller can be updated irregularly whenever new pose
information becomes available. Furthermore, thruster
instructions put into the job queue by the autopilot may be
interrupted and overridden by other higher priority
instructions. For these reasons, the integration time in the PI
controller is implemented as the runtime of each instruction.
Large errors therefore result in a large correcting force
being applied for a long time; if, as the blimp approaches its
set point it receives a new pose estimate, it refines the
instructions to the actuators accordingly. Even if no new
information is received, the actuators will at the latest be
stopped at the first estimated end time, which will be closer to
the target than before. Once the blimp has settled to be within
tolerance of the target it returns to its usual hovering state.
A. Tuning by experiment
To verify this approach experimentally, the normalised
time-domain unit step response of the 𝑦𝑎𝑤 controller in the
tuning tests is plotted in Figure 9. It is seen that whilst there is
good convergence to the target, during the rise period the
variability between tests (shown by the grey ±2σ bounds) is
more than double that during steady state.
With 𝐾𝑝 = −5, this is a subcritical response. The median
5 to 95% rise time of the responses is 4.94 s.
The autopilot was configured to ignore pose estimates
containing Euler angles outside of expected ranges to mitigate
against incorrect pose estimates that could cause very large
actuator commands to be issued. It was noticed that bad pose
estimations were more likely immediately after a rapid motion
of the servos (e.g. going from hard left to hard right) when the
reaction caused the blimp gondola to briefly swing in the 𝑟𝑜𝑙𝑙 direction. In test flights, this out-of-plane motion was damped
by the large inertia of the blimp envelope so did not affect
control stability. Performing Kalman filtering on pose
estimates prior to their use in the autopilot would further help
protect against this source of errors.
B. Comparison with analytical methods
Standard methods for controller tuning require that the
system transfer function be linear and second-order [27]. Even
after the simplification process outlined above was applied to
the blimp dynamics the result was non-linear; a complete
treatment which does not omit coupling between degrees of
freedom would also likely be higher-than-seccond order.
Tuning results from standard methods must therefore be
approached with caution; this project used them to inform the
range of values of 𝐾𝑝 to test.
Approximating the system transfer function as second order
with time delay using Harriott’s method (see [27]) allowed
Nyquist analysis to find the gain margin from the step
response tests. The gain margin of the median line in Figure 9
was 26 db, suggesting 𝐾𝑝 should be set to 19 for a critical
response.
Figure 9 – Normalised time domain unit step response of 𝑦𝑎𝑤 controller at 𝐾𝑝 = −5. Responses have been normalised by
timeshifting such that the step input occurrs at t=0, and scaling such that the average 𝑦𝑎𝑤 before the step = 0 and average 𝑦𝑎𝑤
after step = 1. Median and ±2 standard deviation bounds of the normalised responses are also plotted at each time instant.
0
40
80
120
160
200
240
0 10 20 30 40 50 60 time, s
x1 x2 x3 x4
X
Y
initial point positions
ego framedirectionstr
acke
d X
po
sitio
ns, p
x
-0.2
0.0
0.2
0.4
0.6
0.8
1.0
1.2
-6 -4 -2 0 2 4 6 8 10 12 14 16 18
no
rma
lise
d y
aw
re
sp
on
se
time relative to step, s
Unit Step at t=0 Median Norm. responseMedian+2σ Median-2σ Test 1 Normalised response Test 2 Normalised responseTest 3 Normalised response Test 4 Normalised responseTest 5 Normalised response Test 6 Normalised responseTest 7 Normalised response Test 8 Normalised response
M.ENG RESEARCH PROJECT APRIL 2015 10
In reality this was found to be an over-estimate, frequently
causing the system to overshoot the target or develop unstable
oscilations which required manual intervention to correct.
In contrast, the widespread Zeigler-Nichols time-domain