IMPROVEMENTS TO SITUATIONAL AWARENESS DURING APPROACH AND LANDING THROUGH ENHANCED AND SYNTHETIC VISION SYSTEMS by Justin Stephen Brown A Graduate Capstone Project Submitted to the Worldwide Campus in Partial Fulfillment of the Requirements of the Degree of Master of Aeronautical Science Embry-Riddle Aeronautical University Worldwide Campus Pensacola, FL May 2011
69
Embed
Improvements to Situational Awareness During Approach and Landing Through Enhanced and Synthetic Vision Systems
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
IMPROVEMENTS TO SITUATIONAL AWARENESS DURING APPROACH AND LANDING THROUGH ENHANCED AND SYNTHETIC VISION SYSTEMS
by
Justin Stephen Brown
A Graduate Capstone Project Submitted to the Worldwide Campus
in Partial Fulfillment of the Requirements of the Degree of Master of Aeronautical Science
Embry-Riddle Aeronautical University
Worldwide Campus Pensacola, FL
May 2011
ii
IMPROVEMENTS TO SITUATIONAL AWARENESS DURING APPROACH AND LANDING THROUGH ENHANCED AND SYNTHETIC VISION SYSTEMS
by
Justin Stephen Brown
This Graduate Capstone Project
was prepared under the direction of the candidate’s Research Committee Member, Mr. William L. Little, Adjunct Associate Professor, Worldwide Campus
and the candidate’s Research Committee Chair, Dr. Peter B. Walker, Adjunct Assistant Professor, Worldwide Campus and has been
approved by the Project Review Committee. It was submitted to the Worldwide Campus in partial fulfillment of
the requirements for the degree of Master of Aeronautical Science
PROJECT REVIEW COMMITTEE:
___________________________________ William L. Little, MAS
Committee Member
___________________________________ Peter B. Walker, Ph.D.
Committee Chair
iii
ACKNOWLEDGEMENTS
I would like to thank my wife, Tamara Brown, MPT, for providing the support
and stamina required to care for our daughters while I worked on the Capstone project.
When we started down this journey we were still somewhat newlyweds. We managed
through two PCS moves, the birth of two daughters, two months of daily visits to the
NICU and countless evenings spent in front of the computer. I cannot thank Tamara
enough for her love and patience throughout the five years I've spent working on my
graduate education. Without her, this never would have happened.
I would also like to recognize my parents for teaching me the meaning of
dedication and hard work. Thank you for creating a strong work ethic in me.
I am also very thankful to my committee members who helped me finish this
project remotely with consistent and reliable communication. You all deserve a raise.
Finally, I am sincerely thankful to God for blessing me with family, friends, co-
workers and the means necessary that allowed me to achieve my goals.
iv
ABSTRACT
Researcher: Justin Stephen Brown
Title: Enhanced and Synthetic Vision System’s Improvement to Situational Awareness During Approach and Landing
Institution: Embry-Riddle Aeronautical University
Degree: Master of Aeronautical Science
Year: 2011
In June 1999, American Airlines flight 1420 overran runway 04R in Little Rock,
Arkansas and fell into a ravine killing 10 people including the captain. This accident
occurred during a thunderstorm and contains many of the characteristics involved in
approach and landing accidents. The Flight Safety Foundation’s (FSF) Approach and
Landing Accident Reduction (ALAR) study noted that 76 percent of all jet and turboprop
aircraft accidents involved landing overruns, loss of control, runway incursion, non-
stabilized approaches, or controlled flight into terrain (Killers in aviation, 1998).
Enhanced and synthetic vision systems are technological advances that intend to reduce
pilot workload, especially during the final phase of flight. The primary aim of this
research was to use simulation studies, proto-type research and statistical data to compare
mishap rates and pilot workload during visual flight rules (VFR) and instrument flight
rules (IFR) approaches to landing using traditional navigational aids and enhanced and
synthetic vision technology.
v
TABLE OF CONTENTS
Page
PROJECT REVIEW COMMITTEE ii
ACKNOWLEDGEMENTS iii
ABSTRACT iv
TABLE OF CONTENTS v
LIST OF TABLES viii
LIST OF FIGURES ix
Chapter
I INTRODUCTION 1
Background of the Problem 1
Researcher's Work Role and Setting 5
Statement of the Problem 6
Limitations 7
Acronyms 8
II REVIEW OF RELEVANT LITERATURE AND RESEARCH 10
History of Navigational and Landing Aids 10
Developments in SVS 15
Developments in EVS 18
Situational Awareness Measurement 19
vi
Prior Research of SVS and EVS 21
SVS for CFIT Prevention 22
SVS Symbology 23
Blending EVS and SVS 25
Statement of the Hypothesis 26
III RESEARCH METHODS 27
Research Model 27
Survey Population 27
Sources of Data 28
The Data Collection Device 28
Instrument Pretest 29
Instrument Reliability 30
Procedures 30
Treatment of Data 30
IV RESULTS 31
V DISCUSSION 40
VI CONCLUSION 44
VII RECOMMENDATIONS 46
REFERENCES 48
APPENDICES 52
vii
A BIBLIOGRAPHY 52
B BASELINE HUD FOR IMC-DAY CONDITION 53
C SVS HUD FOR IMC-DAY CONDITION 54
D EVS HUD FOR IMC-DAY CONDITION 55
E COMBO HUD FOR IMC-DAY CONDITION 56
F BASELINE HUD FOR IMC-NIGHT CONDITION 57
G SVS HUD FOR IMC-NIGHT CONDITION 58
H EVS HUD FOR IMC-NIGHT CONDITION 59
I COMBO HUD FOR IMC-NIGHT CONDITION 60
viii
LIST OF TABLES
Table Page
1 Pilot Flight Time in Hours 29
2 ANOVA Results for RMSE Data 31
3 Summary of F-Test Results on SAGAT Scores 36
4 Results of Post-Hoc Analysis for Display Effect on Pilot SA Types 38
ix
LIST OF FIGURES
Figure Page
1 Four Course Radio Range 11
2 Very High Frequency Omnidirectional Radio Range 11
3 Early Forms of Runway Landing Aid Lighting 13
4 Elements of the Instrument Landing System 14
5 Universal Avionics SVS Display 17
6 EVS Using FLIR on Approach 18
7 RMSEs for Each Display Configuration 32
8 RMSEs for IMC Conditions 33
9 RMSEs for Display Configurations by IMC Condition 34
10 RMSEs for Leg by Display Configuration 35
11 Overall SA Scores for Each Display Configuration 37
12 SA Scores for Three Different Types of SA Being Measured 38
1
CHAPTER I
INTRODUCTION
Background of the Problem
The Flight Safety Foundation (FSF) conducted a study in 1998 with intentions of
reducing approach and landing accidents ("Killers in Aviation", 1998). The FSF study
found that 76 percent of all jet and turboprop aircraft accidents involved landing
overruns, loss of control, runway excursions, non-stabilized approaches, or controlled
flight into terrain (CFIT) ("Killers in Aviation", 1998). More specifically the FSF study
found that nearly 60% of ALAs involved CFIT ("Killers in Aviation", 1998). Similarly,
a study conducted by the National Aeronautics and Space Administration (NASA) in
2005 found that 66 percent of all jet accidents occur during the landing or final phase of
flight (NASA, 2006). The study also noted that 81 percent of these ALAs occurred
during instrument meteorological conditions (IMC) (NASA, 2006).
Take for example, the mishap of American Airlines flight 1420. In June 1999,
American Airlines flight 1420 overran runway 04R in Little Rock, Arkansas and fell into
a ravine killing 10 people including the captain. This accident occurred during a
thunderstorm and contains many of the characteristics involved in approach and landing
accidents (ALAs.) The ability of the pilot to determine critical information through
perceived visual cues of the outside environment may be limited by time of day and
various weather phenomena such as fog, rain, snow, and clouds. The Flight Safety
Foundation (FSF) conducted a study in 1998 with intentions of reducing approach and
landing accidents ("Killers in Aviation", 1998).
Various systems have been developed to reduce this accident rate and enhance
2
safety and overcome the issues with limited outside visibility for the pilot, such as radio
navigation, instrument landing systems (ILS), ground proximity warning systems
(GPWS). More recent developments include moving map displays, Global Positioning
System (GPS) capability for improved navigational accuracy, terrain awareness warning
systems (TAWS) and enhanced GPWS.
Regardless of the technological advancement, the aircraft information display
concepts require pilots to perform various mental transformations of display data to a
mental picture of what the outside environment may look like (Prinzel et al., 2002). For
example, the TAWS technology may help to mitigate some factors causing CFIT, its use
generally follows the information processing model of warn then act and, therefore
requires the pilot to be reactive rather than proactive in dealing with terrain hazards
(Prinzel, Comstock, Glaab, Kramer, and Arthur, 2002).
Snow and Reising (1999) stated that what is needed in terms of aircraft
information systems in intuitive technologies that improve pilot SA with respect to spatial
orientation (relative to terrain and flight path) without requiring the pilot to divert visual
attention and cognitive resources away from possible external events and primary flight
references. A proactive system that can help prevent (versus just warn a pilot of) a
potential collision with terrain is needed (Prinzel et al., 2004a).
NASA and its industry partners have designed and prototyped crew-vehicle
interface technologies that strive to proactively overcome aircraft safety issues due to
low-visibility conditions by providing the operational benefits of clear day flight through
cockpit displays, regardless of the actual outside visibility conditions (Bailey, Parrish,
Kramer, Harrah, and Arthur, 2002).
3
Enhanced and synthetic vision systems are technological advances that intend to
reduce pilot workload, especially during the final phase of flight. The titles are
occasionally used interchangeably; individually they are quite different and sometimes
competing technologies.
Synthetic Vision/System (SV/SVS) is a computer generated display image of the
out-of-the-cockpit scene topography based on aircraft attitude, high-precision navigation
instrumentation, and data on the surrounding terrain, obstacles, and cultural features.
SVS databases have been built to support this display technology with real-time integrity
in order to ensure pilot detection of real obstacles and to plan and verify accurate flight
navigation.
Prinzel et al. (2002) suggested that this display concept presents information to
the pilot with a level of accuracy and realism to flying under visual meteorological
conditions (VMC), regardless of the actual weather conditions. Laboratory and field
research experiments have successfully demonstrated both the safety and capability
benefits of SVS technologies in flight (Snow, Reising, Kiggett, and Barry, 1999), landing
(Prinzel et al., 2004a; Schnell, Kwon, Merchant, and Etherington, 2004) and taxi
operations (Wilson, Hoovey, Foyle, Williams, 2002). Prinzel et al. (2004a) suggests that
SVS display concepts are expected to reduce the occurrence of accident precursors
including:
• Pilot loss of attitude awareness
• Pilot loss of altitude awareness
• Pilot loss of vertical and lateral spatial awareness
• Pilot loss of terrain and traffic awareness on approach
4
• Confusion regarding escape or go-around path after recognition of
flight problem
• Pilot loss of SA in relating to the runway environment and
incursion
• Unclear flight path guidance on initial approach
An Enhanced Vision/System (EV/EVS) uses electronic sensors such as forward
looking infrared radar (FLIR) or millimeter-wave radar (MMWR) to augment or enhance
the natural vision while flying an aircraft. Such instruments are used to penetrate weather
phenomena such as rain, snow, fog, and haze.
In 2007 business jets began incorporating EVS displays as a night-vision
technology to complement other traditional navigational aids. Based on the development
of EVS technology the U.S. Federal Aviation Regulations (FAR), Section 91.175 was
amended such that pilots conducting straight-in approaches may now operate aircraft
below publicized Decision Heights (DH) or Minimum Descent Altitudes (MDA), when
using an approved EVS presented on a HUD (Bailey, Kramer and Prinzell, 2006).
EVS instruments are designed to help the pilot see permanently located
obstructions such as buildings, trees, towers, power lines, and terrain in general and are
normally displayed on a heads-up display (HUD). The intended use of EVS parallels
SVS in that both strive to improve safety and SA in low visibility conditions that may
otherwise cause major flight accidents and to provide the operational benefits of VMC
(Bailey et al., 2006).
EVS and SVS technologies are designed to improve situational awareness (SA)
5
for pilots flying the aircraft (PF) and non-flying pilots (NPF) alike. Investigative
interviews and the National Transportation and Safety Board (NTSB) research has shown
that accidents similar to the American Airlines crash, involve pilots that 1) are
overwhelmed with tasks during an IMC and/or night-time approach or; 2) have lost SA
during an IMC and/or night-time approach (Prinzel, Hughes, Arthur and Kramer, 2003).
It is hypothesized that EVS and SVS technology improvements will reduce the
rate of ALA accidents when compared to other navigational aids and traditional
instrument approach procedures by increasing SA and reducing pilot workload during
high-risk phases of flight such as the final approach to landing in low visibility situations.
Despite the technology’s infancy, NASA, the Federal Aviation Administration,
and many other private entities have tested this technology rigorously. The quantifiable
methods typically center on pilot workload and traditional physiological measurement
methods of stress and anxiety (Hughes, 2005c). Practical applications of this research
have promoted further mainstream research to determine safety improvements, allowing
for more efficient air carrier movement especially in congested areas susceptible to poor
weather. Financial effects can only be estimated until implementation is more complete.
Researcher’s Work Role and Setting
The researcher holds numerous Federal Aviation Administration (FAA) licenses
including Airline Transport Pilot (ATP) and a Boeing 737-300 type rating. Other
licenses and certificates include Airplane Single Engine Land (ASEL), Airplane Mutli-
Engine Land (AMEL), Military Instructor Pilot, Military Test Pilot, and Military Crew
Resource Management (CRM) Facilitator. He has served for 10 years as a Naval Aviator
with numerous sorties in wartime theaters.
6
The researcher became interested in improvements of SA during low visibility
approaches to landing while deployed to Misawa, Japan. The deployment spanned the
winter months, and Misawa, Japan is located in the northern part of the main Japanese
island of Honshu. During this deployment, the researcher encountered numerous whiteout
conditions while attempting to land in Misawa. The researcher was flying as copilot
during a hazardous emergency landing during a snowstorm onto a snow-covered runway.
During this landing, the emergency arresting cable was covered by snow and
unknowingly left rigged and unreported, resulting in significant damage to the aircraft
upon landing. From this incident spawned seven years of research and education on how
to improve safety of flight during the segment of the flight that requires the most
situational awareness and removal of cluttering information.
Statement of the Problem
The FAA has authorized synthetic vision systems and enhanced vision systems
for use in flight with certain limitations. The popularity and integration has risen sharply
in its relatively short lifespan. Many pilots have not yet been exposed to the training
required to become proficient in use of EVS or SVS yet exposure is destined to come.
Without question this technology will offer more information to the pilot when visual
cues are no longer available.
The purpose of this project was to determine if EVS and SVS systems improve
SA during the approach and landing phase of flight, or if the addition of the new
technology increased avionic clutter causing a decrease in SA. How is SA or pilot
workload measured? Do experimental results show a decrease in pilot workload? Do
the results show improved performance? The importance of this study showed whether
7
the implementation of additional technology in cockpit could reduce clutter and improve
the pilot’s ability to safely land an aircraft in less than VFR conditions.
Limitations
In general, conducting an experiment with real aircraft in real scenarios to
examine the effects of certain cockpit displays on human performance is difficult and
expensive. Even simulator experiments can prove to be costly and time consuming to
gather a large enough sample size of pilots, high tech instruments, etc.
Additionally, EVS and SVS are young technological advancements. Therefore, a
relatively small number of experiments or studies have been conducted on the effects on
human performance. The researcher detailed a meta-analytic approach to the relevant
experiments and studies in Chapter II regarding the early researchers involved in the
advent of synthetic and enhanced vision systems and their impact on pilot performance
and accident reduction.
Additionally, the researcher noted that the primary scientific study in his research
was Kim's simulator study (2009). Kim set the p-value at .05 to determine outliers that
occur by chance. However, Kim's sample size (see Table 3), involved 49 samples, which
would result in approximately 2 outliers with a p-value of .05. Perhaps Kim's results
would be more accurate with a p-value of .01.
8
Acronyms
ALAR – Approach and Landing Accident Reduction AGL – Above Ground Level ALA – Approach and Landing Accidents ALA – Approach and Landing Accidents ANOVA – Analysis of Variance CAA – Civil Aeronautics Administration CFIT – Controlled Flight Into Terrain CRM – Crew Resource Management DH – Decision Height DIME – Database Integrity Monitoring Equipment DV – Dependent Variables EV – Enhanced Vision EVS – Enhanced Vision System FAA – Federal Aviation Administration FAR – Federal Aviation Regulation FLIR – Forward Looking Infrared Radar FMS – Flight Management System FOV – Field of View FSF – Flight Safety Foundation FTE – Flight Technical Errors GPWS – Ground Proximity Warning System HDD – Heads-Down Display HITS – Highway-In-The-Sky HUD – Heads-Up Display ICAO – International Civil Aviation Organization IFD – Integration Flight Deck IFR – Instrument Flight Rules ILS – Instrument Landing System IMC – Instrument Meteorological Conditions MDA – Minimum Descent Altitude MMWR – Millimeter Wave Radar NASA – National Aeronautics and Space Administration ND – Navigation Displays NPF – Pilot Not Flying NTSB – National Transportation and Safety Board PF – Pilot Flying PFD – Primary Flight Display RMSE – Root Mean Square Error SA – Situational Awareness SAGAT – Situation Awareness Global Assessment Technique SART – Situation Awareness Rating Technique SAS – Statistical Analysis Software STARs – Standard Terminal Arrival Routes
9
SV – Synthetic Vision SVS – Synthetic Vision System SWORD – Subjective Workload Dominance TLX – Task Load Index USAF – United States Air Force VHF – Very High Frequency VFR – Visual Flight Rules VOR – Very High Frequency Omnidirectional Radio Range WAAS – Wide Area Augmentation System
10
CHAPTER II
REVIEW OF RELEVANT LITERATURE AND RESEARCH
History of Navigational and Landing Aids
In the early days of flight, there were no navigation aids to help pilots find their
way. Pilots flew by looking out of their cockpit window for landmarks or by using road
maps. These visual landmarks and maps were fine for daylight operations, but airmail
operated around the clock and the need for mail to be delivered by air expanded.
In 1919, U.S. Army Air Service Lieutenant Donald L. Bruner began using bonfires to
help with night navigation (Preston, 1998). In February 1921, an airmail pilot named
Jack Knight flew the first all-night flight from North Platte, Nebraska to Chicago using
bonfires lit by Post Office staff, farmers and the public (Preston, 1998).
In 1926, the Aeronautics Branch of the Department of Commerce took over
responsibility of building lighted airways and by 1933, 18,000 miles of airway had lights
and 1,500 beacons were in place (Preston, 1988). Regardless of the complexity of
lighting systems, pilots were still required to maintain visual contact with the ground.
Radio communications developed rapidly during this same period allowing for
weather reports to be passed via two-way radio signals and teletypewriter. This let pilots
plan their flight path and diverts around foul weather to maintain VMC. In 1929, Army
Lt. James H. Doolittle became the first pilot to use aircraft instrument guidance solely for
take off, fly a set course, and land. Lt. Doolittle used a four-course radio (Figure 1)
range and radio marker beacons to indicate his distance from the runway (Preston, 1988).
An altimeter displayed his altitude, and a directional gyroscope with artificial horizon
helped Lt. Doolittle control his aircraft’s attitude, without seeing the ground. These
11
technologies became the basis for many future developments in aviation navigation.
Figure 1. Four Course Radio Range
In May 1941, the Civil Aeronautics Administration (CAA) opened its first ultra-
high frequency radio range system for scheduled airline navigation, later expanding to
use such equipment to 35,000 miles of federal airways. In 1944, the CAA began testing a
static-free, very high frequency (VHF) omnidirectional radio range (VOR) (Figure 2)
that allowed pilots to navigate by watching a dial and needle on the instrument panel
rather than by listening to the radio signal audibly (Preston, 1988).
Figure 2. Very High Frequency Omnidirectional Radio Range (VOR)
By the middle of 1952, 45,000 miles of VHF and VOR airways, referred to as
Victor airways, supplemented the 70,000 miles of federally maintained low frequency
12
airways. The CAA began to shut down the low and medium frequency four course radio
ranges (Preston, 1998). In 1961, the FAA began using distance-measuring equipment
(DME) on its entire system. DME allowed aircraft to determine their distance from
known checkpoints in order to confirm their position. DME with VOR greatly improved
accuracy in instrument approaches during less than visual conditions and during
nighttime operations.
By 1973 the last airway light beacon from the system in the 1920s was shut down
and by 1982, the first of 950 new navigation aids equipped with solid-state construction
and advanced features was installed. With the advancement of radar implementation into
the air traffic control system, air traffic made large improvements in managing the
increasing flow of traffic while also increasing safety.
Developments in landing aids progressed parallel to developments in navigational
aids. Airports began using rotating lights at the landing field so they could be found after
dark. In the early 1930s, airports installed the earliest forms of approach lighting. These
indicated the correct angle of descent and the correct alignment with the runway. The
approach path was called glidepath or glideslope. The colors of the lights, their rates of
flash became standard worldwide based on International Civil Aviation Organization
(ICAO) standards.
Radio navigation aids assisted in landing, such as the four-course radio range, in
which the pilot was guided by the strength of Morse code signals. The introduction of
the slope-line approach system was a first in landing aids. Developed in the 1940s, the
aid consisted of lights in a row (Figure 3) that allows the pilot a simple funnel of two
rows that led to the end of the runway (Komons, 1989). The system was inexpensive to
13
build and operate and variations of this same lighting system are still used in airports
worldwide.
Figure 3. Early Forms of Runway Landing Aid Lighting; the top image shows a daytime view; the lower image illustrates a nighttime view
The instrument landing system (ILS) incorporated the best features of both
approach lighting and radio beacons with higher frequency transmissions. The ILS
painted an electronic picture of the glideslope onto a pilot’s cockpit instrument panel.
The first landing of a scheduled U.S. passenger airline using ILS was on January 26,
1938 as a Pennsylvania-Central Airlines Boeing 247 arrived in Pittsburgh from
Washington, D.C. and landed in a snowstorm using only the ILS system (Komons, 1989).
More than one type of ILS system (Figure 4) was tried and eventually the
14
adaption consisted of a course indicator (localizer) that showed drift either left or right
from runway centerline, glidepath to signify drift above or below glideslope, and two
marker beacons to show progress of the approach to the landing field or runway
(Komons, 1989). Approach lighting and other visibility equipment are part of the ILS
and also aid the pilot in landing. The ILS remains very similar to the initial system and
continues to be relied upon as an accurate instrument approach landing aid as of the
writing of this paper.
Figure 4. Elements of the Instrument Landing System
The advent of the GPS provided an alternative source of precision information for
instrument approaches. In the U.S., the Wide Area Augmentation System (WAAS) has
been available to provide precision guidance to Category I standards since 2007. Satellite
based avionics acted as the foundation for the majority of improvements to precision
instrument navigational supplements and landing aids.
Similar to the rapid changes in technological advances for navigational aids in the
1920s and 1930s, the 2000s witnessed rapid upgrades in possibilities to improve aircrew
and passenger safety. However, the addition of more and more information proved to be
more of a distraction in certain situations, such as cell phone use while driving, than of an
15
improvement. Therefore, technological progress was tempered with caution to eliminate
unnecessary information in circumstances that demand high levels of situational
awareness. This created a need for repeated and even continual experiments to measure
pilot performance while exposed to additional information with the objective to remove
unnecessary and distracting data.
In this section, various recent and historical research and studies pertaining to
EVS and/or SVS are reviewed from a human factors perspective and the influence on
situational awareness and pilot workload. It was the author’s intent to discuss the studies
and their assessment of specific features or configurations of EVS and/or SVS
technologies on pilot performance, in order to understand their effect on reducing pilot
workload and improving situational awareness during the approach and landing phases of
flight. Due to the difficulty in measuring situational awareness, the methods used during
these studies will also be discussed.
Developments in SVS
Throughout aviation history, developers have made attempts to improve safety and
increase the ability to successfully aviate and navigate through poor weather and poor
visibility conditions. For example, Jeppesen researched a detailed terrain database and
precision navigation sensors that would be a part of future-generation, three-dimensional
(3-D) flight management system (FMS.) The FMS would supply an on-board SVS with
position, altitude and heading data, along with terrain features, elevations and contours.
The SVS then projects a synthetic image of the outside world onto a HUD or HDD,
similar to the computer generated visuals in a flight simulator (see Figure 5) (Ramsey,
2004).
16
Early developments of SVS had to overcome an Achilles’ heel: access to an
accurate terrain database that could be approved for IFR flight. The U.S. military
controlled access to one of the most precise terrain databases ever developed: the Defense
Mapping Agency’s 90 meter terrain survey that was developed for cruise missile
guidance and strike aircraft attack missions. A major step forward in terrain mapping
came with the February 2000 launch of the space shuttle Endeavor and its 11-day shuttle
radar topography mission (Ramsey, 2004). This accelerated the development of a system
flying in general aviation (GA) aircraft. Coupled with Jeppesen navigation charts, the
system displays a highway-in-the-sky that guides pilots through a series of waypoints,
standard terminal arrival routes (STARS), and ILS approaches to landing. Terrain,
obstructions, and traffic can also be depicted on the display (Ramsey, 2004).
17
Figure 5. Universal Avionics SVS display
Developments in EVS
Enhanced vision systems are designed to use electronic sensors to augment or
enhance the natural vision while flying an aircraft (see Figure 6) (Hughes, 2005b). These
instruments are designed to help the pilot see permanently located obstructions, such as
buildings, trees, towers, power lines, and are normally displayed on a HUD.
18
Figure 6. EVS using FLIR on approach FLIR provides excellent EVS image resolution and is relatively affordable
technology. Also, FLIR systems are compact and installation requires only minor
modifications to the airframe. FLIR could provide pilots with a high-resolution display
of the terrain and runway environment when attempting to make an approach to an
isolated airport on a clear night. However, fog and clouds severely degrade FLIR
performance, rendering it almost useless especially if the clouds are dense.
Clouds and fog, on the other hand, don’t affect MMWR, although heavy
precipitation may reduce its effective range. Yet MMWR doesn’t have as high of a
19
resolution as FLIR. Compared to FLIR systems, MMWR systems are relatively
expensive and hard to adapt to civil aircraft because of their bulky antennas. A
MMWR’s ultra-short wavelength may require changes to an aircraft’s radome to ensure
that it is transparent to three and nine millimeter radio waves (George, 1995).
Situational Awareness Measurement
Due to the difficulty and ambiguity surrounding situational awareness, there are
also difficulties in measuring situational awareness during studies such as those discussed
in this paper. The research, experiments and studies reviewed in the following chapters
use different SA measurement methods.
Situational awareness subjective workload dominance (SA-SWORD) is used to
assess and compare the pilot SA when using two or more different cockpit displays or
interfaces. The SWORD technique is a subjective workload assessment tool that has
been used both retrospectively and predictively. SWORD uses subjective paired
comparisons of tasks in order to provide a rating of workload for each individual task.
When using SWORD, participants rate one task’s dominance over another in terms of
workload imposed.
Situational awareness global assessment technique (SAGAT) is an objective,
diagnostic and sensitive metric that is highly validated for use in a variety of applications.
SAGAT has been successfully used to directly and objectively measure operator SA in
evaluating avionics concepts, display designs and interface technologies. With SAGAT,
mission simulations are frozen randomly, the system displays are turned off and the
simulation is suspended while operators quickly answer questions about their current
perceptions of the situation. Operator perceptions are then compared to the real situation
20
to provide an objective measurement of SA.
The situational awareness rating technique (SART) is a direct self-rating measure
of SA that is more complex than a simple Likert scale. In 1989, R.M. Taylor developed
the SART by eliciting knowledge from pilots and aircrew. Through statistical techniques
he created the SART which consists of ten dimensions of questions to measure SA.
Cooper-Harper rating scale is a set of criteria used by pilots and flight test engineers
to evaluate the handling qualities of aircraft during flight tests. The scale ranges from 1
to 10, with 10 indicating the worst handling characteristics and 1 indicating the best. The
criteria are evaluative and therefore the scale is considered subjective.
The U.S. Air Force’s Revised Workload Estimation Scale (a 7-point Likert scale) is
a subjective technique used to measure mental workload in comparison with two or more
dependent variables.
NASA Task Load Index (NASA-TLX) is a subjective workload assessment tool.
NASA-TLX allows users to perform subjective workload assessments on operators
working with various human-machine systems. NASA-TLX is a dimensional rating
procedure that derives an overall workload score based on a weighted average of ratings
on six subscales. These subscales include mental demands, physical demands, temporal
demands, own performance, effort and frustration.
Root mean square error (RMSE) is a frequently used measure of the differences
between values predicted by a model or an estimator and the values actually observed
from the subject being modeled or estimated. In Kim’s study, RMSE was used to
measure the errors in flight path control by calculating the distance away from centerline
an aircraft or simulator drifted during the study.
21
Prior Research of EVS and SVS
The use of EVS and SVS technology is expected to reduce aircraft accidents, in
particular CFIT, due to the display information enhancing pilot SA under low visibility
conditions, similar to a night-time or inclement approach to landing. Several studies have
been conducted to assess this expectation and to investigate the effect of specific
EVS/SVS features on pilot performance. In general, studies have focused on the effects
of display sizes and corresponding field of view (FOV), guidance images, and tunnel
images on flight path tracking performance, SA, workload, or subjective display ratings
(Kim, 2009).
Researchers such as Prinzel, Schnell, Arthur, Bailey, Hughes, McKinley, Kramer
and others used these measurement techniques to assess the pilot’s ability to safely
maintain an aircraft in stable flight through experiments with symbology, ground
proximity, instrument approach precision, and other variables discussed below. The
previous research has mainly focused on portions or certain aspects of the
implementation of new displays with more information into the cockpit. Sang-Hwan
Kim’s (2009) experiment combined the research into a measurement of all aspects into a
quantifiable study on how pilots respond to an efficient display of additional information.
The results of their work created the basis for Kim’s simulator experiment in 2009, which
combined numerous variables into one study.
Early SVS and EVS studies and articles were centered on the rapidly changing
technology and its integration into mainstream aviation. Prinzel et al. (2002) conducted
flight tests to evaluate the effects of three display concepts. These design concepts were
tested with the primary objective to determine which, if any, of the concepts improved
22
SA and reduced pilot workload in IMC conditions on final approach. In general, Prinzel
et al. confirmed the hypotheses that SVS would provide safety and performance benefits
over traditional navigational instruments (2002).
With a similar objective, Prinzel et al. in another study in 2004 conducted two
experiments to examine the efficacy of SVS displays and to develop field of view (FOV)
and terrain texture recommendations for cockpit display design. In one of their
experiments, they investigated the effects of different types of displays for presenting
SVS information, two types of textures (photorealistic and generic) and two runway
conditions on performance, subjective preference ratings, workload and SA (using SA-
SWORD). Results demonstrated that the different display sizes did not affect flight
performance and that the use of the HUD for presenting SVS information reduced lateral
path error, as compared to the HDD. A reduction in lateral path error led to confirming
their hypothesis that a HUD would reduce pilot workload and improve situational
awareness over a HDD.
SVS for CFIT Prevention
The studies reviewed above have focused on nominal flight operations; however,
other research has been conducted to examine the efficacy of SVS technology for CFIT
prevention in off-nominal situations (Prinzel et al, 2003). In an experiment by Prinzel et
al. (2003), 10 display concepts, including two baseline conditions (a round-dials display
and a primary flight display), and various SVS textures were used to assess operator
CFIT detection. Results revealed that the use of SVS, in general, improved CFIT
detection.
In a second experiment, Prinzel et al (2003) evaluated four display concepts by
23
measuring flight performance, SA and workload during a go-around situation.
Situational awareness was measured using the SART and SA-SWORD methods.
Workload was measured using modified Cooper - Harper ratings. Results confirmed that
the use of the SVS allowed pilots to detect CFIT more efficiently than baseline concepts.
SVS Symbology
These experiments demonstrated the general efficacy of the SVS concept.
Consequently, the effects of guidance and tunnel images, combined with SVS
technology, were investigated. Prinzel et al. (2004c) conducted two experiments to
compare different tunnel and guidance symbology concepts for synthetic vision display
systems presented on HDDs and HUDs. They evaluated the efficacy of these concepts
during complex, curved approaches under instrument meteorological conditions (IMC).
In the first experiment, they focused on a SVS primary flight display (PFD) and
examined four tunnel concepts compared to a baseline (no tunnel) configuration. They
also assessed three guidance symbologies by measuring mental workload using the
United States Air Force’s (USAF) Revised Workload Estimation Scale, SA using SART
and SA-SWORD, a subjective questionnaire, and RMSE flight path control measurement
(Kim, 2009). The results of the first experiment revealed the baseline condition to be
worse than other conditions including tunnel concepts, in terms of path control, workload
and SA. The second experiment evaluated two pathway tunnel concepts and two forms
of guidance for a HUD. Overall, the results demonstrated that presenting any kind of
tunnel feature could produce better performance in terms of RMSE, workload and SA.
Schnell et al. (2004) also evaluated a SVS HDD against traditional navigational
aids through conventional glass cockpit displays to assess whether SVS technology could
24
improve pilot performance, SA and workload. Schnell et al. included navigation displays
(ND) in their simulation setup for providing pilots with more realistic flight situation
information. SA was measured using SAGAT, mental workload using the NASA-TLX
workload assessment tool, flight technical errors (FTE) and eye movements of pilots
when using three different configurations of flight decks.
The configurations included a conventional PFD with displays, a SVS PFD with
navigational displays, and a conventional PFD with an external display. This is an
exocentric display that depicts the planned flight path in the context of the surrounding
terrain. The depiction is centered on the aircraft (Kim, 2009). Results demonstrated, in
general, the use of the SVS display format to improve pilot performance by generating
reduced flight technical errors (FTEs), lower workload scores and short overall visual
scan length. Interestingly, there was no significant difference in SA across display
conditions. That is, the SVS PFD with terrain representation did not seem to improve the
terrain awareness of the pilot.
Schnell et al. (2004) deduced that pilots relied on and trusted the pathway tunnel
to the extent that they did not feel they needed to devote much attention to the aircraft-
terrain situation. Schnell et al. (2004) generalized that pilot workload measures were
lower in the SVS condition than with the conventional PFD.
Problems with using SVS technology included standardizing the quality of
database-oriented system information and controlling the quality of matching real and
synthetic information closely enough to certify the database for practical applications.
Bailey et al. (2002) introduced two possibilities of reducing SVS information errors: 1)
development of a complimentary system called Database Integrity Monitoring Equipment
25
(DIME); and 2) incorporating use of EVS, real-time and non-database elements, blended
into the SVS display.
Blending EVS and SVS
EVS and SVS have been perceived as separate technologies, both designed to
improve pilot situational awareness but neither technology completely provided the entire
picture outside the cockpit. NASA developed a Sensor Enhanced – SVS (SE – SVS)
concept, which utilizes the beneficial aspects of EVS and SVS while mitigating the
negative aspects of each concept (e.g., calibration in SVS images with terrain and poor
EVS display quality due to meteorological conditions) (Bailey et al, 2002).
Bailey, Kramer and Prinzel (2006) compared the general effects of blending
EVS/SVS concepts with and without pathway tunnel images. The comparisons focused
on the fusion of the two displays during low-visibility and landing operations. In the
experiments, four HUD display concepts were tested. This included simulated approach
to landing with the use of SVS displays on approach then transitioned to an EVS image at
500’ above ground level (AGL). This transition was designed primarily because an EVS
camera can only give a usable image when the aircraft is low and close to the ground
(Bailey et al., 2006). Bailey et al. (2006) measured flight path errors and pilot control
inputs during each experimental trial. They also collected subjective questionnaires,
workload ratings, and SA ratings using SA – SWORD and SART. The results of the
study showed that significant improvements in pilot SA without increases in workload
could be provided by the fusion display and the pathway tunnel image (Bailey et al.,
2006).
These results confirmed previous studies (Alexander, Wickens and Hardy, 2003;
26
McKinley, Heidhausen, Cramer and Krone, 2005; Wickens, Horrey, Nune and Hardy,
2004), which showed a synthetic tunnel image to improve flight performance. Through
these studies, it can be stated that the critical objective of improving flight performance
and reducing pilot workload might be largely facilitated by the presentation of tunnel
images in a HUD (Kim, 2009).
Statement of Hypothesis
To summarize, the studies mentioned in this literature review support evidence
that advanced synthetic images in cockpit displays, including blended or fused images
with EVS, synthetic tunnels and terrain features for approach to landing, improve flight
performance and/or pilot SA, and reduce workload (Alexander et al., 2003; McKinley et
al., 2005; Prinzell et al., 2002, 2003; Wickens et al., 2004). Highway-in-the-sky (HITS)
or tunnel images have demonstrated to be a significant factor in improving pilot
performance, particularly flight path accuracy (Alexander et al., 2003).
The findings of combined terrain feature utilities SVS – HUD and EVS – HUD in
Kim's study were hypothesized to improve pilot SA and generated lower mental
workload for pilots than traditional navigational aids (Arthur et al., 2005). Additionally,
the null hypothesis was determined that Kim's experiment would show no change in the
results following the baseline simulator flights as compared to the other test flights.
27
CHAPTER III
RESEARCH METHODOLOGY
Research Model
The Graduate Capstone Project included a meta-analytic approach of multiple studies
and experiments in collaboration to answer the hypothesis. The project employed a
descriptive quantitative research model involving a laboratory experiment conduct by
S.H. Kim in 2009 as the primary focus. Kim's results were collected via SAGAT
measurement devices and three dependent variables (DV) were analyzed with a one-way
analysis of variance (ANOVA) and f-tests.
A meta-analytic approach is a statistical examination of multiple scientific studies and
not an actual scientific study itself. This approach can be used in situations where large
sample sizes are not available, such as in this case with few scientific studies based on the
dependent variable of situational awareness. The strength of a meta-analytic approach
can more powerfully estimate the true effect size as opposed to a smaller effect size
derived in a single study under a given set of assumptions and conditions. The weakness
of this approach was that the sources of bias were not controlled by the method.
The experiment was first developed in consideration of the data collection method
that would be most valid. As previously mentioned, measuring SA is difficult and the
methods are limited. The SAGAT method was employed to quantify the results of Kim’s
experiment (2009).
Survey Population
The sample size in Kim's study was eight pilots recruited to participate in the lab
experiment. The requirements for selection included previous flying experience in
28
commercial aircraft with glass cockpit displays. Due to the relatively recent
developments in EVS and SVS, experience with the use of this technology was not
expected. Data from the demographic survey for the eight pilots include: all pilots were
male; the average age was 58.6 with a standard deviation of 14.4 years; all pilots had
glass cockpit experience; the average flight hour experience was 11,043.8 hours with a
standard deviation of 7,893.1; three pilots had experience with a HUD display in either
actual flight or in a simulator; two pilots had experience with SVS systems (mean of 4
hours) and one pilot had EVS experience (6 hours) in a simulator (Kim, 2009).
Sources of Data
Primary sources of data include task load measurement study conducted by Kim in
2009 at North Carolina State University. Other studies, as cited previously, have been
conducted on specific methods or specific modes of EVS/SVS. However, Kim’s study
(2009) was most appropriate for the specific task of testing the hypothesis that pilot
workload during approach and landing is decreased with the use of EVS and SVS.
The Data Collection Device
Each pilot in Kim’s study completed nine trials. Eight of these trials followed a
within-subjects variable design and one additional trial for collected subject verbal
protocols to measure cognitive task analysis. The data from the first eight trials were
collected using SAGAT measurements of situational awareness. Responses to SAGAT
queries represent a binomial variable – either correct or incorrect. The nature of this
measurement violates quantified statistical test assumptions. With the use of a valid
arcsine function, the data was transformed and traditional parametric data analysis can be
satisfied (Kim, 2009). One-way ANOVAs were then conducted on the transformed
29
SAGAT scores for overall SA, SA by levels and SA by types.
The ninth and last trial data was collected through verbal questioning and
measurement based on the NASA Task Load Index (NASA-TLX) model. Another one-
way ANOVA was used to normalize the rating scores for individual workload scaling.
In addition, all nine trials included a physiological workload measurement – incremental
heart rates (∆HR) – compared to the HR during rest (baseline HR) (Kim, 2009). A single
variable ANOVA was conducted to analyze the effect of display conditions, visibility,
nighttime effects on HR.
Instrument Pretest
All data collection devices were pre-tested to ensure their accuracy and
comprehensiveness. Kim’s experiment (2009) was conducted on a low-fidelity simulator
developed by Integration Flight Deck (IFD) inputs from NASA’s Langley Research
Center, which provided pilots with a full-mission simulator. An ex-Air Force C-130
check pilot with experience with advanced HUDs and experience with NASAs IFD
simulator tested the different HUD configurations in each trial and validated the
simulator.
Table 1. Pilot Flight Time Data in Hours
Pilot Hours Mean Standard Deviation
Total Instrument Time 3781.3 4064.7
Total Night Time 3660.0 3471.8
Total Flight Time 11043.8 7893.1
Total Time Last 12 Months 230.6 204.8
30
Instrument Reliability
SAGAT is the most widely validated of all SA measurement techniques (Endsley,
2000). According to Endsley (2000) numerous studies have been performed to assess the
validity of the SAGAT method and the evidence suggests that it is a valid metric of SA.
The study by Endsley (2000) included two sets of simulation trials on four fighter pilots
with mean scores of .98, .99, .99 and .92.
The SAGAT method, when used to measure SA through the low-fidelity simulator,
maintains its reliability through simplified binomial variable testing. Either correct or
incorrect responses were measured through pilot inputs on the simulator and then
collected and stored on the simulator’s PC-based computer.
Procedures
Kim’s simulator experiment was conducted on four different segments of a typical
commercial flight. The pilots flew these segments first without EVS/SVS to establish a
baseline. Then the pilots flew using SVS and EVS separately followed by a combination
of EVS and SVS. The dependent variables (DV) measured during these trials were
spatial awareness, system awareness and task awareness, which are all functions of SA.
Treatment of Data
All statistical analyses were performed using Statistical Analysis Software (SAS) (Kim,
2009). Residual plots were used to examine and verify linearity, constant variance of
error terms, and independence of error terms and normality of error term distribution
(Kim, 2009). An alpha level of 0.05 was used to identify any significant effects and
interactions.
31
CHAPTER IV
RESULTS
Flight Path Control Performance
The results of the experiment are listed in the tables below including the ANOVA
results on RMSE data for each of the display configurations (baseline, SVS, EVS,
combination), ANOVA results on pilot SA (overall and by SA type – spatial awareness,
system awareness, and task awareness), and ANOVA results for pilot workload (Kim,
2009).
Flight path control performance is illustrated in Table 2 and in Figure 7.
Table 2 ANOVA Results for RMSE Data
Source df Type III SS Mean Square F P Display 3 0.324819 0.108273 14.37 <.0001 IMC 1 0.458698 0.458698 60.89 <.0001 Leg 3 0.021173 0.007058 0.94 0.4236 IMC Leg 3 0.0582812 0.017604 2.34 0.0746 Display IMC 3 0.211148 0.070383 9.34 <.0001 Display Leg 9 0.176666 0.01963 2.61 .0007 Display IMC Leg 9 0.07069 0.007854 1.04 .407
32
Figure 7. RMSEs for each display configuration
ANOVA results (see Figure 8) also revealed that there was a significant effect of
the IMC condition on tracking performance (F(1,223)=60.9, p<.0001). The IMC-day
condition (M=14.3) was associated with greater errors in the tracking task than the IMC-
night condition (M=12.0) (Kim, 2009).
`
12.9
15.7
11.3
14.3
0
2
4
6
8
10
12
14
16
18 RM
SE
Display Con0iguration
RMSEs for Display Con0iguration
Baseline
SVS
EVS
Combo
33
Figure 8. RMSEs for IMC Conditions.
There was no significant effect of leg and no interaction of IMC condition and leg
on RMSE. However, ANOVA results revealed an interaction effect among the display
and IMC conditions. Figure 9 shows the interaction plot. This graph indicates the SVS-
HUD under IMC-day condition produced higher RMSE than the same HUD during IMC-
night conditions. In general, IMC-day conditions were associated with higher RMSE
then IMC-night conditions (Kim, 2009).
0.0
2.0
4.0
6.0
8.0
10.0
12.0
14.0
16.0
Visibility Con=iguration
RMSE
RMSEs for IMC Condition
IMC-‐Day
IMC-‐Night
34
Figure 9. RMSEs for display configuration by IMC condition. ANOVA results also revealed a significant interaction effect between display and
flight leg on RMSE (F(9,223)=2.61, p=.007). Figure 10 shows the RMSEs for each
display configuration for the four simulator legs. The SVS-HUD yielded higher RMSEs
and the EVS-HUD produced lower RMSEs across legs. However, the Combo-HUD in
Leg 4 generated higher tracking error than the other displays in other legs. There was no
three-way interaction effect among the experimental manipulations (F(9,223)=1.04,
p=.4070).
12.8
19.0
12.5 14.2
12.1 11.7 10.4
12.0
0.0
2.0
4.0
6.0
8.0
10.0
12.0
14.0
16.0
18.0
20.0
Basleline SVS EVS Combo
RMSE
Display Con0iguration by IMC Condition
IMC Day
IMC Night
35
Figure 10. RMSEs for leg by display configuration.
Pilot Situational Awareness
Results of an ANOVA applied to pilot SA revealed significant main and
interaction effects of display configuration, IMC condition, and leg of flight on SAGAT
scores including overall SA, and for various levels of SA. Table 3 presents a summary of
F-test results on the situational awareness SAGAT scores.