A Glimpse at Archie: The LOSA Archive Ashleigh Merritt, Ph.D. The University of Texas Human Factors Research Project (UT) First ICAO Global Symposium on.

A Glimpse at Archie: The A Glimpse at Archie: The LOSA ArchiveLOSA Archive

Ashleigh Merritt, Ph.D.Ashleigh Merritt, Ph.D.

The University of Texas Human The University of Texas Human Factors Research Project (UT)Factors Research Project (UT)

First ICAO Global Symposium on TEM & NOSS in ATC

9-10 November, 2005, Luxembourg9-10 November, 2005, Luxembourg

What is Archie?What is Archie?

““Archie” is the LOSA Archive Archie” is the LOSA Archive A database of de-identified airline data A database of de-identified airline data

which is maintained and updated by UTwhich is maintained and updated by UT

Airlines that do a LOSA with The LOSA Airlines that do a LOSA with The LOSA Collaborative contribute their data to Collaborative contribute their data to Archie for research purposes Archie for research purposes

Archie provides the airline perspectiveArchie provides the airline perspective

How old is Archie?How old is Archie?

Five LOSAs in five years to Five LOSAs in five years to develop the methodology develop the methodology

Data in the Archive starts in Data in the Archive starts in 20002000

Had to wait for Archie to grow Had to wait for Archie to grow before doing any before doing any benchmarking or industry-benchmarking or industry-wide analyses.wide analyses. If we think of Archie as a dog, he If we think of Archie as a dog, he

would be a young bloodhoundwould be a young bloodhound

How big is Archie?How big is Archie?

As of September 2005, Archie has data fromAs of September 2005, Archie has data from 23 airline LOSAs23 airline LOSAs 4800 flights4800 flights

4800 flight narratives written by trained 4800 flight narratives written by trained observersobservers

4800 ratings of TEM countermeasure 4800 ratings of TEM countermeasure performanceperformance

17,500+ threats17,500+ threats 12,500+ errors12,500+ errors 2,400+ undesired aircraft states2,400+ undesired aircraft states

Why is Archie here?Why is Archie here?

To show what is possible when you To show what is possible when you build an archive of observational build an archive of observational data based on TEMdata based on TEM

To show what is possible when two To show what is possible when two TEM archives talk to each otherTEM archives talk to each other

Archie’s AttributesArchie’s Attributes

1.1. Archie is loyal and Archie is loyal and faithful faithful

2.2. Archie is a persistent Archie is a persistent and diverse tracker and diverse tracker

3.3. Archie plays well Archie plays well with otherswith others

After five years, Archie remains After five years, Archie remains loyal to its aviation masterloyal to its aviation master

The Archive is housed at UTThe Archive is housed at UT All data are de-identified as to All data are de-identified as to

airline and individualairline and individual No airline has been publicly identifiedNo airline has been publicly identified No individual pilots can be or ever have No individual pilots can be or ever have

been identified from the Archivebeen identified from the Archive

Archie is a good trackerArchie is a good tracker

ATC threats ATC threats Intentional Intentional

NoncomplianceNoncompliance Outstanding Outstanding

performanceperformance

Tracking ATC threat chainsTracking ATC threat chains

Data are based on 10 LOSAs – 2426 Flights – 9450 threats

2350 ATC threats = ¼ of all threats = about 1 per flight

236 mismanaged ATC threats = 30% of all mismanaged threats 10% of all ATC threats were

mismanaged

Mismanaged ATC threatsMismanaged ATC threats

2426 Flights – 9450 threats

236 mismanaged ATC threats

Challenging clearances

(53%)

Runway Changes (14%)

Language difficulty (8%)

Similar call signs (5%)

ATC error (7%)

ATC threats -> ATC threats -> Crew ErrorsCrew Errors

2426 Flights – 9450 threats

236 mismanaged ATC threats

280 errors

Automation errors (20%) – wrong

MCP/FCU altitude setting dialed

Communication errors (27%) -

misinterpret ATC instructions

Aircraft handling (24%) –

unintentional speed, lateral, vertical

deviations

Cross-verification errors (7%)

Ground navigation errors (6%)

ATC threats -> ATC threats -> Crew Errors -> Crew Errors -> UASUAS

2426 Flights – 9450 threats

236 mismanaged ATC threats

280 errors

56 Automation errors

76 Communication errors

67 Aircraft handling errors

19 Cross-verification errors

16 Ground navigation errors

2UAS

58UAS

5UAS

9 UAS

21UAS

ATC threats ATC threats -> UAS-> UAS

2426 Flights – 9450 threats

236 mismanaged ATC threats

280 errors

106 UAS

Lateral deviation 17%

Speed too high 14%

Vertical deviation 10%

Unstable Approach 10%

Continued Landing 9%

Incorrect Automation configuration 9%

Taxiway/runway incursion 6%

Speed too low 6%

Incorrect Aircraft configuration 5% Other 14%

Bottom line: 4% of flights had a

UAS arising from an ATC threat that was

mismanaged. ¾ occurred during

descent/approach/land

Archie & Intentional Archie & Intentional NoncomplianceNoncompliance

Across 22 LOSAs, the average is 40% of Across 22 LOSAs, the average is 40% of flights with one or more noncompliance flights with one or more noncompliance errorserrors

Range: 23% - 90%Range: 23% - 90% Most common noncompliance errors:Most common noncompliance errors:

checklist performed from memory / nonstandard checklist performed from memory / nonstandard checklist usechecklist use

failure to cross-verify MCP/FCU altitude alerter failure to cross-verify MCP/FCU altitude alerter changes changes

PF makes own MCP/FCU changesPF makes own MCP/FCU changes Seems pretty trivial stuff, eh?Seems pretty trivial stuff, eh?

Noncompliance correlates Noncompliance correlates with..with..

Based on 22 LOSAs, airlines with higher rates Based on 22 LOSAs, airlines with higher rates of intentional noncompliance also have more of intentional noncompliance also have more flights with:flights with:

Mismanaged threats (r = .7) Mismanaged threats (r = .7) Mismanaged handling errors (r = .9)Mismanaged handling errors (r = .9) Undesired aircraft states (r = .9)Undesired aircraft states (r = .9) Mismanaged UASs (r =.8)Mismanaged UASs (r =.8)

Noncompliance – a measure of safety Noncompliance – a measure of safety culture?culture?

Archie was shocked…

Archie is a diverse trackerArchie is a diverse tracker

The richness and complexity of the The richness and complexity of the data allow literally thousands of paths data allow literally thousands of paths to be pursuedto be pursued

When did it happen, was it Captain or When did it happen, was it Captain or First Officer flying, how was it managed, First Officer flying, how was it managed, what else was going on, what happened what else was going on, what happened next, has it happened before, does it next, has it happened before, does it happen often, does it happen more on happen often, does it happen more on certain aircraft, does it happen at certain aircraft, does it happen at particular airports, etc. etc. etc. particular airports, etc. etc. etc.

Archie can dig and dig… Archie can dig and dig…

Test Your UnderstandingTest Your Understanding

A.A. Archie would find the flights that had Archie would find the flights that had above average operational complexity above average operational complexity (4 or more threats), and that the (4 or more threats), and that the crews managed very, very well.crews managed very, very well.

In the current Archive of 4800 In the current Archive of 4800 flights, there are 102 of these flights, there are 102 of these defined flights. We can read the defined flights. We can read the flight narratives to explore best flight narratives to explore best practices…practices…

Q.Q. What would you get if you asked Archie to find all What would you get if you asked Archie to find all the flights that have four or more threats, zero the flights that have four or more threats, zero mismanaged threats, zero crew errors, zero UAS, mismanaged threats, zero crew errors, zero UAS, and the observer rated the crew “outstanding” for and the observer rated the crew “outstanding” for overall effectiveness?overall effectiveness?

GOOD BOY, GOOD BOY,

ARCHIE!ARCHIE!

Archie plays well with othersArchie plays well with others

Queries from / data-Queries from / data-sharing with: sharing with:

ATC (NOSS ATC (NOSS group)group)

IATA/ICAO (ITA)IATA/ICAO (ITA) Boeing Boeing Airlines Airlines Incident Incident

Reporting Reporting systemssystems

Transport Safety Transport Safety BoardsBoards

Archie plays well with othersArchie plays well with others

LOSA/TEM/NOSS data can LOSA/TEM/NOSS data can complementcomplement other safety management dataother safety management data

Flight Data Recorder is the aircraft Flight Data Recorder is the aircraft perspective perspective

Incident reporting is the actor’s Incident reporting is the actor’s perspective perspective

LOSA/NOSS is a neutral third party LOSA/NOSS is a neutral third party perspectiveperspective

We’ve never really had that before.We’ve never really had that before.

One Big Advantage: Pilot-ATC see One Big Advantage: Pilot-ATC see the others’ world!the others’ world!

ConclusionsConclusionsYou have to be patient while the methodology You have to be patient while the methodology

matures and the Archive grows, but once matures and the Archive grows, but once in place:in place:

You can benchmark within and across You can benchmark within and across facilitiesfacilities to determine a to determine a facility’sfacility’s safety safety strengths and vulnerabilitiesstrengths and vulnerabilities

You can benchmark across the You can benchmark across the industryindustry to to determine determine systemicsystemic strengths, strengths, vulnerabilities, and best practices within vulnerabilities, and best practices within the industrythe industry

A TEM-based Archive would allow ATC to A TEM-based Archive would allow ATC to “converse” freely with the LOSA Archive “converse” freely with the LOSA Archive on matters of mutual interest to pilots and on matters of mutual interest to pilots and controllerscontrollers

The University of TexasThe University of TexasHuman Factors Research ProjectHuman Factors Research Project

www.psy.utexas.edu/HumanFactors