AAM Ecosystem Aircraft Working Group: Autonomy Scalability

AAM Ecosystem Aircraft Working Group: Autonomy Scalability

AgendaMay 27, 2021

3:00pm - 4:30pm ET

TIME (ET) TOPIC SPEAKER(S)

3:00PM – 3:10PM Welcome Carl Russell, NASA

3:10PM – 3:30PM The Challenge: Scalability of

Automation

Wes Ryan, NASA

3:30PM – 4:20PM Open Guided Discussion with the

Audience, including:

• Active Participates: MS Teams chat and

open microphone

• Listen Only Participants and Polls:

https://arc.cnf.io

Carl Russell, NASA

Wes Ryan, NASA

4:20PM – 4:30PM Next Steps & Closing Remarks Carl Russell, NASA

Wes Ryan, NASA

Platform and Discussion

• Active Participants– Platform: MS Teams– Discussion: MS Teams microphone, chat, and “Raise your hand” functions

• Leave your cameras/webcams off to preserve WiFi bandwidth• Use your mute/unmute button (e.g., remain on mute unless you are speaking) • Enter comments/questions in the chat• Click the “Raise your hand” button if you wish to speak• Say your name and affiliation before you begin speaking

• Listen Only Participants and Polling– Platform: YouTube Live Stream

• Go to https://nari.arc.nasa.gov/aam-portal/ for the link– Discussion and polling: Conferences.io

• Enter https://arc.cnf.io/ into your browser and click on “Aircraft Working Group: Autonomy Scalability”

• Questions will be addressed if times permits or at the facilitator’s discretion

3

Presentations & Discussion

National Aeronautics and Space Administration

www.nasa.gov

Aircraft Working Group: Autonomy Scalability

Wes Ryan, NASA Aeronautics Research Institute (NARI)

Carl Russell, NASA Revolutionary Vertical Lift Technology (RVLT) Project

May 27, 2021

Meeting Purpose/Goal

• Facilitate a discussion on the scalability of automation towards autonomy – Methodical Growth - Ops like Wing to AAM

• Focusing on the art of the possible while also focusing on the future – Logical Path Towards Future Capabilities

• Identify Areas for Further Discussion/Work/R&D/Collaboration

• What is not being done? Key gaps in technology/design/ops

• What in the problem scope, assumptions, or barriers must be changed/removed?

• Goal: Honest Advocates for Capability of Technology and Safety

Live Content Slide

When playing as a slideshow, this slide will display live content

Poll: What is the greatest barrier, in your opinion, to scalability/autonomy?

Today’s Key Focus Areas

• Assurance and Trustworthiness – What do they mean for automation on the aircraft and how do we obtain them?

• Are they different from UTM to AAM?

• Technology/Regulatory Maturity - Path and Process – How do we certify task-based automation towards future autonomy?

• Will regulations need to change before some can be certified?

• Architecture and Resiliency – Can current architectures and designs get to the level where the human is no longer critical?

• Getting to UML-4 – How do we methodically move from where we are now to the needs/capability for UML-4?

• Additional Suggestions?

Live Content Slide

When playing as a slideshow, this slide will display live content

Poll: What research gaps exist?

Supporting Slides

Focus on Aircraft Automation• Is technology mature enough to move from

automation to autonomy? (no human intervention)

• Beyond “levels” of automation, focus on tasks and technology readiness/assurance for each task

• Maturity - Move From Sub-Scale Technical Demonstration to Civil Readiness

• Civil Trustworthiness – Not just trust in intended function (engineering), but trust in resilient proficiency (real-world robust proficiency)

• Assurance - Design and Certification May Need to Evolve – Smarter Architectures

• Goal - Create Automation “Supply Chain” of Viable Products –DAA, C2, Flight Path Management, etc.

Methodical Development/Fielding Buildup Progression

Phase(MTE)

Function CurrentPart 91

CurrentAircraft

Step A Step B Step •••• Step •••• Step ?

ALL Phases

Contingency Pilot Pilot Pilot Pilot Pilot Pilot Auto?

Preflight Flight Plan Pilot Pilot Pilot Pilot Auto Auto Auto

Preflight Walk around Pilot Pilot Pilot Pilot Auto Auto Auto

Ground Ops

Taxi Pilot Pilot Pilot Auto Auto Auto Auto

Takeoff Takeoff Pilot Pilot Pilot Auto Auto Auto Auto

Enroute Aviate Pilot Pilot Auto Auto Auto Auto Auto

Enroute Navigate Pilot Pilot Pilot Auto Auto Auto Auto

Enroute Communicate Pilot Pilot Pilot Pilot Pilot Auto Auto

Enroute VFR-like Separation

Pilot Pilot Pilot Pilot Pilot Auto Auto

Approach Approach Pilot Pilot Pilot Pilot Auto Auto Auto

Approach Missed Pilot Pilot Pilot Pilot Auto Auto Auto

Landing Landing Pilot Pilot Pilot Pilot Pilot Auto Auto

• Identify Mature Technology to Perform Specific Functions

• Automate Mature Functions First

• Human Pilot Remains Responsible for Fewer and Fewer Functions as Automation is Introduced Safely

• Human May Always Be Needed for Contingency Management and Overall Safety of Flight

Related Tech and Regulatory Paths

Intended Function & Use Case

Identify Available

Technology, Procedures

Identify Development & Operational Path For Use

Flight Test & Data

Collection

Pass/Fail, Showing of Compliance

Applicable Regulations,

Policy, Procedures

Set Clear Roles for

Human and Machine

Do They Need To Change?

Technology - Functionally Based Approach to Product Development

Regulation - Functionally Based Approach to Policy & Regulation

FAA Rulemaking

Industry Standards

Consider Technology and

Policy. Are Development Timelines and Assumptions

Viable?MUST ADDRESS BOTH

Evolution of Trustworthiness

Student Pilot

Scenario–based Training With Instructor, Repetition, & Expected Outcomes/Bounded Behavior

Human

Initial Aptitude and Skill

Private Pilot

Commercial Co-Pilot

Commercial Pilot

Basic Proficiency

Resilient Proficiency

Civil Trustworthiness

Experimental Prototype

System

Initial “Aptitude and Skill” in

System Function

Advisory System

Assistive System

Safety Critical System

Proven Basic Proficiency

Proven Resilient Proficiency

Proven Civil Trustworthiness

Simulation & Flight Test to Demonstrate Readiness for Intended Use, Type of Operation, Task Criticality

• Compare Risk-based, Model Based Automation Development to Pilot/Crew Development Process

• How Do We Build In Proficiency, Robust Function?

Must Work-up to Resilient/Robust Assurance in Automation Designs With Bounded Behavior

Suggested Autonomy Path Forward• What: With Industry, Identify Design, Architecture, and

Safety Assurance Steps for Safely Reducing/Removing Pilot Responsibility for Certain Tasks

• What must be different?: We focus on UML 4, but Need New Design and Test Techniques for Civil Trustworthiness of Core Enablers, and Informed Certification Steps/Requirements to Move Forward (i.e. UML 0 to 4)

• What is done Now?: Development is Done by Individual Companies at Their Own Expense Independently

• What’s the Fix?: Broad Industry Agreement on Smarter/Redundant Architectures Combined with Reduced Software DAL That Yield Safer, More Capable Systems With More Affordable Lifecycle Costs

• Challenges: Resistance to Lower DAL in Critical System from Civil Authorities – Perception of “Lowering” Safety

Challenge Points• Poor Implementation or Improper Assumptions Can Lead to Unsafe

Automation – i.e. Assuming Particular Pilot Reaction or Skill Level

• Robust/Resilient Architectures are Required for Safe Increase in Automation Towards Autonomy and AI/ML Implementation

• Barriers Exist For Certification of Complex Architectures – Need to Improve Processes – Every Line of Code Increases Cost

• Need Design Best Practices for Automated Systems – RTA, Partitioning, Self-Checking Algorithms – Fail Functional Design

• Balance Development Assurance with Smart Architecture – Consider “Entire Safety Equation” - Focus on Incremental Improvements

• Like Humans, Do We Need 100% Certainty of “Why” a Behavior Happens, or Can We Simply Measure/Bound Expected Behavior to Safe Limits?

Planning – Trajectory Plan – Weather, Traffic, Passenger Load, Revenue

Decision Making/Response –Mission/Trajectory Management

Perception/Hazard Detection –Trajectory De confliction, Redirection

Control – Flight Path Trajectory

AccurateModels

UnknownModels

AI/ML & Automation Risk Construct

Tactical -ShorterTimeline

Strategic - LongerTimeline

ControlFunctions

PerceptionFunctions

Given:• Perception Functions Are More Difficult to

Model, Less Bound by Known Model/Physics• Control Functions Well Understood,

Governed by Flight Dynamics and Physics of Aircraft/Trajectory Models

• Tactical Decisions Have Shorter Timeline, Greater Urgency

• Strategic Decisions Have Longer Timeline, Less Urgency

Our Shared Challenge:• Where Should We Focus Automation &

AI/ML in the Near Term, Mid Term, Long Term? Safety Critical or Route Efficiency Improvement Functions?

• What Risks/Challenges Do We Face for Moving Automation & AI/ML from Tactical to Strategic Decision Making, or From Control Functions to Perception Functions?

High Criticality

Low Criticality

Aircraft/Agent Level Functions

Enterprise Level Functions

Higher Cert Rigor

Lower Cert Rigor

Most Practical Early Use of AI/ML?

Risk Based & Model Based Integration

• Introduce Automation In Low-Risk Use Cases First, Where Appropriate

• Collect Data & Use Data to Develop/Validate Models

• Analyze Models for Higher Risk Safety Cases to Evolve

• “Build a Little, Test a Little” - Iterative Loop

• Models for Physical Problems Easier to Develop/Mature Than Models for Decision Making and Perception Functions

• Move Technical Maturity Forward for Specific Functions – Combine Functions to Reach Specific Operational Goals for Autonomy From ASTM AC 377 TR

Logical/Purposeful Automation Convergence • Start With Aircraft Centric and Ops Centric Efforts

• Evolution vs. Revolution• Methodical Introduction - Proven Technology Readiness by Test (National

Campaign, UTM, etc.)

• Identify Certification Challenges/Strategies

• Operational Integration & Human/Machine Teaming

• Converge Towards Center With Aircraft and Infrastructure Approach

Converge From Low Risk to High Risk (Safety Risk, Financial, Time, Certification, Infrastructure, etc.)

Aircraft

Functions

NAS

Functions

Converge Towards “Full Autonomy”

Functional Classification: Aviate, Navigate, Communicate, Locate, Separate, Allocate

• The Process By Which A Product is Deemed Safe for Repeatable, Reliable, Civil Use

What Is Safety Assurance?

Ops/PilotErrors

• Not Just Development Assurance

• System safety, airspace, ops, maintenance, & pilot performance all feed into operational safety

• Cannot try to fix ops safety target with increasing 10E-X for system failures

• Must Account For Other Factors

SystemFailures

Weather.Terrain, Other Factors

Airspace/PopulationDensity

Combined Safety

Mitigations

Maint. Errors

Not the

Same!

Operational Safety Target

Design Safety Target

Technology Hype – Lessons Learned

• History Reflects Typical Hype Cycle• Technology Triggers of Electric

Propulsion and Automation• Reaching a Peak of Expectations –

Large Investment• The Level of Rigor and Diligence

Dictates How Deep The “Trough of Disillusionment” Will Be

• Accidents Early Will Deepen the Trough and Delay the Rise to a Plateau of Successful/Safe Ops

• VLJ Lessons Are Applicable –Autonomy Represents a Challenge to eVTOL AAM Departing from the VLJ Path

Image: Wikipedia Hype Cycle

Advisory, Assistive, and Responsible Functions – When Will AI/ML Be Ready?

• Starting with Advisory Functions or Assistive Functions Allows for Human Intervention for Safety Critical Conditions

• Responsible Functions Require “Civil Trustworthiness” to be Proven

• How Do We Build in the “Aptitude” and “Resilient Proficiency” into AI/ML and Automation System?

• Aptitude Measures Ability of the System to Perform its Function

• Resilient Proficiency is the Ability of the System to Perform Repeatedly in Foreseeable Conditions

• Civil Trustworthiness is the Ability to Show the System is Proficient to a Level the Public Trusts

Presented to: UAS Integration Research Roundtable

Presented by: Sabrina Saunders-Hodge, UAS

Research Division Director

AUS-300

Date: February 21, 2019

AdvisorySupplemental, or Emergency Only

Pilot is Primary For Safety

Assistive TechnologySystem Provides Certified/Accurate Data

Shared Responsibility -Pilot is Still Primary For Safety

Mature TechnologySystem Has Proven Reliable Use in Service - Provides Certified, Accurate Data

Pilot is Still Primary For Safety

Technology For Operational CreditSystem Has Proven Reliable Use in Service - Provides Certified/Accurate Data – Regulations Changed Based on Proven Maturity to Allow Credit for Use (i.e. part 91)

Pilot is Backup to Proven Technology

“Responsible” TechnologySystem Has Proven Reliable Use in Service - Provides Certified/Accurate Data Without Reversion to Pilot Backup for Sufficient Number of Hours -Regulations Changed to Recognize Technology as Primary for Intended Use (i.e. 91.3 Pilot in Command)Technology is Primary

For Safety

Lower Approach Minimums

RegulatoryChange

TechMaturation

Technology Not

Approved for

Autoland

NASA Paper: Visual advantage of enhanced flight vision system

during NextGen flight test evaluation

Automation Example – Technology Maturation

24

Live Content Slide

When playing as a slideshow, this slide will display live content

Poll: Follow Up: In term of our "Barriers to Scalability/Autonomy" question, what area wasn't listed? Define area in one or two words.

Upcoming Aircraft WG Meetings

Typically, the Aircraft Working Group holds their meetings on the last Thursday of every month from 3:00PM - 4:30PM ET (12:00PM - 1:30PM PT).

• June 24 2021: Topic: eSTOL

• July 29, 2021: Topic: Pilot’s Perspective

– POCs: • Carl Russell: [email protected]

• Wes Ryan: [email protected]

• Anna Cavolowsky: [email protected]

25