Page 1
ESA UNCLASSIFIED - For Official Use
Study Results on Employing Virtual and Augmented Reality for Spacecraft Operations and Astronaut Training at ESOC and EAC
Ruediger Gad (Terma GmbH), Holger Graf (Fraunhofer IGD),Mehran Sarkarati (ESA ESOC), Chris Scott (ESA EAC)
Results of Studies for ESA EAC & ESOCPerformed by Terma GmbH & Fraunhofer IGD
Page 2
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 2
Outline
• Practical
• High-precision Model-based AR Tracking
• Multi-modal AR Application for Rover Operation
• VR for Astronaut Training
• Integration of AR/VR with ESA Software
• Conceptual
• Agile-like Workflow/Development Processes
• Use Case Identification and Details Development
• Use Case Ideas and Taxonomies
Page 3
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 3
AR Tracking• “HoloLens Tracking”
• Track “Room”
• Detect “Surfaces”
• Model-based Tracking• Track Objects
• 3D-Model of Object
• “Find Edges”
• Initialisation
• Tracking
• VisionLib(Fraunhofer IGD/Visometry GmbH)
Page 4
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 4
AR Tracking• “HoloLens Tracking”
• Track “Room”
• Detect “Surfaces”
• Model-based Tracking• Track Objects
• 3D-Model of Object
• “Find Edges”
• Initialisation
• Tracking
• VisionLib(Fraunhofer IGD/Visometry GmbH)
Page 5
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 5
Interactive Training/Manual PoC – Tracking Precision
Page 6
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 6
Interactive Training/Manual PoC – Visualisation
Page 7
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 7
Interactive Training/Manual PoC – Visualisation
Page 8
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 8
Interactive Training/Manual PoC – Visualisation
Page 9
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 9
• EAC• Astronaut & Rover on
Celestial Body Surface
• Track Real Rover
• ESOC• No Real Rover at AR User
• Virtual Rover Model& HoloLens Tracking
• Scalable View:Detail & Overview Display
Multi-modal AR Application for Rover Operation
Celestial Body SurfaceSpacecraft Orbiting
Celestial Body
Astronaut
RoverBroker(Space)
MRS(Space)AR
SupportEquipment
HoloLens
TM
TC TM
TC
TM/TC
Page 10
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 10
Multi-modal AR Application for Rover Operation
Celestial Body Surface
Spacecraft Orbiting Celestial Body Ground Control
Operator
RoverBroker(Space)
MRS(Space)
AR Sup. Equip.
HoloLens
TM
TC
TM
TC TM/TC
• EAC• Astronaut & Rover on
Celestial Body Surface
• Track Real Rover
• ESOC• No Real Rover at AR User
• Virtual Rover Model& HoloLens Tracking
• Scalable View:Detail & Overview Display
Page 11
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 11
Rover Operation Proof-of-Concept
Page 12
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 12
Rover Operation Proof-of-Concept
Page 13
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 13
VR for Astronaut Training• Virtual Lunar Base, Fire Emergency Training
• Users/Roles
• Astronaut
• Operator
• Trainer
Page 14
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 14
VR for Astronaut Training• Virtual Lunar Base, Fire Emergency Training
• Users/Roles
• AstronautTrainee, in VR
• Operator
• Trainer
Page 15
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 15
VR for Astronaut Training• Virtual Lunar Base, Fire Emergency Training
• Users/Roles
• Astronaut
• OperatorSupport Astronautvia Mission ControlSystem
• Trainer
Page 16
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 16
VR for Astronaut Training• Virtual Lunar Base, Fire Emergency Training
• Users/Roles
• Astronaut
• Operator
• TrainerPrepare & Supervise Training
Page 17
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 17
VR for Astronaut Training Models• Behavioural
• MATLAB (Development)
• SIMSAT (Run-time)
• 3D
• Unity(Develop. & Run-time)
• Elements
• Racks• Buttons, Status Display, …
• Fire Extinguishers• …
• Model IntegrationVR <-> Behavioural
Page 18
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 18
Integration of VR with ESA Software
Preparation Run-time
Key
VR Run-time(MQTT)
Simulation Run-time(CORBA)
MCSRun-time
(CCSDS-MO)
VR Hardware
Integration Middleware
VR Preparation
Simulation Preparation
VR Application
BehaviouralSimulation
I/OInfo.
Preparation Artefact Deployment Runtime Interaction
I/OInfo.
Page 19
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 19
Integration of AR with ESA SoftwareA
R Su
ppor
t Eq
uipm
ent
Hol
oLe
ns
Spac
e-cr
aft
Gro
und
Cont
rol
AR Application
Message-oriented Middleware
MOE Adapter
MQTT (JSON UTF-8)
OpenWire (JSON UTF-8)
Broker MRS
Rover(Real/Simulated)AR User
TM TC
Rov.TC
Rov. TM
TM/TC
TM/TC
Visualisation (out)Interaction (in)
MCS
ARTM/TC
HoloLens
Page 20
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 20
Integration of AR/VR with ESA Software
• Virtual Reality (VR) Study
Page 21
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 21
Integration of AR/VR with ESA Software
Page 22
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 22
Integration of AR/VR with ESA Software
Page 23
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 23
Integration of AR/VR with ESA Software
Page 24
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 24
Integration of AR/VR with ESA Software
Page 25
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 25
Integration of AR/VR with ESA Software
Page 26
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 26
Integration of AR/VR with ESA Software
Page 27
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 27
Integration of AR/VR with ESA Software
Page 28
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 28
Integration of AR/VR with ESA Software
Page 29
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 29
Agile-like Workflow/Development Processes
• Concept Definition (Use Cases / Applications)
• Technologies & Integration Assessment
• Practical Assessment
• -> Proof of Concept (PoC)
TechnologyAssessment
Concept Definition
WorkshopPoC
Development & Assessment
Phase 1 Phase 2
Page 30
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 30
Agile-like Workflow/Development Processes• Concept Definition (Use Cases / Applications)
• “Open”: Broader Scope than the PoC
• Technologies & Integration
• VR/AR & ESA Systems
• Practical Assessment -> Proof of Concept (PoC)
• Selected Use Case(s)
TechnologyAssessment
Concept Definition
WorkshopPoC
Development & Assessment
Phase 1 Phase 2
Page 31
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 31
Use Case Identification and Details Development
• Interviews/Mini-workshops
• Two Parts• Ideation
• High-level Ideas for Application Scenarios
• Derivation/Definition• More Details about (Selected) High-level Application Scenarios
• “Open”• Broad Scope
• Not Limited by PoC Considerations
• A Basis for Future Activities
Page 32
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 32
Ideation “Raw” Results Examples
Page 33
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 33
Application Scenario “Taxonomy”
• Application Domain
• Mission Control
• Robotic Operation
• Virtual Lunar Base (EVA)
• Goal of the Scenario
• Training
• Planning
• Operations
• …
Page 34
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 34
High-level Scenarios
• Training
• Interactive Training/Manuals
• Virtual (AR) Representationof Training Instructions/Manual
• Interactively show procedures/steps.
• Overlay instructionswith real world.
Page 35
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 35
High-level Scenarios (Continued)
• Planning
• Rover Operations Planning
• AR Scene withRover & Environment
• Scientists Mark
• Operators Plan
• Managers Verify
• Multi-user?
Page 36
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 36
High-level Scenarios (Continued)
• Operations
• Spacecraft Operations
• Analyse Situation
• Plan Next Steps
• Multi-user?
Page 37
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 37
Conclusion & Outlook• VR
• Immersive Training (“Costly” Scenarios)
• Planning (e.g., Mars/Moon Rover Operations)
• AR
• Interactive Training/Manual (Near Future?)
• Cooperative Scenarios with Multiple Users
• User Experience of Devices
• VR more Mature
• AR Headsets still Somewhat Uncomfortable
• Challenge: Bridge the Gap between Users and Technology.Users Utility Technology
• Further activities are beneficial.
Page 38
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 38
End
Thank you very much for your attention.
Questions?
Ruediger Gad for the VR & AR Study Teams
[email protected]