DR 3.4: Shared Situation Awareness that Supports ResilienceThe work in Year 3 was planned to address Milestone MS3.4: “A platform for sharing human-robot SA and adaptive human-robot

EUFP7TRADR(ICT-60963)

DR 3.4: Shared Situation Awareness that Supports Resilience Norbert Pahlke1, Nanja Smets2, Elie Saad3, Sohini Sengupta4, Ivana Kruijff-Korbayova4 and TRADR consortium 1 Institut fur Feuerwehr und Rettungstechnologie FDDo 2 TNO, Soesterberg, The Netherlands 3 Technology, the Netherlands 4 DFKI, Saarbruecken Project, project Id: EU FP7 TRADR / ICT-60963

Project start date: Nov 2013 (50 months)

Due date of deliverable: December 2016

Actual submission date: March 2018

Lead partner: FDDO

Revision: FINAL

Dissemination level: PU

This deliverable describes the results in the TDS research and development in Year 4 of the TRADR project. The objective is a scenario response system for robot operations and information/team management in parallel.


Table of Contents ExecutiveSummary...............................................................................................3

1. Tasks,objectives,results...............................................................................5

1.1. Plannedwork..............................................................................................5

1.2. Addressingreviewers'comments...............................................................5

1.3. Actualworkperformed...............................................................................5

1.3.1. TDSinterfacedesignandimplementation...............................................7

1.3.2. TaskManagementInterface....................................................................9

1.3.3. InterfaceforMonitoringWorkloadofMissionActors..........................12

1.3.4. MultimodalInteractionwiththeTDSUsingSpeech..............................13

2. Annexes.......................................................................................................14

2.1.1. R.deKok,B.vanWoenselandN.J.J.M.Smets(2018),“RequirementsandimplementationTRADRTDS".......................................................................14

2.1.2. N.Smets,M.NeerincxandC.Jonker.“AnOntologyofTeamSituationAwarenessforSupportandEvaluation".............................................................14

2.1.3. S.SenguptaandI.Kruijff-Korbayova.“ProcessingMultimodalDialogueforDisasterResponseSupport"..........................................................................15


Executive Summary

Workingtogetherwithrobotsinaresponseteamforcatastrophesrequiresspecialequipment.Theoperationofrobotsshouldbeassimpleaspossible.Thehigherthelevelofautonomythehigherthelevelofusefulness.Insteadofsteeringtherobotstheanalysisofgathered informationcanbe focused.Achieving this objective theWP 3 aims to develop a central system unit.

Role of Shared Situation Awareness in TRADR TheWP3combinesand integrates informationandfunctionality throughacentralmanagementsystem,theTDS.This is thenewplatformfortheendusersfortheiroperationmanagement.Itisanewsituation.Robots,UGVsandUAVs,participateonrescuescenariosasteammembersandsupportthehumansindoingthenecessaryjobs.TheysupportbycontributingtoSA:takingvideosforlivestreamsandphotosforanoverviewandfor2D/3Dmaps,collectingdataforanestimationofnon-visible dangers like explosive atmospheres, radioactivity and more,dependingon the available sensors.But in the samewayas they scan thesurrounding for the humans they do it also for themselves especially byrecording LIDAR information. Based on that they create their own 3Dvirtualworldandassessthemselves inrelationtothat.UGVsarenowableforexploringtheenvironmentautomaticallystepbystepanddependingonthestrategymostefficient.Alternativelyandwiththe3Dmapsaspersistentbackgroundinformationtheycanbeusedforpatrollingtasksaloneorasaswarmwith individual focus inexecution.UAVsarestill steeredmanually,becauseofthemissedtechnologyforanautomaticcollisionavoidanceandduetotheregulationsbylaw.ButevenwithoutthattheUAVdeliversveryfastanoverviewanddetails fromanoptimalposition. Itsupportsalso thevictimsearchaswellassmokeandfiredetection.

ThisroughlistofrobotfunctionsgivesanimpressionontherequirementsfortheTDS.Ontheonehanditmustbepossibletomanagetherobots,onthe other it is important to manage the gathered information and todistribute them to the persons,whowill need them.Over all theTDS is amanagementtoolforallkindofoperationsfromenduserperspective.

Persistence Persistence in WP3 is addressed by enabling end-users to interact with (relevant) information that is persistent within the TRADR system. In effect the TDS


serves as a looking glass and filter into the TRADR databases, as to support persistent SA and allow tactical decisions to be made effectively. Thus, the TDS provides access into information retained over the course of multiple sorties.

Contribution to the TRADR scenarios and prototypes Changing the test strategy to a realistic use, the TDS could demonstrate its use for end users.


1. Tasks,objectives,results

1.1. PlannedworkThe work in Year 3 was planned to address Milestone MS3.4: “A platform for sharing human-robot SA and adaptive human-robot communication”.

1.2. Addressingreviewers'commentsReviewers’recommendation:Acontingencyplanisneededintheshorttermtorecoverfromthedelayofthework.

The consortium implemented the plan shared with and endorsed by the reviewers after the Y3 review meeting. TNO has taken over the responsibility for the TDS development, TNO also engaged a suitable interface designer. The corresponding resources have been shifted from FDDo to TNO.

1.3. Actualworkperformed

NowattheendoftheprojectitwassourendeavourtopresentaTDSwhichfits with the requirements and expectations of the end users. For thispurpose two interview sessions were held parallel to the integration offunctionalityandindependentlyoftheTJEx/TEvalsothatanoptimizedTDSversioncouldbeused for theofficial tests.Agroupof sixprofessional firefighters evaluated during simulated scenarios the TDS regardingfunctionality, layout and quality of visualization for the informationmanagement. It turned out that the layout could be improved concerningclarity.Toomanyfunctionsintheformofbuttonsoverloadedthescreeninthe first impression. Themap, the central situation report, was too smallwhich reduced the situation awareness of the users. Together with aprofessionaldesignerforuserinterfacesthelayoutwasimprovedonamorestructuredbasis.Attheendthemapcouldbepresentedlarger.ThisisnottobeunderestimatedwithregardtothescreensizeoflaptopsortabletPCsthetypicaltoolforoperationsoutsideinthefield.Otherimprovementsledtoaclearoverviewwithonlythereallyimportantfunctionsandinformationatfirst sight, like the alerts (notifications), icons (placemarks) for markingoperation relevant places in the map, layer list and so on. After theimprovement the layout was accepted by the subjects. Furtherimprovements concern the functionality. The handling of the TDS is now


simplified. It is closer to the user’s expectations and experience in othersoftwareproducts.Forexampletheplacingoficonscanbedonebyamouseklick everywhere on themap. Themenu structure is concentrated on theright sideof thewindow. It is thestandardconfigurationof theTDS.Withintroductionofthemodulardesigneachusercanoptimizethelayouttohisneeds. Thatmeans he can arrange the differentmenus like flying palettesandplacethemfreeasheneedstokeeptheoverviewandSA.

The new software developer integrated the bridge to connect the speechrecognitiontotheTDSpushingforwardthemultimodal information input.Theuserscannowsetplacemarksbyspokencommands.

An example for the improvement of the quality of visualization is therenumberingof thealerts (notifications).Thealertsobtainanagentbasedlogicalnumberinchronologicalorderinsteadofcrypticnames.Thatoffersarealsimplificationintheinformationhandlingandcommunication.

Coming back to the challenge that the teamwork between humans androbots is a new situation. So far the TJEx and TEval were tests withpredefinedtasksfocusingthedifferentfunctionalitiesoftherobotsandthesystem in general thus theTDS.Thisprocedure ensured that all functionswere evaluated without exceptions and that the CIP (ContinuousImprovement Process) was applied comprehensively. At the end of theprojectitwasnecessarytochangetheevaluationstrategy.Thekeyinterestwas now the application of the system in realistic exercises without anyguidelines. Could the systemmeet the requirements and support the endusers? To clarify this question a containerwas converted to an operationcontrolcenter.Equippedwithcontrolunitsforrobotsandateamleaderthemanagementoftheoperationwasorganizedandexecutedfromhere.Outofline-of-sighttheoperationoftherobotswasinterestingbecauseofavideodelayproblematicinthepastandconcerningtheresilienceofthesystemingeneral.Itcouldbedemonstrateddespiteofanexistingsmalldelaythatthemanual operation of the robots was fast and possible. Driving breakscouldn’t be observed like in the past. In sum the operations could bemanagedsatisfactorily.

The taskmanagement ontology has been added, and this resulted in twodifferentGUIs for theTDS. The taskmanager and task editor.To allocatenewtasksoreditexistingones,theteamleaderusesataskeditor.AsecondGUIprovides the team leaderwith a taskmanager interface to enable the


team leader to track andmonitor the progress of assigned tasks.Missionactorscantracktheprogressoftheirtasksinthemaindisplaysystemwhichshows the task POI, description and status. The latter property iscontinuously updated by actors and agents throughout the executionprocess.

The actual work performed in WP3 Year 4 included the following:

• TDS interface design and communication (Section 1.3.1)

• Interface for task allocation and management during TRADR missions (Section 1.3.2)

• Interface for monitoring the workload of mission actors (Section 1.3.3)

• Multimodal interaction with the TDS using speech (Section 1.3.4)

Moredetailsareprovidedinthenextsections.

1.3.1. TDSinterfacedesignandimplementationFirstaquickscanoftheuserinterfacewasperformedbyanuserinteractiondesigner, by using the TDS as it was during summer 2017, see Figure1.Subsequently the user interaction designer gathered information aboutwhattheenduserswantanddefineduserstories.Thesetwoprovidedinputfor requirements and an interactiondesign.Next to the changesbasedoninput from the interaction designer, input from end-users and partnerswere also implemented. Implementation also encompassed changes toensurestability.Foramoredetailedoverviewsee(AnnexOverview2.1).


Figure1:TDSSummer2017.

Figure2:TDSafterredesignandimplementation.


1.3.2. TaskManagementInterfaceTouseanddeploythetaskmanagementontologydiscussedinSection1.2.4of document DR 5.5: Deliverable D5.4, two different GUIs have beendesigned.TheseGUIsintegrateandprovidesupportfortaskmanagementinthe searchand rescue tacticaldisplay system (TDS).The latter isused fortrackingthedisasterareaandhasbeendevelopedtoassistUSARteamsinthe TRADR project. It contains a map of the disaster site showing thelocation of actors and the detected POIs (victims, fires, chemical objects,etc.).Italsoprovidessupportforassessingthedisastersitesituationandforgatheringrelevantinformationaboutit.

Figure3:TaskEditorforcreatingandeditingmissiontasks.

To allocate new tasks or edit existing ones, the team leader uses a taskeditor, as shown in Figure3. In thisGUI, the team leaderdefines the taskpropertiesincluding(1)atasktype(inspectiontask,explorationtask,etc.);(2)adescriptioncontainingspecificdetailsorguidelinesfortheoperators;(3)aPOIwhichdefinesthetask'sobjective;and(4)anactorfromthelistofavailableactorssuggestedby thesystem.Otheroptional fieldscanalsobeset, such as (1) a priority; (2) a status (pending, in progress, completed,etc.);(3)alistofrequiredcapabilities;(4)arequiredbatterylevel;and(5)arequiredworkload.


Figure4:TaskManagerfortrackingandmonitoringtasks.

AsecondGUIprovidestheteamleaderwithataskmanagerinterfaceFigure4whichhasbeendesignedtoenabletheteamleadertotrackandmonitorthe progress of assigned tasks. For each task, the GUI displays its type,description,assignedactor,priorityandstatus, toprovide the team leaderwith an overview of the execution progress. Mission actors can track theprogressoftheirtasksinthemaindisplaysystemwhichshowsthetaskPOI,description and status. The latter property is continuously updated byactorsandagentsthroughouttheexecutionprocess.

For every new mission, the main ontology is initialized and loaded in acentral repository (weuse Stardog triple stores http://www.stardog.com}which provide support for querying, inferencing and manipulating theknowledgebasestoredintherepositorybasedonthesemanticsdefinedbyourontology).Toensurethisrepositorymaintainsanup-to-datestateofamission,wedevelopedanduse semanticmodelers to continuouslyupdatethe database with new knowledge acquired during a mission. Thesemodelersmaprawsensordata(e.g.,pointcloud,GPScoordinates,etc.)ontoontologicalconcepts(POIs,locations,etc.)andstoreitintherepository.


Themappeddataisthenusedtodisplayandupdatemeaningfulinformationformonitoring theprogressof amissiononTDS. It is alsoused to reasonabout therepresentedworldandgeneratenotificationsrelatedto the taskbeing executed. The aim is tomanipulate the gathered knowledge for (1)improvingsharedsituationawareness;and(2)assistingtheteamleaderinits job of assigning tasks by providing automated support through taskproposals. For example, when a new POI is detected, the system willpropose a new task so that the team leader allocates it to one of theavailableactors.

Figure5:Activitydiagramforassigningandexecutingmissiontasks.

Throughoutamission, the teamleaderwillcontinuouslyaddnewtasksorupdate existing ones (description, priority, etc.). The activity diagram inFigure5 shows theworkflow forassigningandexecutinga task.First, theteamleaderassignsatasktoanactor.Then,theactorcanacceptitandstartthe execution or can abort it when facing technical issues (e.g., roboterrors). When the task is executed, the actor sets its status to awaitingacknowledgmentusingthetaskmanager.Iftheresultisaccepted,theteam


leadersetsthetaskstatustocompleted.Otherwise,itwillbereassignedorcanceled.

1.3.3. InterfaceforMonitoringWorkloadofMissionActors

AworkloadGUIhasbeendevelopedtomonitortheactorsworkloadduringaTRADRmission.ThisGUI is integrated intoTDS(Figure2) toalloweachactor manually enter their workload. As for the standalone application,illustratedinFigure6,itismeanttobeusedbyanexternalobserverwhoisresponsible for tracking the workload of the team throughout a TRADRmission. The team leader's workload is used as a trigger for workingagreements inordertoregulatethefrequencyofsendingnotificationsandalerts during amission. For example,when the team leader'sworkload isset to 5 (which means high), the system TDS will only display urgentnotificationsandalerts (i.e., if thePOIpriority ishigh).Whereaswhen theworkloadislow,theteamleaderwillbenotifiedmorefrequently,basedonhowtheysettheworkingagreements.


Figure6:WorkloadGUIformonitoringactorsworkloadusingascalefrom1(low)to5(high).

1.3.4. MultimodalInteractionwiththeTDSUsingSpeechWecontinuedthework fromYear3.Using theexperience fromTJex2017andadditionalinputfromtheendusers,wehaveanalyzedtherequirementsonandusabilityofincludingspeech-baseddialogasanadditionalmodalityforinteractionwiththeTDS.Weextendedtheimplementationoftheframe-baseddialoguemanagerandtheNuance.Mixmodelsforspeechprocessing.Speech-baseddialoguehasbeenintegratedwiththeTDS,thusresultinginamultimodal system. Focus was on the operations related to victims andpoints of interest: adding new ones and displaying the existing ones. Themultimodal TDS system was employed during the TRADR EvaluationExercise2017.ThedetailsarepresentedintheAnnex2.1.3.


2. Annexes2.1.1. R.deKok,B.vanWoenselandN.J.J.M.Smets(2018),

“RequirementsandimplementationTRADRTDS"Bibliography: R. de Kok, B. van Woensel and N.J.J.M. Smets (2018),“RequirementsandimplementationTRADRTDS".Technicalreport

AbstractThis document lists the user interface requirements for the TRADR TDS(TacticalDisplaySystem)andtheactualimplementation.Firstaquickscanof the user interface was performed by an user interaction designer, byusing the TDS as it was during summer 2017. Subsequently the userinteractiondesigner gathered information aboutwhat the enduserswantanddefineduserstories.Thesetwoprovidedinputforrequirementsandaninteractiondesign.Thesewereprovidedasinputfortheimplementation.

RelationtoWPThisworkcontributestoT3.4SharingSAforhuman-robotteaming. Since the new design and implementation of the TDS supportssharedSAbetterforthewholehuman-robotteam.

AvailabilityRestricted.

2.1.2. N.Smets,M.NeerincxandC.Jonker.“AnOntologyofTeamSituationAwarenessforSupportandEvaluation"

Bibliography:N. Smets,M.Neerincx andC. Jonker. “AnOntologyofTeamSituationAwarenessforSupportandEvaluation"(2018)TechnicalReport.

AbstractWe formulate an ontologywith the concepts of SA, its effects and how tomeasure SA and use this as support before, during (real-time) andafterwards for evaluationpurposes.Wewill illustratehow this isdonebydescribing the process for small case studies in the USAR domain forhuman-robot teams. Such an ontology makes it possible to choose thecorrect level of SA in thedesign and evaluationprocess for human robot-teams.Theontologywillbeverified for internalconsistencyandvalidatedbyafocusgroup.


Relation to WP This work directly contributes to T3.4 Sharing SA forhuman-robotteaming.

AvailabilityRestricted.

2.1.3. S.SenguptaandI.Kruijff-Korbayova.“ProcessingMultimodalDialogueforDisasterResponseSupport"

Bibliography:S.SenguptaandI.Kruijff-Korbayova.“ProcessingMultimodalDialogueforDisasterResponseSupport"(2018)TechnicalReport.

AbstractInthispaperwepresentanapproachforprocessingmultimodaldialogueinthecontextofhuman-robotteamingfordisasterresponse.Wehaveanalyzedtherequirementsonandusabilityofamultimodaldialogsysteminthecontextwhereahumanteamworksusesrobotstoassistsituationassessmentafterlargeincidents,suchasanindustrialaccidentoranearthquake.Weimplementedaframe-baseddialoguemanager.ForspeechprocessingwehaveusedtheNuance.Mixcloudservices.ThesystemwasemployedinafieldtrialwithDutchfirefightersintheTRADREvaluationExercise2017.

Relation to WP This work directly contributes to T3.4 Sharing SA forhuman-robotteaming.

DR 3.4: Shared Situation Awareness that Supports ResilienceThe work in Year 3 was planned to address Milestone MS3.4: “A platform for sharing human-robot SA and adaptive human-robot

Documents