This project has received funding from the European Union’s Horizon 2020 research and innovation program under grant agreement No 700475 beAWARE Enhancing decision support and management services in extreme weather climate events 700475 D6.2 Data-Source Integration Framework Dissemination level: Public Contractual date of delivery: Month 15, 31 March 2018 Actual date of delivery: Month 15, 31 March 2018 Workpackage: WP6: Public Safety Answering Point Task: T6.2 Data Source Integration Framework Type: Report Approval Status: Final Version: V1.0 Number of pages: 49 Filename: D6.2_beAWARE_Data-Source Integration Framework_2018- 03-31_v1.0.docx Abstract This deliverable reflects the work performed in task T6.2 dedicated to ensure robust and reliable integration of data sources. In particular, the T6.2 objectives are to develop a suitable support framework to create an integration layer that can be deployed on top of varying physical infrastructures to expedite the integration of existing and new data sources. By providing this abstraction from the underlying data sources, it will provide a framework to intelligently manage the data and control flows between the beAWARE platform and the physical environment. To support the integration of existing sensors, camera, video and other structured data, a set of reusable data source connectors will be developed. This deliverable will describe the Data Source Integration Framework layer. Ref. Ares(2018)1756440 - 31/03/2018
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
AbstractThis deliverable reflects thework performed in task T6.2 dedicated to ensure robust andreliable integration of data sources. In particular, the T6.2 objectives are to develop asuitable support framework to createan integration layer that canbedeployedon topofvaryingphysicalinfrastructurestoexpeditetheintegrationofexistingandnewdatasources.Byprovidingthisabstractionfromtheunderlyingdatasources, itwillprovideaframeworktointelligentlymanagethedataandcontrolflowsbetweenthebeAWAREplatformandthephysical environment. To support the integration of existing sensors, camera, video andother structured data, a set of reusable data source connectors will be developed. ThisdeliverablewilldescribetheDataSourceIntegrationFrameworklayer.
beAWARE is a highly-interconnected systemof systems and interdependentmodules thatfeed each other with information regarding evolving and ongoing extreme climateemergencies.Thisincludesacontrolcentersolution,amobileapplication,variousmediaandsocialmediaanalysisservices,sensoranalysisservice,weatherandclimateforecastservices,crisis classification and reasoning services,multilanguageanalysis and generation services,andsupportinginfrastructure.
beAWARE integrates data from multiple sources, including field reports, sensormeasurements,weather data,media, analysis results, and socialmedia posts. In addition,thesystemhastogenerateinformationandinsightaswellasactionsandinstructionswithinthe system and for its users and stakeholders. For this reason, a suitable data sourceintegration framework has been defined to govern the data and information exchangearchitectureandrealization.
Thepurposeof thedata-source integration framework is therefore to assure and supportinteroperability among various stakeholders and operational actors – authority officials,emergencymanagementofficers,analysts, first responders,andcitizens.This isdonebyarobust,dynamic,andflexibleinfrastructurethatallowsinterconnectivityamongthevarioussubsystems,modules, and services. On top of this infrastructure, the system enables andimplements multiple end-to-end interactions, leveraging data exchange, storage, access,acquisition,andvisualizationmechanisms.
This document is themain reference for thedata exchange architectureof thebeAWAREplatformasimplementedtowardbeAWARE'sfirstprototype,dueJune2018.Theframeworkhasbeendefinedtobesufficientlyrobusttoaccommodatepossibleupdatesandextensionsofdataexchangeandcollaboration,astheprojectprogressestowardsthefinalversion,dueDecember2019.
Thisdocumentprovides:a)apublicoverviewoftheoperationaluserrequirementsentailingimpact related todata anddata-sourcemanagement and integration;b) apublic detaileddescriptionofthecommondataintegrationinfrastructure,includingmessaging,dataaccessandquerying,andfilestorageservices;andc)aconfidential,detailedspecificationofeachmodule’sconnectivityrequirements,inputs,outputs,anddataaccessmechanisms.
The operational section is public because it relies on publicly-available operational userrequirements, which constitute common knowledge of public interest. The operationalrequirements have been analyzed for impact on data-source and data user integration,
D6.2–V1.0
Page6
interaction, and interoperability. Themainmodules and entitieswere laid out to supporttheserequirements.Thisprocessanditsoutcomesareofuseasageneralreferenceforthecommunity.
Thesectiondiscussingthegenericinfrastructuresupportingdataexchangeandintegration,includingmessaging,mediastorage,databases,etc.,isalsopublicsinceitreliesonpublicly-available open-source products, and constitutes a benchmark that we advocate to theemergencymanagementtechnologycommunity.
In the applicative layer, the beAWARE modules' utilization of the generic infrastructureaccountsforthevariousconstraints,challenges,technicalrequirements,anddatacontent.This information remains confidential, in order to prevent direct exposure of technicaldetails, data structures, and access control mechanisms, and in order to protect thebeAWAREpartners'intellectualproperty.
This deliverable is the product of a year-long effort to define, organize, manage, design,implement,anddemonstratethedata-sourceintegrationframework,aspartoftaskT6.2inWP6, under the responsibility of MSIL. This effort was heavily supported by IBM in thegenericinfrastructureaspects,andinvolvedalltheothertechnicalpartners–AAWA,CERTH,FMI, IOSB, and UPF. Each technical partner helped derive and define its technicalconnectivityandfunctionaldata-exchangerequirements.Theconsortiumasawholeworkedtogethertoensureend-to-endflowofinformationthroughaseriesofteleconferences,face-to-facemeetings, partner-to-partner discussions and end-to-end integration sessions. Theoperational partners – HRT, PLV, and FBBR – contributed to this effort by reviewing thesystem requirements and overall architecture throughout the project, and ensuring theircompliance with the operational requirements. In addition, PLV and HRT provided usefulcommentsandreviewsonthisdeliverablepriortoitssubmission.
D6.2–V1.0
Page7
AbbreviationsandAcronyms
alt altitude API ApplicationProgrammingInterface ASR AutomaticSpeechRecognition CAP CommonAlertProtocol CDR CentralDataRepository CRCL CrisisClassification CSV Comma-SeparatedValues DB Database GIS GeographicInformationSystem GPS GlobalPositioningSatellite HTM(L) HypertextMarkupLanguage HTTP HypertextTransferProtocol ID identifier IoT InternetofThings IP InternetProtocol JSON JavaScriptObjectNotation KBS Knowledge-BaseServices lat latitude long longitude MRG MultilanguageReportGeneration MSB MessageBus MTA MultilanguageTextAnalysis ODI OpenDataInterface PCI PeripheralComponentInterconnect PSAP PublicSafetyAnsweringPoint Pub/Sub Publish—Subscribe REST Representationalstatetransfer SDK SoftwareDevelopmentKit SMA SocialMediaAnalytics SoS SystemofSystems SQL StructuredQueryLanguage TCP TransmissionControlProtocol TR TechnicalRequirement UDP UserDatagramProtocol UI UserInterface UR UserRequirement URI UniformResourceIdentifier URL UniformResourceLocator USB UniversalSerialBus WFS WeatherForecastServices WP WorkPackage XML ExtensibleMarkupLanguage XSD XMLSchemaDefinition
This document constitutes a reference for the data integration and exchangeframeworkforbeAWARE.
1.2 "beAWARE"
The beAWARE Project is an EU-funded collaboration (#700475) of partners fromseveral countries in Europe to deliver a prototype disastermanagement system forextreme weather conditions. The Project is focused on Flood, Forest Fire, andHeatwavescenarios,andisintendedfordeploymentandtestingofthesescenariosinVicenza(Italy),Valencia(Spain),andThessaloniki(Greece),respectively.Theendusersare the Alto Adriatico Water Authority (AAWA), Valencia Local Police (PLV), andHellenic Rescue Team (HRT), respectively. In addition, the Frederiksborg FireDepartment (FrederiksborgBrandOGRedning, FBBR) contributed to theoperationalrequirementsfortheFirescenario.
The technical partners involved in the project include: Centre for Research andTechnology – Hellas (CERTH) who is also the coordinator of the project; MotorolaSolutions Israel (MSIL),who is also the technicalmanager of the project; IBM IsraelHaifaResearchLabs(IBM);FinnishMeteorologicalInstitute(FMI);FraunhoferInstitutefor Optronics, System Technologies and Image Exploitation (IOSB); and UniversitatPompeuFabra(UPF).
The beAWARE system is an end-to-end solution for collecting information frommultipledatasources–suchasendusers,socialnetworks,sensors,anddataproviders–analyzingit,predictingandassessingemergencies,alertingthepublic,andmanagingfirstresponders'activities.
1.3 TheneedforaData-SourceIntegrationFramework
Themultitudeofdataoriginatorsandconsumers inflexibleconfigurationswithintheBeAWAREplatformmandates a robust framework for data connectivity, integration,processing,andexchangeamongtheinvolvedparties,modules,andservices.
• An Interoperability Layer, in which the multiple operational stakeholders cancollaborate,coordinate,constructandtransferknowledge,andexchangeusefulinformationinordertocarryoutthemissionsincurredbynaturalemergencyanddisaster.
• An Interconnectivity Layer, in which the services and systems supporting theoperational stakeholders can jointly process and streamline content,information,anddata, inordertosupportoperationalproceduresandbusinessprocessesenabledbythesesystems.
• An Intercommunication Layer, inwhich themultiple input-output interfaces ofthe constituent systems can send and receive standardized data structures aspartof the informational transactionsandservicesprovidedby theconstituentsystems.
Thepartnersengagedinthedevelopmentoftheindividualsubsystemswilladdressthedevelopmentof appropriatemechanisms in their respectiveworking area to complywiththisdataintegrationframework.
Similarly, the enduserswill further build the operational transactions andprotocolsbased on the common language and guidelines described and specified in thisdocumentandaspartofongoingwork.
This report is aligned and coordinated with D7.2 – System Requirements andArchitecture, and adheres to theBeAWAREprinciple of unified “publish—subscribe”messagingbus.Still,thisreportdoesnotfocusonthetechnicalitiesofdatamessagingbutontheoverallprinciplesandconventionsthatensureappropriateandconstructiveutilizationof assets suchas themessagebusandmessageprotocolby the technicalpartnersandtheirprovidedsystems.
1.4 Outline
Thisdocumentisstructuredasfollows:
• Section 2 [Public] discusses the requirements for a data-source integrationframework.
D6.2–V1.0
Page15
• Section 3 [Public] discusses the data source integration infrastructure, whichenablestheinteroperabilityamongthebeAWAREmodulesandsubsystems.
• Section 4 [Confidential] discusses the module-level data access mechanisms,receivable and producible data artifacts applicable to each module, and theinteractionswithothermodules.
• Section5[Public]concludesthisreport.
D6.2–V1.0
Page16
2 Data-SourceIntegrationRequirements[PUBLIC]
2.1 Scope
In this section we review and analyze the requirements for data integration andinteractionamongthebeAWAREmodules.Inaddition,wederivedataintegrationandinteractionneedsandfunctionalitiesatthebeAWARE-levelandatthemodule-level.
2.2 UserRequirementsforData-SourceIntegration
Stakeholders of the operational scenarios that beAWARE is required to respond tohavedefinedvariousoperationalandfunctionalrequirementsforthesystem,inorderto support the roles, responsibilities, and activities of human agents during theoccurrenceofthescenario.ThecompletelistofinitialUserRequirementsisdefinedinDeliverableD2.1[2],whichwaspublishedinM6.
For the purpose of this report, the initial user requirements provided by beAWAREpartners are considered as a general reference for an overall understanding of userneeds, expectations, intentions, and constraints with respect to the data sourceintegration framework. The possible impact of each user requirement on dataintegration and information-based interaction has been analyzed throughout thecourseoftheprojectandthearchitectingoftheframework.Theresultsofthemultipleteleconferences and emails exchange discussions are summarized in the followingsections.Theimpactisspecifiedintherightcolumnineachoftheuserrequirementstablesbelow.
User requirements whose impact resembles that of previously analyzed userrequirements refer to the original user requirements from which the impact wasderived(forinstance,UR#202pointsthereadertoUR#128forimpact).Thegoalwastominimize multiple definitions of the same impact, and to consolidate use cases asderivedfromvarioususerrequirements.Forexample,providingasuitablemechanismfortransferringpublicalertsfromthecontrolcentertothecitizensisthesameforalltheuserrequirementsthatrefertogeneratingpublicalerts,regardlessofthetypeofhazardtheyintendtowarnabout,thespecificityoftheapplicablelocationorarea,andthecontentofthealert.
D6.2–V1.0
Page17
Weemphasizethattheimpactsderivedfromtheuserrequirementsarestillorientedasmuchaspossibletotheproblemdomain,andaremostlyinteroperability-driven,sothat they reflect operational needs for connectivity and data exchange asmeans tosupport interactions among operational users, e.g. to allow information sharingbetweentheauthorityandthepublic,orthefirstrespondersandthecontrolcenter.Thus, these impacts are still not allocated to specific modules or data accessmechanisms,andservetogetherasthebasisforthedefinitionofthisarchitecture.
2.2.1 UserRequirementsfortheFloodScenario
Inthissectionwesummarizetheanalysisofdatasourceintegrationandconnectivityimpacts for the user requirements defined for the Flood scenario (SCN#1). Asexplained, the impacts are specified on the rightmost column for each userrequirement,andrequirementswithanimpactsimilartotheonealreadydefinedforpreviouslyanalyzed requirements refer to theuser requirementwhichhas the sameimpact.Sincethisisthefirstscenarios’userrequirementsthatwereanalyzed,mostofthe user requirements have a distinct impact in the connectivity and data exchangeaspects.Anegligentportionoftherequirementsrefertomodule-intrinsiccapabilitiesthat do not impact the overall connectivity or require special data exchangemechanisms.Wehavekepttheserequirementsforreference.Afewrequirementsaresufficientlysimilartobecoveredbyasimilarconnectivityorintegrationfeature,suchasthepublicalertormetricvisualization.
Inthissectionwesummarizetheanalysisofdatasourceintegrationandconnectivityimpacts for the user requirements defined for the Fire scenario (SCN#2). Again, theimpacts or pointers to previously analyzed requirements with a similar impact(including user requirements for SCN#1) are specified on the rightmost column foreachuserrequirement.Sincethisisthesecondscenariotobeanalyzed,andsincethisscenario isabitmorerestricted inscope,mostof theuserrequirementsarealreadycoveredbypreviouslydefinedconnectivityorintegrationfeatures.
Inthissectionwesummarizetheanalysisofdatasourceintegrationandconnectivityimpacts for the user requirements defined for the Heatwave scenario (SCN#3).Similarly, the impactsorpointers topreviously analyzed requirementswitha similarimpact(includinguserrequirementsforSCN#1)arespecifiedontherightmostcolumnforeachuserrequirement.Sincethisisthethirdscenariotobeanalyzed,mostoftheuser requirements are already covered by previously defined connectivity orintegrationfeatures,mostofwhichwereidentifiedforSCN#1.Asmallportionoftherequirementsthatwerenotidentifiedashavingconnectivityordataexchangeimpactshavebeenmarkedas“N/A”.
Basedonthedescriptionofthescenariosanduserrequirements,asetofoperationaluser roles has been defined, in order to cover all the participants of the actualscenarios.TheoperationalroleshavebeendefinedandlistedinTable2-4.
Table2-4.OperationalRoles
ROLE LOCATION INTERFACE RESPONSIBILITIES SCENARIOSEmergencyManager(H)
Field FRAPP Reportonnewandexistingincidents,executeassignedtasks,
reporttaskprogress,receiveoperationalalerts
Flood,Fire,Heatwave
Citizen(H) Field SCAPP Reportonnewandexistingincidents,Receivepublicalerts
Flood,Fire,Heatwave
Sensor(M) Field SENSAN Providesensormeasurements Flood
SocialNetwork(M)
Cloud SMA Generatetweets Flood,Fire,Heatwave
2.4 Modules
Basedontheanalyzeduserrequirements,wehaveformedalistofmodulesthatneedtocollaboratethroughthebeAWAREsystem.Althoughsomeof thesemoduleswereconceptually defined at the inception of the project, some emerged as critical andnecessary during our joint work and analysis. For instance, a Media Hub, whichmanagesrequestsformediaanalysisformultiplemediatypes,hasbeenidentifiedasacriticalnodeintheoverallarchitectureinordertocreatearobustcapabilitytohandleincidentreportswithmultiplemediaattachmentsofvarioustypes.
Table2-5.Modules
FULLNAME OWNER ALIAS TYPEAMICOFloodPredictionServices AAWA AMICO Subsystem
AutomaticSpeechRecognition CERH ASR BackendModule
CentralDataRepository IBM CDR StorageService
CentralMessageBus IBM MSB MessagingService
D6.2–V1.0
Page30
FULLNAME OWNER ALIAS TYPECrisisClassification CERTH CRCL BackendModule
The purpose of this chapter is to describe the central mechanisms for data sourceintegrationinbeAWARE,basedonthepreviously-defineddataaccessmodalities.
SincebeAWAREisacomplexsystemwithmanymodulesdevelopedseparatelyacrossEurope, it became evident quite early in the architectural process that a centralmessagingservicemustbeset-up,ratherthanallowameshofpeer-to-peerinterfacesamongthemultiplebeAWAREmodules.Moreover,all thecommunicationmustpassthrough a centralmanager,which provides governance, routing, delivery assurance,andmonitoring.Forthisreason,themainelementoftheinfrastructureisthemessagebus(MSB).
In addition to theMSB,wehave identified the need for a centralmedia repository,thatwillbeusedforstoring images,videos,audio files,anddatasets,sothatratherthanroutethiskindofinformationasrawattachmentstomessages,referencestothemediafileswillbeprovidedalongwiththemessages,suchthatmediaaccesswillonlybeonthebasisofnecessity,andnetworktrafficwouldbeminimized.
TheCentralMessageBus isusedasthemainvehicleforexchanging informationandnotificationsamongdifferentcomponentsintheplatform.Followingamicro-servicesapproacheachcomponentisautonomoustoalargeextentandallinteractionsamongcomponentsareachievedvia thecentralmessagebus.Hence,eachcomponent thathasinformationthatcouldpotentiallybeofinteresttoothercomponentswillsendamessage through themessagebus, andeach component that needs tobe awareofspecificnotificationsfromothercomponentsshallsubscribetoreceivemessagessenton a specific topic. Different components interacting via the message bus need toagree on a topic name and message format so that information is received andunderstood.
The communication bus is realized by using an instance of aMessageHub service1,deployedinIBM’sBlueMixcloud.Theback-endisbasedonaclusterofApacheKafkaservers, and the interactionwith the service is realizedusing standardKafka clients,embeddedwithinthevariouscomponents.
InarepresentativeflowanewpieceofinformationflowsintothesystemviatheAPP.The APP's back-end uses the message bus as a producer to alert all interestedcomponentsastotheexistenceofanewinputdataitem.Theinterestedcomponents,mainly the components in charge of analyzing the incoming data, receive thecorrespondingmessageviathemessagebuscallbackmechanism,bysubscribingtoitstopicasconsumers,andproceedtoanalyzethenewdatabasedonthecontentofthereceivedmessage.Oncetheanalysisisfinished,onceagainthemessagebusisusedbythe analyzing module, now as a producer of a new topic, to inform interestedcomponentsthatnewanalyzedinformationisavailable.OneofthesubscriberstothiskindofinformationistheKBS.TheKBSturnsasaproducertotheMRGasaconsumer,
Belowisagenericconnectivitydiagram,showingtheMSBasthecentralmodule.SincetheMSB is agnostic to the topics it processes and to the clients that produce andconsume messages based on these topics, the diagram shows them as generic –Producers1,2,and3;Consumers1,2,and3;andTopics1,2and3.
Figure3-1.ConnectivityDiagram-MSB
3.2.4 FeaturesandCapabilities
The abstraction provided is that of a "publish—subscribe" (Pub/Sub) middleware.Pub/Subisamessagingpatterninwhichsendersofmessages,calledpublishers,sendmessageswithoutspecifyingthereceivers,butinsteadcategorizepublishedmessagesinto topics. In turn, potential receivers, called subscribers, subscribe to the topics inwhich they have interest, and subsequently only receive messages they havesubscribedto,withoutknowledgeofwhichpublishersoriginatedthem.Thus,differentcomponentsdonotneed tobe awareof eachother, or evenbe active at the sametime. Inaddition,thePub/Submechanismiscontent-agnosticandendpoint-agnostic,i.e.itdoesnotneedtoknowinadvancewhoarethesendersandreceivers,andwhatmessages they exchange. The central message bus infrastructure ensures that allmessagespublishedonaspecifictopicreachallsubscribersofthatspecifictopic.
D6.2–V1.0
Page39
3.2.5 DeploymentandOperation
TherealizationofthecentralmessagebusisviaaclouddeploymentofMessageHub,which isamanagedApacheKafkaclusterdeployedon the IBMcloud (BlueMix).Theserviceitselfisprovidedbyaclusterof5Kafkabrokers(kafka01,..,kafka05).
The specific connection details are provided to the components running inside theplatform'sKubernetescluster(amanagedsetofcontainerizedapplicationsthatmakeup a logical unit). The connection details include a namespace wide secretcorresponding to the deployment namespace, namely prod. The secret contains anAPI-keyandthelistofKafkabrokers.
3.2.6 DataAccessConventions
Access to the message bus capability is provided by using standard client librarieswhich are available in multiple languages. In addition, there is a REST interfaceavailableaswellforcommunicationwiththemessagebus.
3.2.7 Authentication
All Kafka agents (producers and consumers)must authenticate themselveswith thecentral service in order to receive or producemessages. The authentication is doneusing a certificate assigned to the agent, in order to ensure that only allowed datasourcesinteractwiththebeAWAREmessagebus.
3.2.8 CommonTopicHeader
All topics of messages going through the beAWAREMessage Bus share a commonheader, which includes conventional topic, message, and origin identifiers. TheattributesintheheaderarelistedinTable3-1.
ThetopicheaderstructuregenerallyconformstotheCommonAlertProtocol(CAP)[3].ForeveryattributewealsospecifytheequivalentCAPattribute.CAPisastandardizedformat for distributing alerts, warnings, and notifications, especially in emergency-related systems. Since CAP is in principle an XML standard, we created a JSONtemplateforourheaderthatcorrespondstotheXMLSchemaDefinition(XSD)ofCAPalerts. More information about CAP can be be found here: http://docs.oasis-open.org/emergency/cap/v1.2/CAP-v1.2-os.html.
The beAWARE message bus topics are listed in Table 3-2 below. For each topic, aunique identifier in the formatTOPXXX_TOPIC_NAMEhasbeendefined.Each topic'sproducers(orpublishersinthePub/Subterminology)andconsumers(subscribers)arelisted. While the producer is not familiar with the subscribers, the subscribers canknow the producer's identity thanks to its specification in the common header"Originator"attribute. Inaddition,an indirectly-routedmessagecanhavea "sender"
For this purpose, we deployed an instance of the IBMObject Storage in the cloud,providingunstructuredclouddatastorage,foreasystoringandaccessingcontentfromvariousapplications.
3.3.2 SupportedEntities
Mostof theplatformcomponents interactwiththeCDRforuploadingandaccessingmedia or data files, or to perform both operations. In a representative flow a newvideofileisingestedintothesystemviatheAPP.Theapplicationback-endstoresthevideofile intheCDRanddeclarestheexistenceofthenewfileusingamessagesentvia the MSB. The MEDHUB, which subscribed to receive notifications on the sametopic,proceedstoaccessthestoredfileandanalyzeit.Oncetheanalysisiscompleted,anewfile isstored intotheCDR includingconclusionsreachedbythecorrespondinganalysis component, and information is shared with other system components in asimilarmanner.
D6.2–V1.0
Page43
3.3.3 ConnectivityDiagram
Figure3-2.ConnectivityDiagram-CDR
3.3.4 FeaturesandCapabilities
TheCDRismeantasthecentralsecureaccesspointforstoringandaccessingdatabydifferent platform components using a REST interface. It is manifested as a cloud-enabledfilesystem,providingsharingandcollaborationcapabilities.TheCDRservesasa scalable persistent storage layer for the beAWARE platform. Thus, a secure andprotected data repository serving different platform components working withdifferentkindsoffilesisprovided.
ReadilyavailableSDKsandclient librariesexist foravarietyofpopularprogramminglanguages.Amanagementconsolecanbeusedtoconfigurethegenericserviceforthespecific access patterns. In addition, access policies can be configured based on theneedsofthedifferentsystemcomponents.
Currentlytherearetwomainwaystoaccessthecentraldatarepository,directlyviaaclient library or through a helper application which is deployed as a cloud foundryapplication(https://object-store-app.eu-gb.mybluemix.net)insidetheprojectspacein
D6.2–V1.0
Page44
the IBM cloud. The convenience function exposes a REST interface for uploading,retrieving,anddeletingfiles.
3.4 NoSQLDatabase
3.4.1 Overview
The NoSQL capabilities are mainly used for storing and accessing relevant tweetsinformation. The information is stored as JSON documents, and pointers to it arepassedbetweencomponentsthroughthemessagebus.
For the realization of the NoSQL capabilities we use an instance of a MongoDBdeployedonBlueMix(https://www.compose.com/databases/mongodb).
3.4.2 SupportedEntities
The main users of this capability are the SMA and MTA components. In arepresentativeflow,thetweetercrawlerfeedstheSMAcomponentwithnewtweets.If theSMAdetermines thatacertain tweet is relevant, it stores the tweet ina JSONformat in theNoSQLDB, andpublishes the corresponding link via themessagebus.The textanalysiscomponent in turnpicksup this information fromthemessagebusandextractstherelevanttweetinformationfromtheDBusingthesuppliedlink.
3.4.3 DeploymentandOperation
TheNoSQLcapabilityintheplatformisrealizedviaaclouddeploymentofaMongoDBinstance in IBM’s cloud.AKubernetes secret, namedmongo-bw2-secret, is injectedinto the platform namespace providing the required connection information in theformofthecorrespondingURI.
• a connectivity diagram that shows the inputs (on the left-hand side of thediagram) to the module from other modules and mediators (as specified inSection3), and itsoutputs toothermodulesandmediators (on the right-handsideofthediagram),
• a list of receivable inputs including themapping of the operational entity to atechnical data or information entity, and the relevant data accessmechanismwhichfacilitatesit,
• a listofproducedoutputs including themappingof theoperationalentity toatechnical data or information entity, and the relevant data accessmechanismwhichfacilitatesit,
• a subset of themodule's technical requirements from the full list (specified in[4]) that entail interoperability, connectivity, or data access requirements orchallenges,
• a list of mechanisms and interfaces of the module for receiving input orgeneratingoutput.
Altogether, all inputs into a module must be mirrored by an output from anothermodule, and vice versa, such that eventually all the outputs by any module in thesystemareusedbyatleastoneothermodule.Inputsfromexternaldatasourcessuchas social networks or sensors are indicated accordingly, and regarded as boundary-crossinginputs.Similarly,outputstoexternaldatareceiverssuchasoperationalusersareregardedasboundary-crossingoutputs.
Theservicesandmodulesdescribednextrelyondedicatedinnovativetechnologiesorheavily-customized assets, and may be subject to intellectual property. In addition,specificsofthedataexchangeamongthemodulesarenotnecessarilyrelevantinanycontextorforanyapplication,andstemfortheinternaldefinitionsandspecificationsofthebeAWAREplatform.Duetotheseconsiderations,thissectionisconfidential,andavailableonlytoEuropeanCommissionofficersandbeAWAREConsortiumMembers.
D6.2–V1.0
Page46
The full content of this section is available in Deliverable D6.6 – Data-SourceIntegrationFramework–ConfidentialVersion.
D6.2–V1.0
Page47
5 Conclusion[PUBLIC]ThisdocumenthaspresentedthebeAWAREdata-sourceintegrationframework,aimedtoproviderobust interoperabilityamongtheoperationalagents,connectivityamongthe subsystems andmodules, and data exchange and information-based interactioncapabilitiesacrossthesystem.
We have started with identifying the user requirements which are relevant forinteroperability and connectivity, and the common available data accessmodalities,methods,andsolutions.
Itbecameapparentveryearlyinthearchitectingandhigh-leveldesignprocessthatthesystem architecture must rely on a robust and extensible connectivity and dataexchangeinfrastructure,includingassetslikeamessagebus(Kafka),centraldatabase(MongoDB),andcentraldatarepository(ObjectStore).
Inaddition,eachsubsystemandmodulehasbeendesignedtointeractseamlesslywiththe infrastructure as well as with some of the other modules via alternativeconnectivitychannels,forenhancedperformanceandflexibility.Forinstance,thereisa direct interface between the Flood Forecasting service (AMICO) and theWeatherForecastingservicetoobtainup-to-dateandarea-specificweatherforecasts
We have presented the high-level specification of the common infrastructurecomponents, including the supported data access modalities and exchangedoperational and informational entities. In addition,we have presented the technicalrequirementsandsolutionspecificationforconnectivityanddataexchangeservicesineachofthebeAWAREmodulesandcomponents(availableintheconfidentialversiononly).
Finally,wehaveelaboratedontheinformationalentitiesthatwehaveidentifiedoverthe course of the project. We have specified their attributes and appropriate dataaccessmodalitiesandutilizingfunctionalflows.Theseflowsprovideend-to-endvalueby supporting informationexchangeand information-basedcollaborationamong theoperationalusers.
Thisdocumentmarks theoutcomeof the intensiveworkof all technicalpartners aspart of task T6.2, with continuous feedback from end-users, in order to ensure thefacilitationofa robustdata-source integration infrastructure, and the intelligentandefficient utilization of the infrastructure by themodules developed by the partners.During the project it became increasingly critical to ensure the commonality andstandardizationofdataexchangeprocesses,governthedefinitionofnewdatatopics
D6.2–V1.0
Page48
and ensure compliance across the board, especially for the seamless exchange andmanipulationof informationwhilepreservingacommonsemantic language. Inmostcases,itwasfoundthattheKafka-basedMessageBuswasthemostsuitablemeansofexchangingdataamongthemodules,includingbothinformativedataandnotificationsregardingtheavailabilityofdata inotherchannels. Insomecases,RESTfulAPIswereset-up or exploited for on-demand data requisition and retrieval, which allows forsynchronousinteractionsandtransactions,orforasynchronousrequestsofdatawhennotifications are provided. In addition, during the course of the project and theformulationofthesolution,itbecameclearthatastoragesolutionforlargemediafiles(audio, video, image, and dataset) is necessary, and that links to media would beprovidedaspartofKafkamessagesratherthanrawattachments.
Futurepost-projectorPhase-IIextensionsoftheframeworkmayincludemonitoringofdata acceptance via functional acknowledge messages that will be generated byconsumers as topicked messages in their own right in response to incomingoperational topics. This would allow assurance of data availability to the relevantstakeholders,whetherbytheoriginatoroftheinformationorbyaneutralmonitoringauthoritythatwouldoverseeandlatersupervisetheflowofinformationinthesystem.Additionalenhancementsmayincludedifferentiationofservicelevelsaccordingtothecriticality and frequency of various topics, as well as the adoption of topic filteringconventionsinordertoallowhierarchicalprocessingofinformation(e.g.separationofincident topics by category in order to support easy routing of incident updates torelevantprocessingservices).
Toconclude,thecurrentinfrastructurehasproventobeanoptimalsolution,intermsofcost-benefitbalance,andinthecontextofsupportingthechallengingintegrationofthe beAWARE platform. We continue to monitor the correct utilization of theinfrastructureby thebeAWAREmodules, criticalextensionsandenhancements,and,whenthevolumeandthroughputofdatareachacriticalmass,alsotheevaluationofreliability,performance,availability,affordability,scalability,andoverallservicelevel.
D6.2–V1.0
Page49
6 References
[1] Y.Mordecai,O.Orhof,andD.Dori,“Model-Based InteroperabilityEngineering inSystems-of-Systems and Civil Aviation,” IEEE Trans. Syst. Man Cybern. Syst. (toAppear.,vol.TBD,no.TBD,p.TBD,2016.