Usability and Acceptability of the Sahana Situational- Awareness Platform in Myanmar, Maldives, and the Philippines Nuwan Waidyanatha, Biplov Bhandari, and Lutz Frommberger Sahana Software Foundation, USA {nuwan, biplov, lutz}@sahanafoundation.org Abstract Maldives, Myanmar, and the Philippines are vulnerable to natural disasters. Sendai Framework of Action calls for risk reduction by implementing early warning systems (Zia and Wagner, 2015). A prevailing challenge is for authorities to coordinate warnings across disparate communication systems and autonomous organizations (Smith and Bunker, 2009). Cross-Agency Situational-Awareness platforms and the ITU-T X.1303 Common Alerting Protocol (CAP) interoperable data standards presents themselves as solution for diluting the inter-agency rivalries and interconnection disparities (Pagotto and O’Donnell, 2012). The CAP-enabled Sahana Alerting and Messaging Broker (SAMBRO) was designed to overcome these issues by providing a Common Operating Picture and a platform for all Stakeholders to share and disseminate early warnings. To that end, the CAP-on-a-MAP project implemented SAMBRO and the CAP standard along with the policies and procedures, recommended by Christian (2016) in the three countries. The project evaluated the usability and acceptability of the intervention through a ‘gulf of evaluation’ complexity analysis method and by applying the ‘technology acceptance model’. The users ‘agreed’ that SAMBRO was ‘useful’ and ‘easy to use’. Moreover, they had ‘quite’ a good attitude towards adopting and indicated that it was beneficial. This paper discusses the outcomes of the evaluation and the policy implications that would allow for sustaining and scaling the concept of cross-agency situational-awareness for improving institutional responsiveness to coastal-hazards in the three countries. KEYWORDS – Disaster, Innovation, Application, Information, Performance Introduction Disasters are a major problem worldwide and a serious threat to sustainable development. The rapid an often unplanned expansion of human settlements is exposing more people and economic assets to the risk of disasters and the effects of climate change in these countries (WEF, 2015). Coastal cities are made vulnerable by
18
Embed
Abstract€¦ · ITU-T X.1303 Common Alerting Protocol (CAP) interoperable data standards presents themselves as solution for diluting the inter-agency rivalries and interconnection
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Usability and Acceptability of the Sahana Situational-Awareness Platform in Myanmar, Maldives, and the
Philippines
Nuwan Waidyanatha, Biplov Bhandari, and Lutz Frommberger
Sahana Software Foundation, USA
{nuwan, biplov, lutz}@sahanafoundation.org
AbstractMaldives, Myanmar, and the Philippines are vulnerable to natural disasters. Sendai
Framework of Action calls for risk reduction by implementing early warning systems
(Zia and Wagner, 2015). A prevailing challenge is for authorities to coordinate
warnings across disparate communication systems and autonomous organizations
(Smith and Bunker, 2009). Cross-Agency Situational-Awareness platforms and the
ITU-T X.1303 Common Alerting Protocol (CAP) interoperable data standards
presents themselves as solution for diluting the inter-agency rivalries and
interconnection disparities (Pagotto and O’Donnell, 2012). The CAP-enabled Sahana
Alerting and Messaging Broker (SAMBRO) was designed to overcome these issues
by providing a Common Operating Picture and a platform for all Stakeholders to
share and disseminate early warnings. To that end, the CAP-on-a-MAP project
implemented SAMBRO and the CAP standard along with the policies and procedures,
recommended by Christian (2016) in the three countries. The project evaluated the
usability and acceptability of the intervention through a ‘gulf of evaluation’
complexity analysis method and by applying the ‘technology acceptance model’. The
users ‘agreed’ that SAMBRO was ‘useful’ and ‘easy to use’. Moreover, they had
‘quite’ a good attitude towards adopting and indicated that it was beneficial. This
paper discusses the outcomes of the evaluation and the policy implications that would
allow for sustaining and scaling the concept of cross-agency situational-awareness for
improving institutional responsiveness to coastal-hazards in the three countries.
The subjective assessment presented the users with the TAM questionnaire. Figures 3,
4, and 5 show the average scores associated with the ease-of-use, usefulness, and
attitude towards using SAMBRO and the CAP warning standard, respectively. The
scale for ease-of-use and usefulness were based on the likert scale: 1 = strongly
disagree, 2 =disagree, 3 = neither agree nor disagree, 4 = agree, and 5 = strongly
agree. On average the users 'agreed’ that SAMBRO was a “useful” tool and was
closer to ‘agreeing’ that it was “easy to use”.
The assessment for quantifying the
attitude towards using SAMBRO in their
business practices, used a likert scale
between 1 and 7. The lower extreme ‘1’
indicated that it was bad, harmful,
foolish, and negative and the upper
extreme indicate it was good, beneficial,
wise, and positive. These terms would
carry a prefix of ‘extremely’, ‘quite’,
‘slightly’, or ‘neither’ (e.g. slightly-good
or quite-bad). The users had to chose a
number between 1 and 7 to complete a sentence: “all things considered I find
SAMBRO to be __________ for my Organization’s warning dissemination practices.”
The results in Figure 5 show that, on average, the users’ attitude towards adapting
SAMBRO was closer to being one that was “quite” good, beneficial, wise, and
positive.
Discussion and FindingsThe key question we asked was “did the technology and the people work as planned
on the day of the controlled-exercise?” The overall observation was that there were
several shortcomings and the users were unprepared. The overall outcomes of the
measurable results were above the median but not convincingly strong to support that
the usability was satisfactorily high. There were several technical, organizational and
policy challenges that the project met, in relation to operationalizing SAMBRO. The
Figure 4: User perceived ease of use Figure 5: User perceived usefulness
Figure 6: User perceived attitude for using
challenges will be exemplified through the data from the controlled-exercises and
related findings.
User engagementThe project implementers emphasized the need for frequent training, experience using
the system and have a fully prepared and tested implementation before the controlled-
exercises. The controlled-exercises should not be a surprise for any of the participants.
As part of the participatory user-centered design and development approach
Stakeholders were engaged through several workshops and activities. They underwent
several cycles of the design, develop, implement, test, and redesign cycles. In many
cases the participation was inconsistent with new faces at each event.
The commitment from the lead organizations were good but the other beneficiary
organizations treated these activities as just another event that they had to participate
in. Moreover, the participants were junior staff who did not have the authority to
convince the senior management to continue and scale the program within their own
organizations. As a result it was only the lead organizations that accepted and began
utilizing the SAMBRO system for disseminating CAP-enabled warning messages.
Table 2 shows Maldives to have a significantly low number of participants, relative to
Myanmar and the Philippines. The Maldivian Stakeholders did not perceive the
exercises important over other commitments. For example, on the day of the exercises
the Myanmar Meteorological Services, who represented a larger portion of the project
participation and involvement, were busy working on their expiring ISO certification
to be able to continue as an accredited service to support the national aviation
services.
Technology Usability
Table 3 shows the Philippine users exercise a level of inconsistency and uncertainty
because the controlled-exercise was the first exposure for some of the users. PAGASA
users were the only participants competent in issuing tropical cyclone warnings. It is
because they had been using their own homegrown CAP message publishing system
before SAMBRO was introduced. However, issuing flood warnings was new to them;
a reason for the drop in the average simplicity score. (Table 3)
Myanmar DMH users had been testing and practicing the use of the system. However,
they had not practiced the task of updating an alert. Typically, a messaging cycle
requires first issuing an alert, then updating its status with new verified information,
and finally issuing an all-clear message when the threat has ceased. The difficulties
were mostly in populating the CAP 'info' block because it requires a lot of detail with
respect to the event information. The users exercised doubt in changing the
information when they were updating a message.
Philippines and Maldives (Island Nations), data in Table 3, show similar patterns of a
bell-shape timing distribution to issue an alert, update, and all-clear message. When
an event is detected the duty officers have the authority to verify the event on their
own and issue a message. It is during the message update that they consult with the
supervising officers and experts to receive and verify the telemetries to include in the
message update. This process take more time; hence, the increase relative to the first
issuing of the event specific alert message. Myanmar, on the other hand, consults with
the DMH supervising officers and experts to verify all information before issuing the
first alert. Therefore, they spend a longer duration with the first alert. Subsequent
updates and all-clear messages are handled by the duty officers and does not require
supervision.
In relation to a tsunami alert, the time to complete an alert is too long. Cyclones,
volcanoes, and floods are slow onset hazards. A tsunami originating from the pacific
ring of fire, near the Philippines, gives Metro Manila and the eastern coastline less
than 10 to 20 minutes. If the tremor is not felt, such as in the middle of night when
sleeping, there is very little time to issue a wakeup alert and for people, living on the
Islands, to evacuate. The same applies to Myanmar, if a tsunami was to originate of
the northern parts of the Bay of Bengal. The nearest coastal inhabitants would have
less than 20 minutes to evacuate. Table 3 shows both countries, Philippines and
Myanmar, to have exceeded the time allowed to issue a warning allowing for an
adequate response time to save lives.
The Myanmar CAP message Publishers, relative to the other two countries, frequently
experimented with the system before the controlled-exercises. Nevertheless, Table 3
shows the average CAP content errors to be similar between Philippines and
Myanmar. The users need to develop their CAP competency to give prominence to
message completeness. The Researchers examined each and every CAP message file
to identify any information elements with inappropriate or inconsistent content to
count the number of errors for each information element. There are a total of 42
information elements of which less than 10 are frequently used. To that end the users,
on average, incorrectly used or populated 4 of the 10 CAP information elements. The
All-Clear stage of the alerting cycle shows the most number of errors. The concept of
issuing an all-clear message is important and one that was introduced through the
project and new to them.
Although Myanmar users were, relatively better prepared on the day of the exercise,
their Internet failed. As a result during the cyclone alerting exercise, they spent more
than 32 minutes before they could send out the message. The same issue prevailed
with the earthquake-tsunami controlled-exercises when it took the team more than 16
minutes to issue an update message. The duration was calculated starting from the
time the user exercised the software login process to the time the alert message was
received by a recipient, either by email, mobile-app, or social media. The fault was
not at the DMH data center; instead it was the Ministry of Transportation and
Communication data center that serves as the DMH Internet Service Provider. Their
data center electricity is unreliable, and when the power fails the Internet fails.
Diederichs et al. (2014) indicate that related constructs to user acceptance are
usability or user experience which is known to enhance the probability of user
acceptance. SAMBRO offered a generic user interface that could be used for all-
hazards. Predefined message templates, area polygons, and warning priorities would
populate an alert, update, or all-clear messages at the time of authoring. However, the
project learned that hazard specific graphic user interfaces could further simplify and
improve the usability, which in turn would improve the acceptability. The mobile-app
instigated this finding because the mobile-app has a limited number of data inputs it
was perceived easier to use. A simpler graphic user interface would reduce the number
of inputs, chance of errors, and other inconsistencies. These lessons will serve as
inputs to future versions of SAMBRO releases.
Technology AcceptanceThe users in the last-mile communities had not receive any training and were not
exposed to the SAMBRO system and work flows prior to the exercises. A noteworthy
findings is that the community participant expressed usefulness and attitude towards
using scores are in par with that of the Publishers (Figure 4 and Figure 5). Despite the
repetitious request from the project asking to train and expose to SAMBRO the
Stakeholders at every layer of the administrative structure, it was unfulfilled. The
main reason was the silo-ed and bureaucratic structures did not make it easy on the
lead organizations to get the approval from the national layers of the other beneficiary
organizations. Each country has a National Disaster Management Committee.
However, collaboration among member organizations, in projects as such, remains a
challenge.
One reason for the users, in Myanmar and the Philippines, to score low on the
acceptance was because SMS was not operational as notification service. All the users
were keen on being notified through SMS. Myanmar and Philippines were unable to
acquire a SMS gateway for disseminating alerts. Bhandari et al. (2016) discuss the
intricacies of operationalizing an SMS gateway in the three countries. In summary –
We learned that the Myanmar Post and Telecommunications (MPT), now under the
Ministry of Transportation and Communication, same as DMH, had never offered a
Bluk SMS service before. They do offer a decoupled SMS notification service,
whereby a human operator re-creates and sends the text to a list of Government senior
managers. PAGASA could have leverage the Philippines “free mobile disaster alerts
law”; however, remains unused since it was approved in June 20141. The project
facilitated for the Communications Authority of Maldives (CAM) to provide an SMS
gateway through the mobile operators. It never materialized; instead NDMC worked
out a public private partnership with a local Bank to purchase a bulk SMS service.
CAP and SAMBRO allows for presenting warning content in multiple languages. A
separate CAP ‘info’ block can be designated for each language specific content.
Message subscribers (or recipients) have the option of indicating their choice of
language when configuring their own user specific messaging parameters on
SAMBRO. Myanmar is taking advantage of this to issue alerts in Burmese and
English. The last-mile users, in the Philippines and Maldives, were keen in receiving
warning messages in the local languages; i.e. in Tagalog/Cebuano and Divehi,
respectively. However, the national authorities assumed that it was adequate to
provide the messages in English. The project did not conduct a comprehension or
appropriateness test to confirm this decision instead left it to the discretion of the
national implementers to decide on their implementation policies.
1 An act mandaring the telecommunications service providers to send free mobile alert in the even to natural and man-made disasters and calamities, Republic Act No. 10639: http://www.gov.ph/2014/06/20/republic-act-no-10639/
Policy Implications
Scaling the situational-awareness platform
The national stakeholders, in the three countries, are considering scaling CAP to
integrate alert feeds from other alerting authorities as well as expand on the recipient
user base. After realizing the outcomes and potential of the cross-agency situational-
awareness platform for the efficient and effective coordination of alert, other
stakeholders are, now, keen on getting on board. For such the lead organizations were
planning on presenting the intervention and the outcomes to their respective National
Disaster Management Committees.
CAP working group
From a policy perspective the National Disaster Management Committees might
consider paving ways to improve cross-agency situational-awareness within their
jurisdictions. They need to establish policies and plans to foster a wider scale of
stakeholder engagement and adoption. For such, a first step might be with establishing
a CAP working group. This is an important starting point for directing resource for
setting national training, certification, standards implementation, and guidelines for
operations. It would foster uniformity across all stakeholders.
ConclusionThe CAP interoperable warning standard and the SAMBRO cross-agency situational-
awareness platforms are operational in Maldives, Myanmar, and the Philippines. This
is an important step towards integrating the silo-ed national organizations for
improving institutional responsiveness to all-hazards. There is a growing need for
establishing national policies and plans to better integrate systems and especially the
last-mile communities, which remains neglected. Cross-agency situational-awareness
and SAMBRO is capable of bridging those gaps. However, it is a new concept to the
beneficiaries. Nevertheless, it is an intervention that was perceived useful with a
positive attitude towards adoption and scaling beyond the project phase. Lessons
learned point towards the need for some adjustments and enhancements to SAMBRO
to improve the ease of use.
AcknowledgementThis work was made possible through the United Nations Economic and Social
Commission for Asia and the Pacific Trust Fund for Tsunami, Disaster, and Climate
Preparedness (No. 2014-35). This work could not have been possible without the
cooperation of the Governments of Maldives, Myanmar, and the Philippines. The
Authors also wish to express gratitude to the project partner: the Asian Institute of
Technology in Thailand.
Reference
A. Zia, and C.H. Wagner, “Mainstreaming early warning systems in development and
planning processes: Multilevel implementation of Sendai framework in Indus
and Sahel”, International Journal of Disaster Risk Science, 6(2), pp. 189-199.,
2015
B. Bhandari, A-B. Marthafifsa, M-K. Hazarika, F. Boon, L. Frommberger, and N.
Waidyanatha. (2016). Intricacies of Implementing an ITU-T X.1303 Cross-
Agency Situational-Awareness Platform in Maldives, Myanmar, and the
Philippines, ITU Kaleidescope, Thailand, 13-17 November 2016
E. Christian. (2016). Towards a CAP-enabled National Alert System: Some Notes for
the Government of Mexico, 01 July 2016. Consulted 2017-01-10 on the web: