Crowdsourced Damage Mapping for Disaster Emergency ...

What is crowdsourcing?Crowdsourced damage mapping

2015 Nepal earthquakeResearch on collaborative/crowdsourced damage mapping

Crowdsourced Damage Mapping for DisasterEmergency Response - the 2015 Nepal Earthquake

Case StudyUnited Nations/India Workshop on the Use of Earth Observation Data

in Disaster Management and Risk Reduction: Sharing the AsianExperience, 8-10 March 2016

Michal Bodnar

Beihang University

March 9, 2016

Michal Bodnar Crowdsourced Damage Mapping for Disaster Emergency Response - the 2015 Nepal Earthquake Case Study



Table of Contents

1 What is crowdsourcing?

2 Crowdsourced damage mappingRemote sensing based damage assessmentHistory of crowdsourced damage mapping

3 2015 Nepal earthquakeTomnod campaign

4 Research on collaborative/crowdsourced damage mapping




Table of Contents








What is crowdsourcing?

Definition”A distributed problem solving and production model. Problems arebroadcast to an unknown group of solvers in the form of an open call forsolutions. Users – also known as the crowd – typically form into onlinecommunities, and crowd submits solutions. The crowd also sorts fromthe solutions, finding the best ones”.

Chilton, 2010




Famous crowdsourcing projects

Wikipediaa

Zooniverseb

OpenStreetMapc

many others ...ahttps://www.wikipedia.org/bhttps://www.zooniverse.orgchttps://www.openstreetmap.org


https://www.wikipedia.org/

https://www.zooniverse.org

https://www.openstreetmap.org



User-Generated (Spatial) Content

Source: Michal Bodnar




Crowdsourcing

Poblet et al., 2014 divided crowdsourcing activities into 4 types:

Types of crowdsourcing1 Crowd as a sensor - passive, raw data (data from mobile devices)2 Crowd as a social computer - passive, unstructured data (data from

social media, such as Twitter, Facebook)3 Crowd as a reporter - active, semi-structured data (social media with

the purpose of informing, such as Ushahidi)4 Crowd as microtasker - active, structured data, using special tools

(such as HOT, Tomnod, SBTF, ...)




Crowdsourcing pyramid

Figure: Crowdsourcing pyramid (Poblet, 2014)




Remote sensing based damage assessmentHistory of crowdsourced damage mapping

Table of Contents









Crowdsourced damage mapping

Remote sensing based damage assessmentrapid damage assessment is one of the key parts of the emergencyresponse stage for majority of the disasters (earthquake, typhoon,hurricane, ...) (Boccardo and Tonolo, 2013)conducting rapid damage assessment by analyzing (very)high-resolution satellite imagery has been very much used in recentyears, with 2010 Haiti earthquake regarded as a real expansion inusing such sources of informationin such cases, method of visual image inspection has been proved asthe most accurate one over the (semi-)automated methods (Dongand Shan, 2013; Barrington et al., 2011; Voight et al., 2011; Chini,2009; Lemoine et al. 2013)the timeliness of such derived product is the most important factor(Lemoine et al., 2013; Voight et al., 2011; Chini, 2009; Barringtonet al., 2011)






in cases, when the extent of the damage is high, remote sensingexperts are not capable of analyzing the satellite imagery solely bythemselves in a timely manner ⇒ approach of crowdsourcing hasbeen usedemploying ”crowd as a microtasker” strategyconsisting of 3 parts (Barrington et al. 2011):

1 dividing the task into manageable components (microtasking)2 motivating a large user base to contribute (crowdsourcing)3 combining all responses of various quality into a complete solution

(consensus)






Source: https://irevolution.files.wordpress.com/2014/05/screen-shot-2014-05-29-at-6-30-27-am.pngMichal Bodnar Crowdsourced Damage Mapping for Disaster Emergency Response - the 2015 Nepal Earthquake Case Study




Image source: http://files.umwblogs.org/blogs.dir3114files201304MadsNissen Rampen144.jpg





Image source: http://christianals.com/wp/wp-content/uploads/2012/05/Haiti001.jpg





Image source: http://inapcache.boston.com/universal/site graphics/blogs/bigpicture/new zealand quake/bp36.jpg




Tomnod campaign

Table of Contents








Tomnod campaign

Nepal earthquake 2015

Situation overviewApril 25, 2016, 11:56 am, 7.6 magnitude earthquake as recorded byNepal’s Seismological Centre (NSC)76 km northwest of the capital city of Kathmandufollowed by more than 300 aftershocks greater than magnitude 4.0(as of June 7, 2015), out of which four aftershocks were greaterthan 6.0 (Government of Nepal, 2015)31 out of 75 districts were affected, out of which 14 were declared’crisis-hit’

Source: http://alibi.com/image/pix_id/ol_3644/Flag-of-Nepal.jpg


http://alibi.com/image/pix_id/ol_3644/Flag-of-Nepal.jpg



Tomnod campaign


Source: Michal BodnarMichal Bodnar Crowdsourced Damage Mapping for Disaster Emergency Response - the 2015 Nepal Earthquake Case Study



Tomnod campaign

Nepal earthquake 2015 - Damage assessment





Tomnod campaign

Tomnod

a crowdsourcing project which aims to engage public to help withsearching through the satellite imagerymission: to utilize power of crowdsourcing to identify objects andplaces in satellite imagesbelongs to Digital Globe company, which also provides the veryhigh-resolution imagerytheir campaigns have wide range of focus (humanitarian crises,disasters, mapping basic infrastructure)




Tomnod campaign

HOT (Humanitarian OpenStreetMap Team)

evolved from OSM, ”the Wikipedia of Maps”mission: to apply the principle of open data sharing to humanitarianresponse and economic developmentit was formed during (informally) and then after 2010 Haitiearthquakefree for everyone to use, but requires a sign up process and a bit ofexperience as well




Tomnod campaign

Tomnod vs HOT





Tomnod campaign


Tomnod campaignthe campaign started on April 27four different features were mapped:

1 Damaged building2 Damaged road3 Major destruction4 Tent/shelter

the users were asked to tag these features by comparing pre- andpost- event imagery




Tomnod campaign




Tomnod campaign




Tomnod campaign

Tomnod image coverage




Tomnod campaign


Tomnod campaignI have analyzed Tomnod data in these ways:

1 using intrinsic methodsby looking at its geographical distributionby investigating the dataset’s attributes (agreement and score)by looking at the taggers and their statistics

2 by comparing against the official data fromUNOSAT/NGA/Copernicus

the data for the analysis was emailed by the Tomnod team andcontained the tagged features until May 3for my analysis, I have worked with vector data of both Tomnod andUNOSAT/NGA/Copernicus




Tomnod campaign

Tomnod - Number of tags per class (as of May 3)





Tomnod campaign

Tomnod - Geographical distribution

Nepal has 5 levels of administrative units1

Nepal ADMIN units1 ADMIN 1 level - 5 Development regions

Far-WesternMid-WesternWesternCentralEastern

2 ADMIN 2 level - 14 Zones3 ADMIN 3 level - 75 Districts4 ADMIN 4 level - 3157 Village development committees5 ADMIN 5 level units

1https://en.wikipedia.org/wiki/Administrative_divisions_of_NepalMichal Bodnar Crowdsourced Damage Mapping for Disaster Emergency Response - the 2015 Nepal Earthquake Case Study

https://en.wikipedia.org/wiki/Administrative_divisions_of_Nepal



Tomnod campaign




Tomnod campaign


Distribution per ADMIN 1 region [%]

Region Damagedbuilding Damaged road Major

destruction Tent/shelter

Eastern 0.1 0 0.07 0Central 90.37 50 70.31 93.87Western 7.09 29.27 6.03 6.13Mid-Western 0 0 0 0

Far-Western 0 0 0 0

out ofNepal 2.44 20.73 23.59 0

Source: Michal Bodnar.




Tomnod campaign




Tomnod campaign


Distribution per ADMIN 2 region [%]

Region Zone Damagedbuilding

Damagedroad

Majordestruction

Ten-t/shelter

Central Bagmati 76.9 30.49 30.19 93.87Janakpur 8.72 13.41 26.66 0Narayani 4.74 8.54 6.72 0

Western Dhaulagiri 0 0 0 0Gandaki 6.76 25.61 5.93 5.94Lumbini 0.34 3.66 0.01 0.19





Tomnod campaign




Tomnod campaign


Distribution per Bagmati zone [number of features]

District Damagedbuilding

Damagedroad

Majordestruction Tent/shelter Total

Bhaktapur 70 0 23 0 93Dhading 79 1 241 11 332Kabhrepalanchok 86 4 20 0 110Kathmandu 177 2 61 308 548Lalitpur 149 7 19 144 319Nuwakot 967 7 353 27 1354Rasuwa 38 0 6 0 44Sindhupalchok 39 2 184 0 225





Tomnod campaign




Tomnod campaign


Distribution per Janakpur zone [number of features]

District Damagedbuilding

Damagedroad

Majordestruction Tent/shelter Total

Dhanusha 2 1 24 0 27Mahottari 8 3 391 0 402Sarlahi 147 3 383 0 553Sindhuti 24 4 3 0 31Ramechhap 1 0 0 0 1Dolakha 0 0 0 0 0





Tomnod campaign


Top 5 ADMIN 3 districts per tag typePosi-tion

Damagedbuilding

Damagedroad

Majordestruction Tent/shelter

1 Nuwakot Manang Chitawan Kathmandu2 Kathmandu Nuwakot Mahottari Lalitpur3 Lalitpur Gorkha Sarlahi Gorkha4 Sarlahi Lalitpur Nuwakot Nuwakot5 Kabhrapalnchok Chitawan Dhading Dhading





Tomnod campaign


First observationsADMIN 1 - in total, 80 % of all the features were within CentralDevelopment RegionADMIN 2- 53 % of all the features were tagged in Bagmati zone,followed by 17,5 % features within Janakpur regionADMIN 3 - Nuwakot district (Bagmati zone) contained 23,8 % of allthe features tagged, followed by Sarlahi (Janakpur zone) andKathmandu (Bagmati zone) districtsGandaki zone (epicentre of the earthquake) had a very few featuresclassified compared to Bagmati zonethere were 21 % and 23,5 % of Damaged road and Majordestruction features, which fell into area of out Nepal, respectively




Tomnod campaign

Tomnod campaign dataset

Dataset attributesFID - a unique identifiertype - the type of the tagged featuretagger id - an ID of the user who placed a tagscore - the confidence score calculated by Tomnod’s CrowdRankalgorithm for each featureagreement - the number of people who placed a tag over a samefeature




Tomnod campaign

Tomnod - agreement attributeminimum 1maximum 43mean 7.39standard deviation 4.97

Source: ArcGIS 10.2, Michal Bodnar




Tomnod campaign

Tomnod - Frequency of agreement values per class





Tomnod campaign

Tomnod - Statistics of agreement values per class

Damagedbuilding Damaged road Major


minimum 1 1 1 2maximum 42 19 43 12mean 7.05 4.95 8.27 4.09standarddeviation 4.46 4.41 5.39 1.62

Source:ArcGIS 10.2, Michal Bodnar.




Tomnod campaign

Tomnod - score attributeminimum 0.85011maximum 1mean 0.973867standard deviation 0.0395





Tomnod campaign

Tomnod - Frequency of score values per class

Source: ArcGIS 10.2, Michal BodnarMichal Bodnar Crowdsourced Damage Mapping for Disaster Emergency Response - the 2015 Nepal Earthquake Case Study



Tomnod campaign

Tomnod - Statistics of score values per class

Damagedbuilding Damaged road Major


minimum 0.850111 0.855924 0.850126 0.851148maximum 1 1 1 1mean 0.971581 0.976194 0.97762 0.961046standarddeviation 0.041003 0.041529 0.037503 0.041183

Source: ArcGIS 10.2, Michal Bodnar.




Tomnod campaign

Tomnod - Statistics about taggers

42,5 % of taggers tagged only 1 feature77,85 % of taggers tagged 5 or less features11,86 % of taggers tagged 10 or more features

Taggers and tagsnumber of taggers 1141minimum number of tags 1maximum number of tags 117mean number of tags 4.99standard deviation 9.17




Tomnod campaign

Tomnod - Statistics about taggers




Tomnod campaign

Tomnod - top 10 taggers

Top 10 taggers tagged together for almost 13 % of all the taggedfeatures.

Top 10 taggers of Tomnod campaign

Rank ID No. oftags

averagescore

averageagreement tagged

1 3234567 117 0.961266 3.6 MD2, T/S2 92197 97 0.97919 8.39 MD, DB, T/S3 14927859 90 0.959876 5.42 MD, DB, T/S4 584393 72 0.990796 9.78 MD5 882027 66 0.969293 3.27 T/S6 713428 63 0.977455 7.57 MD, DB7 4809316 57 0.966001 6.96 DB8 14877422 52 0.977467 7.75 MD, DB9 3922680 52 0.984814 6.90 MD, DB10 14900490 49 0.978841 6.81 MD, DB


2Major destructionMichal Bodnar Crowdsourced Damage Mapping for Disaster Emergency Response - the 2015 Nepal Earthquake Case Study



Tomnod campaign

Comparison against official datasets

What was compared?1 Tomnod Damaged building vs. UNOSAT-Copernicus building

damage2 Tomnod Major destruction vs. NGA damaged zones3 Tomnod Tent/shelter vs. NGA IDP Camps

Methodology1 the total number of items on district level was compared2 the completeness (error of ommission and commission were

computed)




Tomnod campaign

Comparison against official datasets

Limitationsthe data from official institutions do not necessarily have to be themost precise onethe data from official institutions can also lack completeness (mostprobably do)the date of the release of the datasets differed from each otherthe underlying imagery used for various assessments was also not thesame




Tomnod campaign

Comparison no.1 - Tomnod Damaged building vsUNOSAT-Copernicus




Tomnod campaign


Tomnod UNOSAT-CopernicusDate of dataset May 3 May 7Number of features 2087 6634Classification Damaged Destroyed

Severe damageModerate damage

Data type Point Point





Tomnod campaign






Tomnod campaign

Comparison no.2 - Tomnod Major destruction vs NGA




Tomnod campaign


Tomnod NGADate of dataset May 3 May 7Number of features 3004 8241Classification Damaged Destroyed

Severe damageModerate damagePossible damage

Data type Point Polygon





Tomnod campaign





Tomnod campaign

Comparison no.3 - Tomnod Tent/shelter vs NGA




Tomnod campaign


Tomnod NGADate of dataset May 3 April 29Number of features 522 2111Data type Point Polygon





Tomnod campaign





Tomnod campaign

Accuracy assessment

following the methodology of Foulser-Piggott et al.2013, I tried toassess the quality of Tomnod data against the official data bycomputing its:

1 error of ommission - refers to the number of features in the referencethat were not tagged by Tomnod

2 error of commission - refers to the number of Tomnod features thatwere tagged a certain class but it was not included in the referencedataset

sampling was performed by choosing areas of approximately 2 x 2km, where both Tomnod features and reference data hadapproximately the same number of instancesfor each analysis, 4 such areas were used, preferably from differentdistrictsthe analysis was simplified by not taking into the account a differentclassification level of NGA damaged zones or UNOSAT-Copernicusbuilding damage assessment




Tomnod campaign

Results of the assessment

Accuracy assessment [%]Major


error ofommission 71.2 76.88

error ofcommission 39.1 10.7





Tomnod campaign

Accuracy assessment - Discussion

sampling was shown to be very challenging, particularly forcomparison of building damage, where it was almost impossible tofind areas of overlapthe lowest error of commission was performed by Tent/shelterfeature, while the omission error was the lowest for Damagedbuilding featurethe quality of the analysis highly depends on the fact how precisewas the assessment provided by UNOSAT/Copernicus/NGAthe best assessment of Tomnod would be done by using groundtruth data (not available now)




Tomnod campaign

Conclusion from analyzing Tomnod data

Major destruction feature had the highest number of featuresDamaged road feature did not cause enough attention fromvolunteers and counted together only 82 features taggedin general it seems like volunteers can be quite good at spotting thefeatures of the same shape and material, such was the case ofTent/shelter featurethe most challenging seems to be the building damage assessmentsurprisingly, a high proportion of Major destruction features wastagged outside of Nepal, which would not have a use for groundrespondersGandaki zone, where the earthquake struck, had really lowproportion of tagged features compared to Bagmati zone, wherecapital city of Kathmandu is located




Tomnod campaign

Conclusion from analyzing Tomnod data

the average agreement value was 7.3 and the overall distributionproved have a long tail tendencyin total there were 1141 taggers for 5695 features with themaximum number of 117 tags for a volunteer43 % of volunteers tagged only 1 feature, while only 12 % of themtagged 10 or morethe next analysis could look at:

the quality level of individual taggerscorrelation between agreement or score values and the accuracythe comparison against the ground truth data




Tomnod campaign

My opinion on crowdsourced damage assessment

generally, it is a great way how to engage people in both working forgood cause and recognizing a power of satellite imagerydue to importance of timeliness, it can become a good source ofinformation in the emergency response stageit can serve as a good indication of the signalizing the hotspots ofwhere damage was occuredthe quality of collaborative damage mapping still remainsquestionable, but we need to learn which features the volunteers arebest at classifying




Table of Contents








Quality of crowdsourced damage assessment

assessing the quality by comparison against ground truth is usuallyconducted a long time after the disaster, once the field survey wasconductedsuch approach is however not helpful to be used during theemergency response stageit is important to look at the quality of the crowdsourced damageassessment using intrinsic data measures and assess itsfitness-for-purposefor crowdsourced damage assessment, this task is really hard one assuch process is an output of three variables:

1 Volunteers2 Image3 Damage




Factors influencing the quality of crowdsourced damagemapping

Source: Michal Bodnar.Michal Bodnar Crowdsourced Damage Mapping for Disaster Emergency Response - the 2015 Nepal Earthquake Case Study



First steps - GEOCAN accuracy model

Foulser-Piggott et al.,2013 proposed a qualitative multi-attributedecision model as a first approach to assess quality of thecrowdsourced damage based on the multiple factorsto each of the attributes, they assigned a certain weightthis weight assessment was based solely on the experience andassumptions of the authors, but not by performing any kind ofexperimental analysis

Source: Foulser-Piggott et al., 2013.Michal Bodnar Crowdsourced Damage Mapping for Disaster Emergency Response - the 2015 Nepal Earthquake Case Study



Enhanced decision model - experimental study

in the previous model, various factors are omitted, especiallydemographic characteristics of the volunteersthere will be an experimental study performed at Beihang universityon a certain number of volunteers, who will be studied over a certainperiod of time over a series of eventsvolunteers will be chosen in a way they would provide a sample ofthe population which is usually employed in such events (differentlevel of experience, age, nationality, gender)each event, they will be asked to perform a damage assessment ofthe area for a certain time by analyzing different type of imagery(airborne, satellite)the results will be then used to compute the relative influence ofeach of the attributes discussed on the overall quality and to build amulti-criteria qualitative decision model




Thank You for Your attention!




References

[1] Goodchild, M.F. Citizens as sensors: the world of volunteered geography [J].Geojournal , V69 (4), 2007: 211-221[2] Dong L., Shan J. A comprehensive review of earthquake-induced building damagedetection with remote sensing techniques [J]. ISPRS Journal of Photogrammetry andRemote Sensing, V84, 2013: 85-99[3] Poblet M., Garcıa-Cuesta E., Casanovas P. IT Enabled Crowds: Leveraging theGeomobile Revolution for Disaster Management [C]. Proceedings of the SintelnetWG5 Workshop on Crowd Intelligence: Foundations, Methods and Practices: 2014[4] Goodchild M.F., Linna Li. Assuring the quality of volunteered geographicinformation [J]. Spatial Statistics, V1, 2012: 110 - 120[5] Foulser-Piggott R., Spence R. and Brown D. The use of remote sensing forbuilding damage assessment following 22nd February 2011 Christchurch earthquake:the GEOCAN study and its validation [R]. Cambridge Architectural Research Limited,2013: 1-45[6] Barron Ch., Neis P., Zipf A. A Comprehensive Framework for IntrinsicOpenStreetMap Quality Analysis [J]. Transactions in GIS, V18 (6), 2014: 877-895[7] Pourabdollah A., Morley J., Feldman S., Jackson M. Towards and AuthoritativeOpenStreetMap: Conflating OSM and OS OpenData National Maps’ Road Network[J]. ISPRS International Journal of Geo-Information, V2, 2013: 704-728




References

[8] Chilton S. Crowdsourcing is radically changing the geodata landscape: case studyof OpenStreetMap [G].[9] Barrington L. et al. Crowdsourcing earthquake damage assessment using remotesensing imagery [J]. Annals of Geophysics, 54, 6, 2011: 680-687.[10] Lemoine G., Corbane C., Louvrier C., Kauffmann M. Intercomparison andvalidation of building damage assessments based on post-Haiti 2010 earthquakeimagery using multi-source reference data [J]. Natural Hazards and Earth SystemSciences Discussion: 1, 2013: 1445 - 1486.[11] Government of Nepal, National Planning Commission. Nepal Earthquake 2015:Post Disaster Needs Assessment [R], 2015: 1-20.[12] Voigt S., Schneiderhan T., Twele A., Gahler M., Stein E. and Mehl H. RapidDamage Assessment and Situation Mapping: Learning from the 2010 Haiti Earthquake[J]. Photogrammetric Engineering & Remote Sensing, V77 (9), 2011: 923-931[13] Chini M. Earthquake damage mapping techniques using SAR and Optical remotesensing satellite data [J]. Advances in Geoscience and Remote Sensing, 2009: 269-279[14] Boccardo O., Giulio Tonolo F. Haiti earthquake damage assessment: review of theremote sensing role [J]. International Archives of the Photogrammetry, RemoteSensing and Spatial Information Sciences, 2012, V39-B4: 529-532




References

[15] Haklay M. How good is volunteered geographical information? A comparativestudy of OpenStreetMap and Ordnance Survey datasets [J]. Environment andPlanning B: Planning and Design, V37, 2010: 682-703[16] Helbich M., Amelunxen Ch., Neis P., Zipf A. Comparative Spatial Analysis ofPositional Accuracy of OpenStreetMap and Proprietary Geodata [C]. GI forum 2012:Geovisualization, Society and Learning, 2012: 24-33.[17] Jackson P.S., Mullen W., Agouris P., Crooks A., Croitoru A., Stefanidis A.Assessing Completeness and Spatial Error of Features in Volunteered GeographicInformation [J]. ISPRS International Journal of Geo-Information, V2, 2013: 507-530[18] Haklay M., Basiouka S., Antoniou V., Ather A. How Many Volunteer Does ItTake To Map An Area Well? The validity of Linus’ law to Volunteered GeographicInformation[J]. The Cartographic Journal, V47 (4), 2010: 315-332[19] Barron Ch., Neis P., Zipf A. A comprehensive Framework for IntrinsicOpenStreetMap Quality Analysis [J]. Transactions in GIS, V18 (6), 2014: 877-895.


Crowdsourced Damage Mapping for Disaster Emergency ...

Documents