Subword and spatiotemporal models for identifying actionable information in Haitian Kreyol

Subword and spatiotemporal models

for identifying actionable information

in Haitian Kreyol

Robert Munro

Stanford University

CoNLL 2011

Munro, Robert. "Subword and spatiotemporal models for identifying actionable

information in Haitian Kreyol." Proceedings of the Fifteenth Conference on

Computational Natural Language Learning. Association for Computational Linguistics,

2011.

http://www.robertmunro.com/research/munro11kreyol.pdf

January 12, 2010

Messages start streaming in

Messages start streaming in

80,000 messages

80,000 messages

80,000 messages

80,000 messages

80,000 messages

80,000 messages

Crowdsourced (Mission 4636)

Feedback

US Marines

◦ “Saving lives every day.”

FEMA:

◦ “The most comprehensive and up-to-date

map available to the humanitarian

community.”

World Food Program

◦ “We delivered food to an informal camp of

2500 people that you identified for us.”

Sudden-onset language

processing

Sudden-onset language

processing

Sudden-onset language

processing

Sudden-onset language

processing

Prioritization

Only 2% of messages were

‘actionable’

◦ An identifiable location

◦ Medical, S+R, water, clustered food

requests, security, unaccompanied

children.

How can we prioritize the

actionable items in the original

Haitian Kreyol?

Can we leverage the models for

more-sparse information sources?

Evaluation data

Mission 4636. 40,811 text-messages

sent to a free number, ‘4636’, in Haiti.

(predominantly in Haitian Kreyol, with

translations, UN-defined categories,

and geolocation)

Radio Station. 7,528 text-messages

sent to a Haitian radio station.

Twitter. 63,195 Haiti-related tweets.

Variation is the norm

mesi mèsi mèci meci merci

Kreyol French

Abbrv. Full Form Pattern Meaning

s’on se yon sVn is a

av`en av`eknou VvVn with us

relem rele mwen relem call me

wap ouap uVp you are

map mwen ap map I will be

zanmimzanmi mwen zanmim my friend

lavel lave li lavel to wash (it)

Evaluation data

Combining spatio-temporal

models

Streaming architecture

Build from initial items

timeModel

Streaming architecture

Predict (and evaluate) on incoming

items

◦ (penalty for training)time

Model

Streaming architecture

Repeat / retrain

timeModel

Streaming architecture

Repeat / retrain

timeModel

Streaming architecture

Repeat / retrain

timeModel

Streaming architecture

Repeat / retrain

timeModel

Streaming architecture

Repeat / retrain

timeModel

Features

G : Words and ngrams

W : Subword patterns

P : Source of the message

T : Time received

C : Categories (c0,...,47)

L : Location (longitude and latitude)

L : Has-location (a location is written

in the message)

Features

Subword models

◦ Full-forms and normalizations (Munro and

Manning 2010)

Abbrv. Full Form Pattern Meaning

s’on se yon sVn is a

av`en av`eknou VvVn with us

relem rele mwen relem call me

wap ouap uVp you are

map mwen ap map I will be

zanmimzanmi mwen zanmim my friend

lavel lave li lavel to wash (it)

Time / Space / source

Timestamp (in place of discounting)

Phone-number of sender

Spatial tile-membership:

Additional streaming models

Message contains an identifiable

location

timeModel

Additional streaming models

Message contains an identifiable

location

Prediction for 47 categories

timeModel

timeModel

timeModel

timeModel

…

Final streaming model

Prediction for ‘is actionable’

timeModel

Final streaming model

Prediction for ‘is actionable’

timeModel

timeModel

timeModel

timeModel

…

Combines features with predictions

from Category and Has-Location

models

Evaluation

100 training epochs

◦ Calculated on predictions over epochs 2-

100

Comparison of two-tier architecture

with Oracle ‘has-location’ and

‘Categories’

Identification of actionable messages

in Radio Station and Twitter messages

(full results in paper)

Feature-based improvements

Subwords and Source (G, T,W, P)

Temporal feature (G, T)

Words/Ngrams (G)

0.326

0.252

0.207

Combined models

All Features/Models

Spatial clusters (L)

Words/Ngrams (G)

0.855

0.756

0.207

Outperforming the oracle

Location (two-teir prediction)

Location (oracle)

Words/Ngrams (G)

0.310

0.274

0.207

Negative results from filtering

Oracle true-neg filtering

All Features/Models

Words/Ngrams (G)

0.428

0.855

0.207

Social media

Twitter

Radio station

All Features/Models

0.969

0.904

0.855

Conclusions (usability)

Subword and spatio-temporal models

can give a 10-fold increase in

prioritization

Adding multi-tiered streaming models

can give a 50-fold increase in

prioritization

Cross-domain adaptation is possible

for need(le)-in-haystack information

extraction from social media

Questions?

Appendix: abstract

Crisis-affected populations are often able to maintain digital

communications but in a sudden-onset crisis any aid organizations

will have the least free resources to process such communications.

Information that aid agencies can actually act on, ‘actionable’

information, will be sparse so there is great potential to

(semi)automatically identify actionable communications. However,

there are hurdles as the languages spoken will often be

underresourced, have orthographic variation, and the precise

definition of ‘actionable’ will be response-specific and evolving.

We present a novel system that addresses this, drawing on 40,000

emergency text messages sent in Haiti following the January 12,

2010 earthquake, predominantly in Haitian Kreyol. We show that

keyword/ngram-based models using streaming MaxEnt achieve up to

F=0.21 accuracy. Further, we find current state-of-the-art subword

models increase this substantially to F=0.33 accuracy, while

modeling the spatial, temporal, topic and source contexts of the

messages can increase this to a very accurate F=0.86 over direct

text messages and F=0.90-0.97 over social media, making it a viable

strategy for message prioritization.