Contrasting Public Opinion Dynamics and Emotional Response ...

Contrasting Public Opinion Dynamics andEmotional Response during Crisis

Svitlana Volkova1,3, Ilia Chetviorkin2, Dustin Arendt1, and Benjamin VanDurme3

1Pacific Northwest National Laboratory, Richland, WA USA2Computational Mathematics and Cybernetics, Lomonosov Moscow State University,

Moscow Russia3Center for Language and Speech Processing, Johns Hopkins University, Human

Language Technology Center of Excellence, Baltimore MD [email protected],[email protected],[email protected],

[email protected]

Abstract. We propose an approach for contrasting spatiotemporal dy-namics of public opinions expressed toward targeted entities, also knownas stance detection task, in Russia and Ukraine during crisis. Our anal-ysis relies on a novel corpus constructed from posts on the VKontaktesocial network, centered on local public opinion of the ongoing Russian-Ukrainian crisis, along with newly annotated resources for predicting ex-pressions of fine-grained emotions including joy, sadness, disgust, anger,surprise and fear. Akin to prior work on sentiment analysis we align tra-ditional public opinion polls with aggregated automatic predictions ofsentiments for contrastive geo-locations. We report interesting observa-tions on emotional response and stance variations across geo-locations.Some of our findings contradict stereotypical misconceptions imposed bymedia, for example, we found posts from Ukraine that do not supportEuromaidan but support Putin, and posts from Russia that are againstPutin but in favor USA. Furthermore, we are the first to demonstratecontrastive stance variations over time across geo-locations using story-line visualization1 technique.

Keywords: social media analytics, spatiotemporal analysis opinion ofopinion dynamics, emotion prediction, storyline visualization

1 Introduction

Social media data has been extensively used for a variety of monitoring andpredictive tasks, both online activity [15] and real-world events, such as real-timelarge-scale health analytics [8,19,9], multi-source disease forecasting [38,40], stockmarket prediction [5], voting outcome forecasting [30,4,20], political movementsand protest activity detection [3], and real-time mood changes [21]. Moreover,signals extracted from real-time social media data have been successfully used1 Storyline visualization is available at http://www.cs.jhu.edu/∼svitlana/

2 S. Volkova, I. Chetviorkin, D. Arendt, and B. Van Durme

for situational awareness e.g., to analyze online sentiments during disasters andhazard events [47,27,43,2].

In this work we study public opinion and emotions expressed through socialmedia during the Russian-Ukrainian crisis. We formulate a problem as a stancedetection task – automatically determining from posts whether the author is infavor of the given target, against the given target, or whether neither inferenceis likely, for example the post in favor Putin is:– Sergey, среди моих друзей, и друзей моих друзей таких нет, зато все как

один поддерживают Путина. Даже те, кто еще два года назад поддер-живали всяких навальных, изменили свое мнение. Возвращение Крыма,это конечно мощный консолидирующий фактор. My friends and friendsof my friends support Putin. Even those who two years ago supported Naval-niy changed their minds. The return of the Crimea is certainly a powerfulconsolidating factor.Our analysis focuses on a period of nine months – from September 2014

to March 2015. We start by collecting public data relevant to the crisis fromthe VKontakte (VK) social network, which is the most popular social networkin Russia and Ukraine, by filtering relevant posts using topical keywords (ortargets of stance). We then apply models for stance detection and develop newmodels for emotion classification in Russian and Ukrainian. Finally, we analyzehow stance regarding targeted topics (i) evolves over time within each countryby analyzing positive opinions (author is in favor) and negative opinions (authoris against), and (ii) differs between contrasting populations (Russia vs. Ukraine)by measuring correlations between opinion proportions toward targeted topics.The main contributions of this work include:– Performing large-scale contrastive targeted opinion (stance) and emotion

analysis on thousands of messages posted in Russia and Ukraine duringcrisis on a new corpus from the VK social network.2

– Building a new model for fine-grained emotion prediction for low-resourcelanguages e.g., Russian and Ukrainian.3

– Measuring spatiotemporal variations in targeted opinions between contrastivegeo-locations e.g., Russia and Ukraine.

– Visualizing contrastive stance dynamics to qualitatively evaluate differencesacross geo-locations over time using storylines.

– Contrasting opinions expressed through social media with traditional polls.Similar to recent studies on large-scale public opinion polling [30,39], the

results of this work demonstrate that signals from public social media can bea faster and less expensive alternative to traditional polls [6]. Moreover, unlikethe existing approaches, our study is the first to perform emotion analysis (inaddition to traditional opinion mining) for languages other than English [12,50],and quantitatively estimate differences in targeted opinions (or stance) betweencontrastive populations over time using storyline visualization.

2 Anonymized VK corpus is available upon request at [email protected] Pre-trained models for emotion prediction and data annotated with 6 Ekman’s emo-tions in Russian and Ukrainian can be found at http://www.cs.jhu.edu/∼svitlana/

Contrasting Public Opinion Dynamics during Crisis 3

Table 1: Sample stance targets used to filter VK posts relevant to the crisis inRussian (translated to English). We focus on named entities (Person, Organiza-tion, Location) and other event triggers.

Post-revolution periodEnglish Locations: Kiev, Ukraine, Russia, USA, America, Moscow, Crimea

People: Yanukovych, PutinOrganizations: European Union (EU), North Atlantic Treaty Org (NATO)Events: Maidan (Independence square), Euromaidan, berkut, revolution,titushki, banderovtsy, terrorism, war, separatists

Russian Locations: Киев, Украина, Россия, США, Америка, Москва, КрымPeople: Янукович, ПутинOrganizations: Европейский союз, Евросоюз, НАТОEvents: майдан, евромайдан, революция, титушки, бандеровци, беркут,терроризм, война, сепаратисты

Conflict periodEnglish Locations: Donbass, Donetsk, Debaltseve, Ilovaisk, Mariupol, Lugansk Peo-

ple Republic (LPR), Donetsk People Republic (DPR)Events: battle of Debaltseve, battle of Ilovaisk, rebels, junta, cyborgs, hu-manitarian aid, ukrops, pravy sektor, pravoseki, cease-fire, pravosek, cease-fire, negotiations, checkpoint, fire, punishers, national army, krimnash,ukrofashysty, quiet period, h**lo

Russian Locations: Донбас, Донецк, Дебальцево, Иловайск, Мариуполь, Луган-ская Народная Респуб. (ЛНР), Донецкая Народная Респуб. (ДНР)Events: Дебальцевский котел, Иловайский котел, ополченцы, хунта,киборги, гуманитарная помощь, укропы, правый сектор, прекращениеогня, правосек, перемирие, переговоры, блокпост, обстрел, каратели,нацгвардия, крымнаш, укрофашисты, режим тишины, х*йло

2 VKontakte Data

VK is the most popular social network in Russia and the former Soviet Unionarea. As of 2013, it was used by 106 million users worldwide with 9.5 millionusers in Moscow, 5.2 million in Saint-Petersburg and 2.75 million in Kyiv.

Sampling Real Users Our original VK dataset of topically relevant posts –messages that include one or more keywords k ∈ K defined in Table 1 consistsof 3.3 million posts from 1 million users. The post per user ratio (PPU) is3.3 meaning that in the original VK data many users produced only one post.However, taking into account a well-known issue of sampling biases in socialmedia [44], we sub-sampled the original collection to eliminate bots, trolls,4,5,6

4 Social Network Analysis Reveals Full Scale of Kremlin’s Twitter Bot:https://globalvoices.org/2015/04/02/analyzing-kremlin-twitter-bots/

5 Inside Putin’s Campaign Of Social Media Trolling And Faked Ukrainian Crimes:http://www.forbes.com/sites/paulroderickgregory/2014/05/11/inside-putins-campaign-of-social-media-trolling-and-faked-ukrainian-crimes/#238cfd72629d

6 Ukraine conflict: Inside Russia’s ’Kremlin troll army’:http://www.bbc.com/news/world-europe-31962644


spam, news accounts as well as hyper-active and hypo-active users as describedbelow. We removed users without locations listed in their profiles, users withmore than 500 or less than five friends, and users with more than 300 posts(hyper-active) and less than two posts (hypo-active) over nine months.

As a result, we collected “real”, moderately active VK users with marked geo-locations – 49,208 unique users with 597,247 topically-relevant posts (onaverage 12 posts per user).

Geo-location Distribution Our dataset contains 772 unique locations inUkraine (143 locations have more than 100 posts) and 1,378 unique locations inRussia (217 locations have more than 100 posts). In Figure 1 we report messagedistribution for the top 17 locations that include at least 2,500 crisis-relevantposts per location that are represented by 62% of all posts in Russian portionof our dataset, and 77% of all posts in Ukrainian portion.

Fig. 1: Top geo-locations in Russia and Ukraine (red circle represent the numberof VK posts per location).

Gender and Age Distribution This is the first study that analyzes publicposts from the VK social network. For large-scale passive polling, it is importantto be aware of how gender and age distributions in the sample are different fromthe demographics in the population. It has been reported that VK is equallypopular among men and women.7 However, our dataset contains 64% postsfrom male users, 25% posts from female users in Russia (11% have not reportedgender), and 59% posts from male users and 29% posts from female users inUkraine (12% have not reported gender). These statistics may suggest that crisistopics are more discussed among male rather than female users in VK.

Table 2: Age distribution in our VK data sample.Geo-location 13 – 17 18 – 24 25 – 34 35 – 44 45 – 54 55 – 64 65 + N/AUkraine 1% 8% 16% 7% 4% 3% 1% 40%Russia 1% 5% 13% 9% 7% 3% 2% 40%

Table 2 reports age distribution in our VK data sample from two geo-locations – Russia and Ukraine. We observe that the sample is skewed towardyounger population (18 – 44 y.o.) in both geo-locations. Unlike gender assign-ments, 40% of posts from our sample do not have age labels available.7 VK demographics: http://www.slideshare.net/andrewik1/v-kontakte-demographics


Topic Dynamics Figure 2 shows the number of posts with topical keywordsproduced between Sept 2014 and Mar 2015 in Russia and Ukraine. We observethat Putin is the least popular keyword relative to the five most discussed (trend-ing) topics during that period – Donbas, Crimea, DNR, LNR and war (sortedby popularity). We found several spikes in keyword popularity that happenedto overlap with major events relevant to the crisis – Minsk I8 and Minsk II.9 Asexpected, we captured a significant decrease in keyword popularity around thetime of the New Year holiday across all keywords in both countries.

Fig. 2: Top 5 topic dynamics over time in both countries.

3 Approach

This section describes the approach we used for opinion classification and emo-tion detection, and outlines the limitations of our analysis.

3.1 Targeted Opinion Prediction

For stance prediction we used the state-of-the-art opinion classification systemfor Russian – POLYARNIK [18] that relies on morphological and syntactic rules,affect lexicons and supervised models [26]. Classification performance was mea-sured on an external social media dataset from Twitter and achieved the F1score of 0.62 [7,25].

We applied opinion classifiers to assign positive (in favor), negative (against)or neutral stance to every message in our dataset. Example posts in Russian withthe assigned opinion scores are shown in Figure 3. To study geo-temporal vari-ations in public opinions between contrasting sub-populations we used severalmetrics as defined below.

Within within each country c ∈ C, for each keyword k ∈ K and time periodt ∈ T we calculate positive opinion score:

s+(k, c, t) =positive posts toward k at time t in country c

total posts toward k at time t in country c.

8 Minsk I: https://en.wikipedia.org/wiki/Minsk_Protocol9 Minsk II: https://en.wikipedia.org/wiki/Minsk_II


Fig. 3: Opinion and emotion classification approach: example VK posts withpredicted targeted opinions (stance) and fine-grained emotions.

For each keyword k ∈ K and time t ∈ T we calculate positive score ratio:

∆s+(k,UA

RU, t) =

s+(k, c = UA, t)

s+(k, c = RU, t).

The above metrics allow us to capture opinion drift within each countryover time [30,29], as well as to contrast stance differences between two countries.More precisely, positive opinion score will provide the insights on how positiveUkraine is toward Crimea every week if c = Ukraine, t = week ∈ {09/2014 −03/2015}, and k = Crimea. Similarly, positive score ratio will demonstratehow much more or less positive Ukraine is toward Crimea compared to Russia.

3.2 Emotion Classification

We constructed our emotion dataset from an independent sample of crisis-relateddiscourse generated on Twitter. For that we bootstrapped noisy hashtag anno-tations for six basic emotions put forward by Ekman10 [11] as has been donesuccessfully for English [28,48]. Twitter users sometimes use the hashtag #ра-дость (#joy) to signal when they are happy, #страх (#fear when they arescared (Figure 3). As discussed in [10] and [13], such annotations are extremelysparse. Therefore, we extended our emotion set with emotion synonyms col-lected from WordNet-Affect [46], Google Synonyms, and Roget’s Thesaurus andtranslated them to Russian and Ukrainian.

A native speaker of Russian and Ukrainian then manually re-validated tweetsannotated with emotion hashtags. In total we collected 5,717 tweets annotatedwith anger (11%), joy (25%), fear (20%), sadness (37%), disgust (4%), and sur-prise (3%). In addition to emotional tweets, we sampled more tweets from the 1%Russian Twitter feed and manually validated 3,947 tweets that do not express10 We prefer Ekman’s emotion classification over others e.g., Plutchik’s, because we

would like to compare the performance of our predictive models to other systems.


Table 3: Emotion classification quality estimated using weighted F1 comparedto other systems [28,48].

Emotion Mohammad [EN] Volkova [EN] This work [RU]anger 1,555 0.28 4, 963 0.80 606 0.29disgust 761 0.19 12,948 0.92 242 0.22fear 2,816 0.51 9,097 0.77 1,120 0.61joy 8,240 0.62 15,559 0.79 1,444 0.66sadness 3,830 0.39 4,232 0.62 2,138 0.61surprise 3849 0.45 8,244 0.64 167 0.56All 21,051 0.49 52,925 0.78 5,717 0.58

any emotions. We applied our emotion dataset bootstrapped from Twitter toannotate VK posts similar to other predictive analytics transferred across socialmedia e.g., from Twitter to Facebook [41].

We took a two-step approach to predict emotions in crisis-relevant posts. Wefirst classify tweets as emotional and non-emotional, and then predict one of sixEkman’s emotions in a subset of emotional posts only.

We trained our emotion classifier using an implementation of a log-linearmodel in scikit-learn [33]. We relied on binary word unigram features extractedfrom stemmed11 posts annotated with six basic emotions as described above.In addition to lexical features we extracted a set of stylistic features includingemoticons, elongated words, capitalization, and repeated punctuation and takeinto account negation [32]. We demonstrate model prediction quality using 10-fold cross validation on our emotion dataset and compare it to other existingdatasets for English in Table 3.

3.3 Limitations

To the best of our knowledge, this is the first study that performs large-scalecontrastive opinion analysis relevant to the Russian-Ukrainian crisis on the VKsocial network. However, our approach has several limitations. First, our opinionand emotion prediction models do not yield 100% accuracy. Thus, some postsmight be mislabeled. However, doing the analysis on such a large scale allowsus to overcome noise in affect labels [30]. Second, we do not claim that ourfindings are representative of the whole population in Russia or Ukraine. Wedraw conclusions from sampled data from only one social network – VK, whichby nature is more representative of younger populations in both countries.

4 Results

This section discusses spatiotemporal variations in topic popularity, target-specificopinion correlations and opinion drift over time across two countries, fine-grainedemotion analysis and storyline visualization applied to our data.11 Morphological analyzer for Russian: https://pypi.python.org/pypi/pymorphy2


4.1 Contrasting Topic Popularity across Geo-Locations

To estimate popularity of crisis-relevant topics we measure the frequency of poststhat contain one or more topics including hashtags e.g., #putin, #euromaidan.In Figure 4 we show the most popular topics in our VK dataset for Russia andUkraine. We observe that Donbas was discussed more in Ukraine, but war inRussia over the same time period. There were significantly more posts aboutPutin and USA in Russia compared to Ukraine. On the other hand, North At-lantic Treaty Organization (NATO) and European Union (EU), Donetsk People’sRepublic (DNR),12 and Luhansk People’s Republic (LNR)13 were discussed morein Ukraine compared to Russia.

Donbas 36%War 13%

DNR/LNR 18%

Putin 9%

USA 6%NATO 7% EU 1%

Crimea 21%

Our Crimea 1%Maidan 6%

(a) Ukraine

Donbas 29%

War 16%

DNR/LNR 16%

Putin 11%

USA 12%NATO 4%

Crimea 26%

Our Crimea 1%Maidan 6%

(b) Russia

Fig. 4: Popular crisis-relevant topics in Russia and Ukraine.

4.2 Measuring Targeted Opinion Correlations

Overall, we found that 99% of posts in our dataset were assigned either positiveor negative polarity (in favor or against). Therefore, we correlate positive polarityscores s+(k, c, t) expressed toward targeted entities k for every week t for eachcountry c (assuming that s+(k, c, t) + s−(k, c, t) = 1).

We calculate Pearson correlations between targeted opinion scores s+(k, c, t)in two countries. We present the results in Figure 5, where blue represents pos-itive correlations e.g., opinions are changing in one direction, either positive ornegative, and red stands for negative correlations e.g., opinions are changing ina different direction. Below we outline our key findings on opinion correlationsbetween Russia and Ukraine. We found:– Positive sentiment toward EU↑ in Ukraine contrasts with negative toward

NATO↓, DNR↓, and Donbas↓ in Russia.– Positive sentiment toward NATO↑ in Ukraine contrasts with negative toward

USA↓ in Russia.– Positive sentiment toward OurCrimea↑ in Ukraine contrasts with negative

sentiment toward NATO↓, DNR↓, and Donbas↓ in Russia.

12 Donetsk people’s republic: https://en.wikipedia.org/wiki/Luhansk_People’s_Republic13 Luhansk People’s Republic: https://en.wikipedia.org/wiki/Luhansk_People’s_Republic


Maidan

Crimea

OurC

EU

NATO

USA

Putin

DNR

War

Donbas

Keyword Correlations Between Ukraine and Russia

Fig. 5: Correlogram for targeted opinions (stance) between two countries (blue– positive correlations, red – negative).

– Positive sentiment toward USA↑ in Ukraine contrasts with negative towardwar↓, DNR↓, and Donbas↓ in Russia.In addition, we found several major differences in stance expressed by two

populations toward the same targeted entity:– EU have strong negative correlations between two countries (ρ = −0.4).– USA and maidan have low positive correlations (ρ = 0.05 and ρ = 0.34).– OurCrimea and Putin have positive correlations (ρ = 0.59 and ρ = 0.65).– LNR, DNR, NATO, Donbas, Crimea, and war have strong positive correla-

tions (as high as ρ = 0.94 for the war).Below we outline some examples of opinions that contradict stereotypical

misconceptions imposed by media discovered in our data.– Negative opinions toward maidan in Ukraine:

Яценюк говорил неправду слишком долго. Когда в феврале на Майданелилась кровь, он с Кличко и Тягнибоком жали руку «енакиевскому ли-деру» Yatsenyuk told a lie for too long. He was with Klitschko and Tyagnibokand shook hands with “the leader of Yenakiyevo” while there has been blood-shed at the Independence square (maidan).

– Negative opinions toward EU in Ukraine (in contrast, media reports Ukraini-ans are in favor joining EU [16,36,49]):Евросоюз кинул западенцев, а Россия - жителей Донбасса. EU betrayedresidents of Western Ukraine, and Russia – residents of Donbas.

– Positive opinions toward USA in Russia:Только санкции заставят русских оставлять деньги в России и не смот-реть на запад и Aмерику как на единственный источник благ и цивили-заций. Only sanctions will force Russians to leave money in Russia, and donot look to the West and America as the sole source of wealth and civilization.


– Negative opinions toward Putin in Russia (in contrast, media reports themajority of Russian population are in favor Putin [22]):Есть международные договоры, которые нарушил Путин, устроив ан-нексию Крыма. There are international treaties that has been violated Putinduring annexation of Crimea.

4.3 Estimating Contrastive Opinion Drift Over Time

To measure opinion dynamics we calculate positive score ratios ∆s+(k, UARU , t)

using location-specific targeted opinions expressed in Ukraine vs. Russia.We present our results in Figure 6 where blue bars show when targeted

opinions are more positive in Ukraine compared to Russia, and red bars reflectwhen targeted opinions are more positive in Russia compared to Ukraine. Weplot the number of weekly posts that contain targeted entity for both countries toreflect the amount of evidence contributing to weekly stance estimates e.g., more

(a) Maidan (b) Crimea

(c) EU (d) NATO

(e) Putin (f) USA

Fig. 6: Spatiotemporal variations in contrastive opinion dynamics. Blue barsshow when targeted opinions are more positive in Ukraine vs. Russia. Red barsshow when targeted opinions are more positive in Russia compared to Ukraine.Dashed lines represent the total number of weekly tweets with targeted entitiesin Russia (red) and Ukraine (blue).


posts – more confident estimates. We noticed several topics including Crimea,EU, Putin and USA are more popular in Russia whereas maidan, NATO aremore discussed in Ukraine. We observed more positive opinions were expressedtoward maidan in Ukraine than in Russia, and more positive opinions expressedtoward Crimea, Putin in Russia than in Ukraine.

4.4 Visualizing Contrastive Opinion Dynamics

We use storyline visualization to understand opinion dynamics at a coarse levelof detail across a hand curated set of targeted entities. Storyline visualizationshows how entities interact over time [17,24,31,37,42]. This technique encodestime on the horizontal axis, and interactions on the vertical axis. When twoentities are interacting, their storylines are drawn close together, otherwise theyare drawn apart. Entities having the same state at the same time are assumedto be interacting with each other.

To visualize opinion dynamics, we process the original opinion time series s+(k, c, t) into a format usable by the storyline visualization. For that we estimatethe mean number of posts with positive and negative stance toward targetedentity per day, and normalize them by the total number of posts per day thatmention targeted entity. This is done for each targeted entity, geo-location andday, by subtracting the mean and dividing by the standard deviation of thattargeted entity across all days. This puts the positive and negative stance foreach targeted entity on a consistent scale so that meaningful comparisons canbe made across targeted entities over time.

We present some example storyline visualizations around important eventsrelevant to the crisis – G20 meeting in Nov 2014 and Minsk II agreement inFeb 2015 in Figure 7 and 8, respectively. We consider whether targeted opinionstoward a particular topic in specific location and a day was elevated (i.e., morethan one standard deviation larger than the mean) for both positive and negativeopinions. This resulted in four distinct clusters corresponding to the combina-tions of {normal, elevated} × {positive, negative} per topic, location and day.As shown in Figures 7 and 8 orange cluster contains stance within one standarddeviation above the mean. Green cluster represents opinions that are more infavor (two or more st. dev. above). Red cluster represents opinions that are morenegative. Blue cluster represents opinions that are both more positive and morenegative relative to the mean. Each topic is represented with two storylines, oneeach for the populations from Russia and Ukraine, so that differences over timebetween these populations can be visually identified and understood.

More specifically, Figure 7 demonstrates contrastive opinion drift towardUSA in Russia and Ukraine in November 2015. Storyline visualization allowsus to see how opinions toward USA in Russia move to a positive cluster (green)from the mean cluster (orange) on 11/7 and 11/8 (shown on the left), and opin-ions toward USA in Ukraine move to a positive cluster on 11/5. To put thesefindings in context, we found that trending news stories on 11/5 included “A


Fig. 7: Storyline visualization of opinion dynamics toward USA expressed in Rus-sia (Ru) and Ukraine (Ua) during G20 summit (Nov 1 – 15, 2014).

Fig. 8: Storyline visualization of opinion dynamics toward Donbas expressed dur-ing G20 summit between Nov 3 – 19, 2014 (left), and toward war expressedduring Minsk II agreement between Feb 1 – 19, 2015 (right).

bomb could be to blame for the Russian jet crash”,14 and on 11/7 included“Putin suspends all flights from Russia to Egypt in wake of crash”.15

Figure 8 (left) highlights contrastive stance dynamics toward Donbas (theEastern part of Ukraine where the conflict escalated) around G20 meeting inNovember 2014. We observe that on 11/16 opinions expressed in Russia towardDonbasmoved to a negative cluster (red), on 11/17 – to a positive cluster (green),and on 11/18 – back to the average opinion cluster (orange). News reported on11/17 that include targeted entities of interest e.g., Putin, Donbas, EU, DNR,LNR and war are: “E.U. to Toughen Sanctions on Ukrainian Separatists, butNot Russia”16 and Poroshenko’s post on Twitter “We are prepared for a scenarioof total war. We don’t want war, we want peace, we are fighting for EU values.Russia doesn’t respect any agreement.”17

Figure 8 (right) demonstrates contrastive stance dynamics toward war aroundMinsk II meeting on February 11, 2015. We observe that opinions toward war

14 http://www.cbsnews.com/videos/cbs-news-trending-stories-for-november-5-2015/15 http://theweek.com/10things/580982/10-things-need-know-today-november-7-201516 http://www.nytimes.com/2014/11/18/world/europe/eu-to-toughen-sanctions-on-

ukraine-separatists-but-not-russia.html?_r=117 http://www.cnbc.com/2014/11/17/german-economy-minister-rejects-tougher-

sanctions-on-russia.html#


in Ukraine move to a positive (green) cluster on 02/14 and 02/16 after theagreement to cease fire. In contrast opinions toward war in Russia became morenegative – move to a negative (red) cluster on 02/17.

4.5 Emotion Analysis

After applying our two stage emotion classifier we found that the resulting emo-tion predictions are extremely sparse. Our model predicts only 4% posts to beemotional. On one hand, it further confirms that crisis-related discourse is factualand opinionated rather than emotional. On the other hand, it does not allow usto perform contrastive emotion analysis over time. Therefore, we report emotiondifferences between two countries on an aggregate level rather than by analyzingemotion score ratios over time. Figure 9 demonstrates that users express moreemotions e.g., sadness, surprise, and anger toward Donbas across countries; sad-ness toward USA, fear toward Crimea and Donbas in Russia; fear toward Putinand Donbas, disgust toward maidan in Ukraine.

ru.disgust

ru.surprise

ua.surprise

ua.fear

ru.fear

ua.disgust

ua.anger

ru.anger

ua.sadness

ru.sadness

ua.joy

ru.joy

USAPutinNATOMaidanLNROurCrimeaCrimeaEUDonbas

War

Heatmap

Value-2 0 2

015

Color Key

Count

Fig. 9: Fine-grained targeted emotion analysis for Russia (ru) and Ukraine (ua).Red represents high post counts, blue represents low counts scaled over columns.

5 Discussion

Our results demonstrate that one can infer spatiotemporal variations in opiniondynamics from social media. But do these results reflect observations gatheredby traditional polling techniques? Unfortunately, there is no real time pollingdata on concepts related to crisis similar to the Gallup’s organization “EconomicConfidence” index18 used in [30] or CDC data used in [39]. Despite that, in orderto evaluate our findings we rely on recent polls from several internationally rec-ognized organizations. Below we analyze how our findings on the most discussedand controversial topics across two countries align with public poll data.18 http://www.gallup.com/poll/122840/gallup-daily-economic-indexes.aspx


EU, NATO, and USA An IFES survey [16] finds that half of Ukrainiansbelieve that their country would be better off with closer relations with Europethan with Russia, which is in line with our findings on positive sentiments inUkraine toward EU, NATO and negative toward Putin as shown in Figure 6.Similar polls done by the Razumkov Centre [36] and the Democratic InitiativesFoundation [49] conclude that 43.2% of Ukrainians believe that their countrywill gain if it joins the EU and that 64% of Ukrainians would vote for NATOaccession in a hypothetical referendum. Regarding negative sentiments towardthe USA expressed in Russia, a recent poll done by Pew Research Center reportsthat only 15% of Russians have a favorable opinion of America [34].

Crimea and Putin Posts about Crimea authored in Ukraine are less posi-tive compared to those authored in Russia which is again in line with publicpolls. Russian media reports 55% of Russian citizens strongly support and 33%support Putin’s actions on Crimea [23]. Tweets mentioning Putin in Russia aresignificantly more positive than those in Ukraine. Again, Russian media reportsthat based on recent polls 83% of Russian citizens trust president Putin [22]. An-other independent poll done by Pew Research Center finds that Russians praisePutin despite their country’s economic troubles.14 On the other hand, Gallup’sinterviews done in 2014 report a drastic change in approval of Russian leadershipin Ukraine – the approval drops by as high as 45%, and is now as low as 2% [35].

6 Related Work

With the dramatic rise of text-based social media more researchers started fo-cusing their work on tracking spatiotemporal opinion dynamics toward politi-cians [45,51], events [14,1], or controversial issues e.g., vaccinations [39]. Lesswork focused on quantitatively evaluating or predicting opinion changes overtime e.g., [29] built models to predict directional sentiment polarity change. Ourwork is the most similar similar to [30] and [39] that estimate opinion dynamicsand correlate it with real world data e.g., Gallup’s polls or CDC reports. Unlikethese works we not only align our findings with traditional opinion polls butalso correlate opinion changes toward controversial topics between contrastivepopulations over time and develop a novel model for emotion detection for lowresource languages.

7 Conclusions

We proposed a transparent and deterministic approach to perform large-scalecontrastive opinion analysis on social media during crisis. We developed an emo-tion classification model for Russian and Ukrainian and qualitatively measuredspatiotemporal differences in opinions and emotions between contrastive popu-lations toward controversial topics. We developed storyline visualizations thatallow to better capture contrastive opinion drift over time. We supported ourfindings with public opinions obtained using traditional polls.


References

1. André Luiz Firmino Alves, Cláudio de Souza Baptista, Anderson Almeida Firmino,Maxwell Guimarães de Oliveira, and Anselmo Cardoso de Paiva. A spatial andtemporal sentiment analysis approach applied to Twitter microtexts. Journal ofInformation and Data Management, 6(2):118, 2016.

2. Zahra Ashktorab, Christopher Brown, Manojit Nandi, and Aron Culotta. Tweedr:Mining Twitter to inform disaster response. Proceedings of the International Con-ference on Information Systems for Crisis Response and Management (SCRAM),2014.

3. Marco T Bastos and Dan Mercea. Serial activists: Political Twitter beyond influ-entials and the twittertariat. New Media & Society, 2015.

4. Adam Bermingham and Alan F Smeaton. On using twitter to monitor politicalsentiment and predict election results. Joint Conference for Natural LanguageProcessing (IJCNLP), 2011.

5. Johan Bollen, Huina Mao, and Xiaojun Zeng. Twitter mood predicts the stockmarket. Journal of Computational Science, 2(1):1–8, 2011.

6. Linchiat Chang and Jon A Krosnick. National surveys via rdd telephone interview-ing versus the internet comparing sample representativeness and response quality.Public Opinion Quarterly, 73(4):641–678, 2009.

7. Ilia Chetviorkin and Natalia Loukachevitch. Evaluating sentiment analysis systemsin russian. Proceedings of ACL, 2013.

8. Cynthia Chew and Gunther Eysenbach. Pandemics in the age of Twitter: contentanalysis of tweets during the 2009 h1n1 outbreak. PloS one, 5(11):e14118, 2010.

9. Courtney D Corley, Diane J Cook, Armin R Mikler, and Karan P Singh. Text andstructural data mining of influenza mentions in web and social media. InternationalJournal of Environmental Research and Public Health, 7(2):596–615, 2010.

10. Munmun De Choudhury, Michael Gamon, and Scott Counts. Happy, nervous orsurprised? classification of human affective states in social media. In Proceedingsof ICWSM, 2012.

11. Paul Ekman. An argument for basic emotions. Cognition & Emotion, 6(3-4):169–200, May 1992.

12. Rui Fan, Jichang Zhao, Yan Chen, and Ke Xu. Anger is more influential than joy:sentiment correlation in Weibo. arXiv preprint arXiv:1309.2402, 2013.

13. Roberto González-Ibáñez, Smaranda Muresan, and Nina Wacholder. Identifyingsarcasm in Twitter: A closer look. In Proceedings of ACL, pages 581–586, 2011.

14. Shen-Shyang Ho, Mike Lieberman, Pu Wang, and Hanan Samet. Mining futurespatiotemporal events and their sentiment from online news articles for location-aware recommendation system. In Proceedings of the First ACM SIGSPATIALInternational Workshop on Mobile Geographic Information Systems, pages 25–32.ACM, 2012.

15. Nathan O Hodas and Kristina Lerman. The simple rules of social contagion.Scientific reports, 4, 2014.

16. IFES. September 2015 Public Opinion Survey in Ukraine, 2015.http://www.ifes.org/surveys/september-2015-public-opinion-survey-ukraine.

17. Nam Wook Kim, Stuart K Card, and Jeffrey Heer. Tracing genealogical datawith timenets. In Proceedings of the International Conference on Advanced VisualInterfaces, pages 241–248. ACM, 2010.

18. Ekaterina Kuznetsova, Natalia Loukachevitch, and Ilia Chetviorkin. Testing rulesfor a sentiment analysis system. In Proceedings of International Conference Dialog,pages 71–80, 2013.


19. Vasileios Lampos, Tijl De Bie, and Nello Cristianini. Flu detector-tracking epi-demics on twitter. In Machine Learning and Knowledge Discovery in Databases,pages 599–602. Springer, 2010.

20. Vasileios Lampos, Daniel Preotiuc-Pietro, and Trevor Cohn. A user-centric modelof voting intention from social media. In Proceedings of ACL, pages 993–1003,2013.

21. Thomas Lansdall-Welfare, Vasileios Lampos, and Nello Cristianini. Effects of therecession on public mood in the uk. In Proceedings of the 21st international con-ference companion on World Wide Web, pages 1221–1226. ACM, 2012.

22. Levada. Ukraine, Crimea, Sanctions, 2015. http://www.levada.ru/2015/04/06/ukraina-krym-sanktsii/.

23. Center Levada. Vladimir Putin: Trust, Evaluations, Attitudes, 2015. http://www.levada.ru/2015/03/27/vladimir-putin-doverie-otsenki-otnoshenie/.

24. Shixia Liu, Yingcai Wu, Enxun Wei, Mengchen Liu, and Yang Liu. Storyflow:Tracking the evolution of stories. IEEE Transactions on Visualization and Com-puter Graphics, 19(12):2436–2445, 2013.

25. Natalia Loukachevitch and Ilia Chetviorkin. Open evaluation of sentiment-analysissystems based on the material of the russian language. Scientific and TechnicalInformation Processing, 41(6):370–376, 2014.

26. Natalia Loukachevitch and Ilia Chetviorkin. Refinement of russian sentiment lex-icons using ruthes thesaurus. Proceedings of the 16th All-Russian Conference onDigital Libraries: Advanced Methods and Technologies, Digital Collections, 2014.

27. Benjamin Mandel, Aron Culotta, John Boulahanis, Danielle Stark, Bonnie Lewis,and Jeremy Rodrigue. A demographic analysis of online sentiment during hurricaneIrene. In Proceedings of the Second Workshop on Language in Social Media, pages27–36. Association for Computational Linguistics, 2012.

28. Saif M. Mohammad and Svetlana Kiritchenko. Using hashtags to capture fineemotion categories from tweets. Computational Intelligence, 2014.

29. Le T Nguyen, Pang Wu, William Chan, Wei Peng, and Ying Zhang. Predictingcollective sentiment dynamics from time-series social media. In Proceedings of the1st International Workshop on Issues of Sentiment Discovery and Opinion Mining.ACM, 2012.

30. Brendan O’Connor, Ramnath Balasubramanyan, Bryan R. Routledge, and Noah A.Smith. From tweets to polls: Linking text sentiment to public opinion time series.In Proceedings of ICWSM, pages 122–129, 2010.

31. Michael Ogawa and Kwan-Liu Ma. Software evolution storylines. In Proceedingsof the 5th International Symposium on Software Visualization, pages 35–42, 2010.

32. Bo Pang, Lillian Lee, and Shivakumar Vaithyanathan. Thumbs up?: sentimentclassification using machine learning techniques. In Proceedings of EMNLP, pages79–86, 2002.

33. Fabian Pedregosa, Gael Varoquaux, Alexandre Gramfort, Vincent Michel,Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, RonWeiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau,Matthieu Brucher, Matthieu Perrot, and Edouard Duchesnay. Scikit-learn: Ma-chine Learning in Python. Journal of Machine Learning Research, 12:2825–2830,2011.

34. Jacob Poushter. Key findings from our poll on the Russia-Ukraineconflict, 2015. http://www.pewresearch.org/fact-tank/2015/06/10/key-findings-from-our-poll-on-the-russia-ukraine-conflict/.


35. Julie Ray and Neli Esipova. Ukrainian Approval of Russia’s Leader-ship Dives Almost 90%, 2015. http://www.gallup.com/poll/180110/ukrainian-approval-russia-leadership-dives-almost.aspx?g_source=Ukraine&g_medium=search&g_campaign=tiles.

36. Centre Razumkov. Will Ukraine mostly gain or lose if it joins the EU?, 2015.http://www.uceps.org/eng/poll.php?poll_id=675.

37. Khairi Reda, Chayant Tantipathananandh, Andrew Johnson, Jason Leigh, andTanya Berger-Wolf. Visualizing the evolution of community structures in dynamicsocial networks. In Computer Graphics Forum, volume 30, pages 1061–1070. WileyOnline Library, 2011.

38. Theodoros Rekatsinas, Saurav Ghosh, Sumiko R Mekaru, Elaine O Nsoesie, John SBrownstein, Lise Getoor, and Naren Ramakrishnan. Sourceseer: Forecasting raredisease outbreaks using multiple data sources. Timeline, 7:8, 2007.

39. Marcel Salathé and Shashank Khandelwal. Assessing vaccination sentiments withonline social media: implications for infectious disease dynamics and control. PLoSComput Biol, 7(10):e1002199, 2011.

40. Mauricio Santillana, André T Nguyen, Mark Dredze, Michael J Paul, Elaine ONsoesie, and John S Brownstein. Combining search, social media, and tra-ditional data sources to improve influenza surveillance. PLoS Comput Biol,11(10):e1004513, 2015.

41. Maarten Sap, Gregory Park, Johannes Eichstaedt, Margaret Kern, David Stillwell,Michal Kosinski, Lyle Ungar, and Hansen Andrew Schwartz. Developing age andgender predictive lexica over social media. In Proceedings of EMNLP, 2014.

42. Yuzuru Tanahashi, Chien-Hsin Hsueh, and Kwan-Liu Ma. An efficient frameworkfor generating storyline visualizations from streaming data. IEEE transactions onvisualization and computer graphics, 21(6):730–742, 2015.

43. Teun Terpstra, A de Vries, R Stronkman, and GL Paradies. Towards a realtimetwitter analysis during crises for operational crisis management. In Proceedings ofISCRAM, 2012.

44. Zeynep Tufekci. Big questions for social media big data: Representativeness, va-lidity and other methodological pitfalls. arXiv preprint arXiv:1403.7400, 2014.

45. A. Tumasjan, T. O. Sprenger, P. G. Sandner, and I. M. Welpe. Predicting electionswith Twitter: What 140 characters reveal about political sentiment. In Proceedingsof ICWSM, pages 178–185, 2010.

46. Ro Valitutti. Wordnet-affect: an affective extension of wordnet. In Proceedings ofLREC, pages 1083–1086, 2004.

47. Sarah Vieweg, Amanda L. Hughes, Kate Starbird, and Leysia Palen. Microblog-ging during two natural hazards events: What Twitter may contribute to situa-tional awareness. In Proceedings of the SIGCHI Conference on Human Factors inComputing Systems, pages 1079–1088, 2010.

48. Svitlana Volkova and Yoram Bachrach. On predicting sociodemographic traits andemotions from communications in social networks and their implications to onlineself-disclosure. Cyberpsychology, Behavior, and Social Networking, 18(12):726–736,2015.

49. Ievgen Vorobiov. Surprise! Ukraine Loves NATO, 2015. http://foreignpolicy.com/2015/08/13/surprise-ukraine-loves-nato/.

50. Jingying Wang and Tingshao Zhu. Classify sina weibo users into high orlow happiness groups using linguistic and behavior features. arXiv preprintarXiv:1507.01796, 2015.

51. Yu Wang, Tom Clark, Eugene Agichtein, and Jeffrey Staton. Towards trackingpolitical sentiment through microblog data. Proceedings of ACL, 2014.

Contrasting Public Opinion Dynamics and Emotional Response ...

Documents