Sentiment and Textual analysis of Create-Debate data

GENDER BASED BEHAVIOR IN CO-AUTHORSHIP NETWORK

Poorva PotdarSentiment and Textual analysis of Create-Debate dataEECS 595 End Term Project

Hello Everyone,I am Poorva Potdar and masters student in Comp sci-And My final project is related to sentiment analysis of web-data.1EUREKA!! Getting the IdeaWhy sentiment analysis?Huge amount of opinionated Text on webSentiment Analysis on web popularity of a product, movie or a person as such.

Idea:Create Debate online debating forum where people argue for/against some topic.Mine for the salient text features for agreement/disagreement posts.

Today there is just a large amount of opinionated text on web we can mine these opinions, extract polarity of words toGuage the popularity of product, movie or a person.

For the project, I chose to do some kinda of text analysis on the debates or opinions expressed by people on the createDebate social networking site!2Math

-14308 Debates.-983800 Sentences!-178290 Posts, -9194 Users - Labeled datasetNeutralAgreementDisagreementStructural Analysis Certain features of the language in the post that make it a high score agreement/disagreement post.Behavioral Analysis Aspects of Users behavior that give him a high rank on the forum.Creating the Haystack .So the first step was to get the dataset in place Drago asked me to work with Amjad, phd student at CLAIR on this one and I got the debate dataset Which consists of around 14 thousand debates, 900 thousand sentences, 170 thounsand posts and 9000 users.

Each of the posts in the debates had a labeled score associated with it like on this website , users can go and assign points, approve/disapprove previous arguments.

Users can either start a new argument which is termed as neutral and has a polarity sign 0.Users can approve the existing arguent termed as agreement post from here on.Users can disapprove the existing argument termend as disagreement post from here on.

As seen each post has a labeled point associated.

So, the idea was to really do two kinds of analysisStructural analysis try to find salient features in the text of the post that make it a highly ranked post? Behavioral analysis For a particular author, we could find his aggregated score for all the posts he is written., and thus find his labeled rank.The idea was to get a predicted ranking of the authors based on the patterns they depict and then correlate the predicted and the gold standard rankings!

3What's the gain?Influence detection in a community

Sub-Group Detection

Stance Identification Are there any visible groups with a particular stance?

Predict the Crowd Trend for a particular topic of interest?

Text Summarization

So this kind of analysis can be used in several NLP areasWhich are4

Finding the needle - structural features .I tried to mine for several salient features that could potentially give me some results Is readability of post a good measure IS polarity of post indicative of its rank?Is there a popular dependency pattern observed?What role do emoticons play?5Experiment 1 : Polarity MeasureIntuition : Is the number of +ve/-ve words an indicative of how popular a post is?Tool Opinion Finder/ Wordnet.Output of processed data by Opinion Finder.It think it's wrong to assume that in order to be a revolutionary thinker you have to be crazy MPQAPOL Indicates the polarity of the word like badMPQASRC Indicates the opinion source in the sentence like ItMPQASD Direct subject expression in the sentence like said

Result : No evident correlation between number of polar words and the rank of the postAuthors use equal distribution of positive and negative words while expressing agreement/disagreement.

PostsAgreement PostsDisagreement PostsPositive words-0.008240.012647Negative words-0.010240.01392For finding a polarity of a post, I used opinion finder to look for +ve ve wordsI first used sentiwordnet for this but sentiword doesnt handle negations well not good and so onOpinion finder actually considers the context of the sentence as well. in do not agree agree is tagged as negative

6Experiment 2 : Readability MeasureIntuition : Do the posts that are more readable/formal gain higher scores?Tool Flesch Toolkit to analyze the Flesch Readability measure for each post.Calculated Pearsons coefficient between the labeled score and Flesch score for each of the posts.

Result : High correlation - the more formal the language of a post, the more is the points associated with it.

Eg 1 : good times . . .bring it back ! -------------=-=-=-=-=-=-=-=-=-=-=-=-==-=-=- )))))))))))) [Flesch 0, Labeled points - 1]Eg 2 : Vegetables is often seen as more healthy than eating meat. [Flesch 93.12, Labeled points 29 (max)]

PostsAgreement PostsDisagreement PostsPearsons correlation for flesch readability0.2069740.169236In the Flesch Reading Ease test, higher scores indicate material that is easier to read; lower numbers mark passages that are more difficult to read. The formula for the Flesch Reading Ease Score (FRES) test is7Experiment 3 : Emoticon analysisIntuition : Do Emoticons in agreement/disagreement posts have any correlation with their labeled scores?Tool CMU Ark Tagger [Stanford Parser doesnt scale well].Pearsons coefficient between the labeled score and number of +ve/-ve emoticons for agreement/disagreement posts.

Result : High correlation between number of emoticons and rank of disagreement posts. Analysis : authors tend to use expressive emoticons like smiles to give a sarcastic opinion regarding a particular argument. Hey! Whats that supposed to mean?;) , Sure If you say so :P.

PostsAgreement PostsDisagreement PostsPositive emoticons-0.023750.38943Negative emoticons-0.0035270.03421I used the CMU ark tagger to mine the various emoticons and suprising they gave a high correlation with the disagreement postsAuthors tend to use sarcasm in disagreement posts8Experiment 4 : Dependency ParseIntuition : Do highly ranked agreement/disagreement posts depict a popular dependency pattern? Agreement posts tend to express an agreement early on in the post, while disagreement is mild.Tool Stanford Parser Syntactic and Dependency Parse of the posts.

Result: A lot of highly ranked agreement posts showed a popular dependency pattern as follows that begins with -I->nsubj->+ve [I agree to, I like your point, I up-voted your argument]

Stanford Parser + ExtractDependencies Code to traverse PRP to PRP$

Sentiwordnet

PostsAgreement PostsPearsons coeff with I->nsubj->+ve pattern0.252146In this experiement, ran Stanford dependency parser over the posts to find any popular patternAnd I observed that highly ranked agreement posts begin with this pattern Authors tend to express their agreement at the start of any agreement post.9

Finding the needle - behavioral features .Tried to find some salient patterns in the behavior of an author to get their predicted ranking-

10Which Authors get the highest rank? -1Intuition : To find if average number of times an author participates in a thread has a correlation with his ranking?

Pearsons coefficientAverage number of times an author participates in a thread.0.489Result : There is a pretty evident positive correlation of an authors points to the number of times he participates in the discussion posts per thread.

Basic experiments - avg number of times an author participates in a thread how many times he comments in a thread.11Which Authors get the highest rank?-2Intuition : To find if authors who participate in some kind of discussion/ or start a new thread get a high rank ?

Pearsons coefficientAuthors who agree 0.847Authors who disagree0.770Authors who start a new thread.0.60

Result : Rating of authors who agree > Rating of authors who disagree more > Rating of authors who start a new debate.Authors who participate more in discussions are more popular.

In this experiment , I wrote a script to aggregate the number of times an author agrees with somone, disagree or starts an altogether new post.Plotted the correlations and the pearsons coefff to find that authors who tend to agree with someone get a higher ranking authors who disagree

12Which Authors get the highest rank?-3Intuition : To find if a authors that participate early/late in discussion fetch more ranking?

Pearsons coefficientAuthors who participate early0.1990Authors who participate late-0.00358Result : Authors participating late in discussion are likely to have higher ranking.By Intuition, authors who come late in discussion already know the opinion bias.Participating early doesnt help in ranking

Next one in this catergory I tried to see if the time when an author enters a discussion is criticalLet say an author comments on a post early in the lifecule of the post And one author comes laterOn an average based on the correlation figures we can say that authors who come late are highly ranked than -13Get the Ranking of Authors w.r.t featuresTrained a linear regression model using Wekas Libsvm and got a predicted ranking of all authors based on the features.Got a correlation coefficient by comparing these rankings vs the gold standard rankings.

SVMs Correlation CoefficientGold Standard Rankings/ Predicted Rankings.0.300Result : The feature vector set shows a decent correlation with the actual rankings.

Finally fed in all the features to SVM to get a predicted ranking of authors based on featyre.I cal the correlation coeff to find it was positive whch says that there was a dependency between the features and ranking.14Future WorkIn this project, I essentially looked at some of the structural and behavioral featuresThe opinion finder tool also tells whether it is a subjective or objective. One of the future Experiments to find if there exists a correlation between subj/obj sentences and score of post?Does the length of the post matter?Going forward - consolidate all these features and results in the database and make it available as an open-source dataset

Thank You!