Leveraging Joint Interactions for Credibility Analysis in ...resources.mpi-inf.mpg.de/impact/credibility... · A model to capture joint interaction between language, topics, users

Leveraging Joint Interactions for Credibility Analysis in News Communities

Subhabrata Mukherjee and Gerhard WeikumMax Planck Institute for Informatics

CIKM 2015

Motivation

➢ Media plays a crucial role in public dissemination of information

➢ However, people believe there is substantial media bias in news in view of inter-dependencies and cross-ownerships of media companies and other industries (like energy)

➢ 4 out of 5 Americans among younger generations do not trust major news networks [Gallup poll, 2013]

➢ This work: Credibility Analysis of News Communities

News Community

➢ A news community is a news aggregator site (e.g., reddit.com, digg.com, newstrust.net) where:➢ Users can give explicit feedback (e.g., rate, review, share) on the

quality of news➢ Interact (e.g., comment, vote) with each other

➢ However, this adds user subjectivity as users incorporate their own bias and perspectives in the framework

➢ Controversial topics create polarization among users which influence their evaluation

Contributions

● A model to capture joint interaction between language, topics, users and sources leading to better prediction than the ones in isolation

● User expertise, source trustworthiness, language objectivity, topical perspective and article credibility mutually reinforce each other

● A supervised Conditional Random Field model that can capture these interactions, and handle real-valued ratings

s1 s

1

d1

r11

r12

u1

u2

d2

r22

u2

y1

y2

C1 C

2

Source

Article

Review

User

ExampleFACTORS

s1 s

1

d1

r11

r12

u1

u2

d2

r22

u2

y1

y2

C1 C

2

Source

Article

Review

User

Alternet.org(progressive/liberal)

Why do conservaties hate your children?

Ratings

Discussions(liberal vs.conservative)

Example

Topic: Climate

FACTORSInstantiation

s1 s

1

d1

r11

r12

u1

u2

d2

r22

u2

y1

y2

C1 C

2

Viewpoint, Expertise

Why do conservaties hate your children?

Ratings


Example

Topic: Climate

Source

Article

Review

User

FACTORSFEATURES

s1 s

1

d1

r11

r12

u1

u2

d2

r22

u2

y1

y2

C1 C

2


Emotionality, Discourse

Ratings


Example

Topic: Climate

Source

Article

Review

User

FACTORSFEATURES

s1 s

1

d1

r11

r12

u1

u2

d2

r22

u2

y1

y2

C1 C

2




Example

Ratings

Topic

Source

Article

Review

User

FACTORSFEATURES

s1 s

1

d1

r11

r12

u1

u2

d2

r22

u2

y1

y2

C1 C

2



Bias, Viewpoint, Expertise

Example

Topic

Ratings

Source

Article

Review

User

FACTORSFEATURES

s1 s

1

d1

r11

r12

u1

u2

d2

r22

u2

y1

y2

C1 C

2

Source

Article

Review

User

Article Credibility Rating?

Trustworthiness

Objectivity

Credibility

Expertise

TaskFACTORSATTRIBUTES

Credibility Analysis

➢ Given a set of news sources generating news articles, and users reviewing them on different qualitative aspects with mutual interactions:➢ Jointly rank the sources, articles, and users based

on their trustworthiness, credibility,and expertise

Credibility of Statements in Health Communities[S. Mukherjee et al.: KDD‘14]

s1 s

1

d1

r11

r12

u1

u2

d2

r22

u2

y1

y2

C1 C

2

Source

Article

Review

User

Objectivity

Language Features

Assertives, Factives, Hedges, Implicatives, Report, Discourse, Subjectivity etc.

1. M. Recasens, C. Danescu-Niculescu-Mizil, and D. Jurafsky. Linguistic models for analyzing and detecting biased language. In ACL, 2013.2. S. Mukherjee, G. Weikum, and C. Danescu-Niculescu-Mizil. People on drugs: Credibility of user statements in health communities. KDD, 2014.

➢ Only 33% of the articles have explicit tags

➢ Use Latent Dirichlet Allocation to learn the latent topic distribution in the corpus of news articles

Topic Features

Source Features

Category Elements

Engagement answers, ratings (given / received), comments etc.

Agreement Inter-user agreement

Topics perspective and expertise

Interactions user-user, user-item, user-source

User Features

s1 s

1

d1

r11

r12

u1

u2

d2

r22

u2

y1

y2

C1 C

2

Source

Article

Review

User

Article Credibility Rating?

Source Models

Article Language Model, Topic Model

Review Language Model, Topic Model

User Models

How to aggregate?

Given a factor, with its features, use Support Vector Regression to learn a model that will predict its rating for an article.

Probability Mass Function for discrete labels:

Probability Density Function for continuous ratings:

Conditional Random Field

Clique potential

Energy Function

User Potential

Source Potential Language Potential Topic Potential

Clique: source, article, <users>, <reviews>

error of predictor SVR

partitions the user space

user expertise

source trustworthiness

Energy Function

language objectivity topical perspective

Σ needs to be positive definite for inverse to exist → {α, β, γ} > 0

Makes sense: predictor reliability should be positive

The joint p.d.f is a multivariate gaussian distribution

Maximize log-likelihood with respect to log λk

instead of λk

Prediction is the expected value of the function given by the mean of the Multivariate Gaussian distribution:

Constrained optimization problem.Gradient ascent cannot be directly used.

Experiments: NewsTrust

Data available at: http://www.mpi-inf.mpg.de/impact/credibilityanalysis/

Predicting User Ratings

Users, Articles, Ratings

+Time

+Review Text

+Review Text andInteractions

1. Y. Koren. Factorization meets the neighborhood: A multifaceted collaborative filtering model. KDD, 2008.2. J. McAuley and J. Leskovec. Hidden factors and hidden topics: Understanding rating dimensions with review text. RecSys, 2013.3. J. J. McAuley and J. Leskovec. From amateurs to connoisseurs: modeling the evolution of user expertise through online reviews. In WWW, 2013.

Predicting Article Credibility Ratings




Ranking Trustworthy News Sources

Ranking Expert Users:

Sample Output: Most and Least Trust Sources on Sample Topics

Conclusions

➢ Joint interaction between language, topics, users and sources lead to better prediction in multiple tasks

➢ User expertise, source trustworthiness, language objectivity, topical perspective and article credibility mutually reinforce each other

Ongoing Work

➢ Analyze temporal evolution of these factors

➢ Communities are inherently dynamic in nature➢ Source trustworthiness, and user expertise change

with time

➢ To this end we propose an Experience-aware Item Recommendation for Evolving Review Communities, ICDM 2015.

Leveraging Joint Interactions for Credibility Analysis in ...resources.mpi-inf.mpg.de/impact/credibility... · A model to capture joint interaction between language, topics, users

Documents