Computational Communication Science: building the toolchainvanatteveldt.com/wp-content/uploads/2019_atteveldt_zurich.pdf · Computational Communication Science Data Gathering Analyzing

Computational Communication Science Data Gathering Analyzing content Conclusion

Computational Communication Science:building the toolchain

Wouter van Atteveldt

April 2019

Computational Communication Science: building the toolchain Wouter van Atteveldt


CCS: What? Why? How?










Computational Communication Science

• What, why, how?• Building the Toolchain:

• Data gathering: scraping & tracking• Analysis: Text and beyond• Transparency: Data and code sharing




Computational Communication Research

Welcoming your submissions!







Why CCS?

• Explosive increase in available data, tools, processing• Most "big data" is communicative• Potential radical boost to study of communication• But has numerous problems, challenges, pitfalls




CCS: challenges & pitfalls

• Accessibility of data• Representativeness/validity of ’found’ data• Validity of computational methods• Ethical conduct• Skills & Infrastructure







Elements of the toolchain

• Data: How do we get the ’digital traces’?• Analysis: From (textual) traces to data• Sharing: From one to many



Goals and challenges

Why do we need digital trace data?

Fragmentation of information• Minimal mass media effects?• TV as last homogenous medium, demographically

challenged?Specific effects of online communication

• E.g. Fear of online filter bubbles / polarisation




Ideal goal

• Message consumption/production data• Full text and metadata• Of representative sample of population

• and/or fully connected subsample(s)• Linked with attitude/behaviour measures




Getting text: Technical challenges

• News: can be scraped, retrieved via Nexis etc• Paywalls make it more difficult• Social media companies tries to block scraping




"We are in the post-API age"

(Deen Freelon, PolComm, forthcoming)Computational Communication Science: building the toolchain Wouter van Atteveldt



Getting text: Legal challenges

• Copyright/database law can block scraping, blockssharing

• Contract law can block scraping of restricted content• Hacking laws might make scraping actually illegal• Laws are uncertain and vary over jurisdictions/time• Many researchers are anarchists, many institutions

cautious(IANAL!)




Gathering digital trace dataDesktop browsing:

• Desktop plugin(e.g. ASCoR personalized communication)

• History donation(e.g. Web Historian; Menchen-Trevino)

Mobile app use• App can access (e.g. MobileDNA)

Mobile phone logs• App can access (e.g. Kobayashi & Boase)

Mobile news browsing• problem: how to get it :-)



CCS.Amsterdam: Mobile news tracking

Mobile news viewing: Challenges

We want to know what people see on their mobile, but• Most mobile browsers don’t allow plugins• HTTPS/encrypted app communication makes

proxy/MITM difficult• esp. combined with certificate pinning

• Most apps have in-app browsing; proprietary protocols




Mobile news viewing: possibilities

1 Browser sync + desktop plug-in / application• Can build on browser plugins

2 GDPR requests by user• No facility to make request on behalf user• Instructions needed for each app• (need something akin to FSD)




CCS.Amsterdam: Tracking the filter bubble

• NWO-Joint Escience Data Science (JEDS) programme• Goal: develop mobile tracking, analyse effects of mobile

news on attitudes• Team:

• Social science: me, Damian Trailling (UvA) , JudithMoller (UvA), Felicia Locherbach (VU)

• Engineering: Antske Fokkens (VU), Laura Hollink(CWI), Jisk Attema (NLeSC), Laurens Bogaardt(NLeSC)

• Law/normative theory: Natali Helberger (UvA)




Mobile Tracking

1 Test sync & plugin strategy2 Extend WebHistorian with (mac) app for Safari3 Use facebook ’data download’ for facebook engagement4 (hopefully) Work with news agencies for in-app data

(See Van Atteveldt et al., ICA 2019; and informal postconf)




The need for content analysis

• Trace data often partially unstructured/symbolic (text,speech, image, video)

• Need to convert ’text to data’• Measure relevant quantities• In a valid and scalable way

• Focus on text• But really cool stuff is happening with image analysis,

see ICA pre-conf!




What are the relevant quantities?Depends on RQ, but often see message as (collection of)statement(s):

• source• topic and/or target• tone/sentiment

This can yield measurement per message, or per statement• Per message is easier, but many texts contain multiple

statements• Construct semantic network from text

• (Core Sentence approach; political claims analysis; NET)




What techniques do we need?• Identifying actors: (easiest)

• Dictionaries, Named Entity Recognition, Coreferenceresolution

• Identifying issues/topics: (doable)• "Automatic text classification"• Dictionaries, (Structural) Topic modeling, Supervised

machine learning• Identifying tone/sentiment: (hard!)

• "Sentiment anlysis"/"Opinion extraction"• Dictionaries, Supervised machine learning

• From text to statements• "Semantic Role Labeling" (sort of)• Syntactic analysis, Supervised machine learning



CCS.Amsterdam: Sentiment Analysis

CCS.Amsterdam: Text analysis

• Multiple projects on Sentiment analysis, syntacticanlaysis, deep learning, crowd coding, topic modeling, etc

• Members (i.a.): Damian Trilling (UvA), Anne Kroon(UvA), Kasper Welbers (VU), Antske Fokkens (VU)




The importance of sentiment analysis

• Many theories in (political) communication connected totone

• Issue positions• Negative campaigning• Conflict news• Reviews, reputation, etc

• Tone is notoriously hard to define & measure(automatically)

• Ambiguous• Creative




Test case: Dutch economic news

• Is news positive or negative about economy?• Interesting for retrospective voting, framing, news bias• Should be ‘best-case’ scenario for automatic analysis

• Relatively unambiguous• Relatively factual

• RQ: Can we automatically measure the tone of economicnews?




How do off-the-shelf dictionaries do?

• Mark Boukes et al., ICA 2018• Compare undergrad coders with existing dictionaries










What else can we try?

Compare (triple-coded) gold standard with:• Undergrads• Dictionaries• Crowd coding• (translation + dictionaries)• Machine learning




Gold standard

• Selected ~100 headlines from ICR sample• Coded independently by me, Mark, Mariken van der

Velden ($α$=.78)• All differences resolved except 9 disagreements

• (E.g. “Interest rates hit zero”, “Greece will be fine for acouple more weeks”, “Aging population puts brake onhouse prices”)




Crowd coding

• Crowd coding promising solution for sentiment coding• Decision is simple / “intuitive”• More cheap coders > Fewer better coders• Method:

• Same n=88 sentences• Each sentences coded by ~5 coders• (.02$ per sentence/coders, 15$ total)• Simple instructions, single question• Use gold questions to filter coders













Machine learning

• Train on 6,203 manually coded headlines• Test on gold sample• 3 methods:

• ’Traditional’ SVM on document-lemma matrix• ’Traditional’ neural net with embeddings

(document-embeddings matrix)• ’Deep learning’ Convolutional Neural Network




Convolutional Neural Network• Most Traditional ML uses only frequencies• Misses context, relations between words

• "it wasn’t bad, it was actually quite good"• "it wasn’t good, it was actually quite bad"

• Convolutional neural networks use word order• ’deep learning’ method originating from image analysis• N-grams of words representations are concatenated,

pooled per unit• Pooled output is then used as input for regular learning• See: Yoav Goldberg, Neural Network Methods for

Natural Language Processing; Anne Kroon et al, 2019ICA on Dutch embeddings.




Convolutional Neural Network




Results

• How do all methods compare to gold standard?• How do methods correlate with each other?









Open Science: Sharing & Transparancy

What is open science?

• Research Transparency:• data access• transparent design• analytical transparency

• Oppenness boosts:• Reproducibility• Robustness• Replicability• Generalizability

(e.g. Bowman & Keene, 2018; Klein et al., 2018; Munafò etal., 2017; Nosek et al., 2015)







Why open CCS?

Opportunity:• Data and tools are digital• Culture of open source, sharing• Possibility for reproducible research

Need:• Big data can be abused easily• Skills and tools can be scarce• Strong need for openness: share, inspect, improve




Open CCS: Challenges

Data sharing challenges:• Fear of being scooped• Proprietary data, copyright, legal uncertainty• Not enough incentives

Code/Tool sharing challenges:• Fear of being caught making mistakes• Effort required to turn research code into software• Not enough incentives






What’s next? Towards an open CCS

What’s next? Towards an open CCS

• Building the toolchain, doing the research• Focus on validity, usability, re-usability:

• Sharing the tools• Sharing the data• Sharing the results• Sharing the skills

• This effort needs to be collaborative!


Computational Communication Science: building the toolchainvanatteveldt.com/wp-content/uploads/2019_atteveldt_zurich.pdf · Computational Communication Science Data Gathering Analyzing

Documents