Visual Analytics for Linguistics - Day 3 ESSLLI

Visual Analyticsfor Linguistics -

Day 3

Olga Scrivner

Course Info

Charts

TextVisualization

ITMS

PreprocessingData

DataVisualization

Cluster Analysis

Topic Modeling

Google Book API

Visual Analytics for Linguistics - Day 3

Olga Scrivner


Day 3

Olga Scrivner

Course Info

Charts

TextVisualization

ITMS

PreprocessingData

DataVisualization

Cluster Analysis

Topic Modeling

Google Book API

What You Will Learn

DAY 1 Introduction to Visual Analytics

DAY 2 Visualization Methods, Design, and Tools

DAY 3 Working with Unstructured Data

DAY 4 Working with Structured Data

DAY 5 Advanced Analytics


Day 3

Olga Scrivner

Course Info

Charts

TextVisualization

ITMS

PreprocessingData

DataVisualization

Cluster Analysis

Topic Modeling

Google Book API

Our Materials - Web Site

http://obscrivn.wixsite.com/visualization




Day 3

Olga Scrivner

Course Info

Charts

TextVisualization

ITMS

PreprocessingData

DataVisualization

Cluster Analysis

Topic Modeling

Google Book API

What We Need

I Interactive Text Mining Suite

I Voyant

I R and Rstudio

I R libraries: ggplot2, plotly, reshape2


Day 3

Olga Scrivner

Course Info

Charts

TextVisualization

ITMS

PreprocessingData

DataVisualization

Cluster Analysis

Topic Modeling

Google Book API

What We Need

I Interactive Text Mining Suite

I Voyant

I R and Rstudio

I R libraries: ggplot2, plotly, reshape2


Day 3

Olga Scrivner

Course Info

Charts

TextVisualization

ITMS

PreprocessingData

DataVisualization

Cluster Analysis

Topic Modeling

Google Book API

Quiz: Which Chart Are You?

https://www.sisense.com/blog/quiz-chart/

https://www.sisense.com/blog/quiz-chart/


Day 3

Olga Scrivner

Course Info

Charts

TextVisualization

ITMS

PreprocessingData

DataVisualization

Cluster Analysis

Topic Modeling

Google Book API

Creating a Bar Chart

I The value of a column in the data set. This is done withstat=“identity” , which leaves the y values unchanged.

I The count of cases for each group - each x valuerepresents one group.


Day 3

Olga Scrivner

Course Info

Charts

TextVisualization

ITMS

PreprocessingData

DataVisualization

Cluster Analysis

Topic Modeling

Google Book API

Creating a Bar Chart - Sample


Day 3

Olga Scrivner

Course Info

Charts

TextVisualization

ITMS

PreprocessingData

DataVisualization

Cluster Analysis

Topic Modeling

Google Book API

Creating a Bar Chart - Sample


Day 3

Olga Scrivner

Course Info

Charts

TextVisualization

ITMS

PreprocessingData

DataVisualization

Cluster Analysis

Topic Modeling

Google Book API

Creating a Bar Chart - Values


Day 3

Olga Scrivner

Course Info

Charts

TextVisualization

ITMS

PreprocessingData

DataVisualization

Cluster Analysis

Topic Modeling

Google Book API

Creating a Bar Chart - Counts

To get a bar graph of counts, we do not map a variable to y,and we use stat=“count”


Day 3

Olga Scrivner

Course Info

Charts

TextVisualization

ITMS

PreprocessingData

DataVisualization

Cluster Analysis

Topic Modeling

Google Book API

Creating a Bar Chart - Counts


Day 3

Olga Scrivner

Course Info

Charts

TextVisualization

ITMS

PreprocessingData

DataVisualization

Cluster Analysis

Topic Modeling

Google Book API

Title


Day 3

Olga Scrivner

Course Info

Charts

TextVisualization

ITMS

PreprocessingData

DataVisualization

Cluster Analysis

Topic Modeling

Google Book API

Creating Line Chart


Day 3

Olga Scrivner

Course Info

Charts

TextVisualization

ITMS

PreprocessingData

DataVisualization

Cluster Analysis

Topic Modeling

Google Book API

Creating Line Chart


Day 3

Olga Scrivner

Course Info

Charts

TextVisualization

ITMS

PreprocessingData

DataVisualization

Cluster Analysis

Topic Modeling

Google Book API

Creating Area Chart

http://www.r-graph-gallery.com/136-stacked-area-chart/

http://www.r-graph-gallery.com/136-stacked-area-chart/


Day 3

Olga Scrivner

Course Info

Charts

TextVisualization

ITMS

PreprocessingData

DataVisualization

Cluster Analysis

Topic Modeling

Google Book API

Creating Scatter Plot

http://www.r-graph-gallery.com/272-basic-scatterplot-with-ggplot2/

http://www.r-graph-gallery.com/272-basic-scatterplot-with-ggplot2/


Day 3

Olga Scrivner

Course Info

Charts

TextVisualization

ITMS

PreprocessingData

DataVisualization

Cluster Analysis

Topic Modeling

Google Book API

Creating Bubble Plot

https://plot.ly/r/bubble-charts/



Day 3

Olga Scrivner

Course Info

Charts

TextVisualization

ITMS

PreprocessingData

DataVisualization

Cluster Analysis

Topic Modeling

Google Book API

Creating Bubble Plot




Day 3

Olga Scrivner

Course Info

Charts

TextVisualization

ITMS

PreprocessingData

DataVisualization

Cluster Analysis

Topic Modeling

Google Book API

Creating Heatmap

http://www.r-graph-gallery.com/215-interactive-heatmap-with-plotly/




Day 3

Olga Scrivner

Course Info

Charts

TextVisualization

ITMS

PreprocessingData

DataVisualization

Cluster Analysis

Topic Modeling

Google Book API

Creating Heatmap




Day 3

Olga Scrivner

Course Info

Charts

TextVisualization

ITMS

PreprocessingData

DataVisualization

Cluster Analysis

Topic Modeling

Google Book API

Creating Heatmap


Day 3

Olga Scrivner

Course Info

Charts

TextVisualization

ITMS

PreprocessingData

DataVisualization

Cluster Analysis

Topic Modeling

Google Book API

Creating Word Cloud


Day 3

Olga Scrivner

Course Info

Charts

TextVisualization

ITMS

PreprocessingData

DataVisualization

Cluster Analysis

Topic Modeling

Google Book API

Word Cloud - Contest - 10 min

I Create your own word cloudI Look at the function - type ?wordcloud2 and run

I Can you change a shape of your cloud?I Save (or make a screenshot) and post it on

twitter/facebook etc


Day 3

Olga Scrivner

Course Info

Charts

TextVisualization

ITMS

PreprocessingData

DataVisualization

Cluster Analysis

Topic Modeling

Google Book API

Why Analyze Text?

The “epic transformation of archives” - shifting from print todigital archival form (Folsom, 2007)

“As our collective knowledge continues to be digitized andstored (...) it becomes more difficult to find and discover

what we are looking for.” (Blei 2012)


Day 3

Olga Scrivner

Course Info

Charts

TextVisualization

ITMS

PreprocessingData

DataVisualization

Cluster Analysis

Topic Modeling

Google Book API

Text Mining Challenges

source - 1) Dan Jurafsky, 2) Text Mining with R for Social Science Research (Ryan Wesslen)


Day 3

Olga Scrivner

Course Info

Charts

TextVisualization

ITMS

PreprocessingData

DataVisualization

Cluster Analysis

Topic Modeling

Google Book API

Basic Terminology


Day 3

Olga Scrivner

Course Info

Charts

TextVisualization

ITMS

PreprocessingData

DataVisualization

Cluster Analysis

Topic Modeling

Google Book API

What is Bag of Words?

I Simplest way to quantify text

I Word order ignored

I Term counts per document

I N-grams (uni-grams, bi-grams)

Source - Chris Manning


Day 3

Olga Scrivner

Course Info

Charts

TextVisualization

ITMS

PreprocessingData

DataVisualization

Cluster Analysis

Topic Modeling

Google Book API

Preprocessing

I Tokenization (splitting words)

I Cleaning (lower case, punctuation)

I Stemming

I works, worked → work

I Filter (stopwords)

I and, the, a

Source - Wesslen


Day 3

Olga Scrivner

Course Info

Charts

TextVisualization

ITMS

PreprocessingData

DataVisualization

Cluster Analysis

Topic Modeling

Google Book API

Preprocessing



I Stemming



I and, the, a

Source - Wesslen


Day 3

Olga Scrivner

Course Info

Charts

TextVisualization

ITMS

PreprocessingData

DataVisualization

Cluster Analysis

Topic Modeling

Google Book API

Preprocessing



I Stemming



I and, the, a

Source - Wesslen


Day 3

Olga Scrivner

Course Info

Charts

TextVisualization

ITMS

PreprocessingData

DataVisualization

Cluster Analysis

Topic Modeling

Google Book API

Macro-analysis

Concept Macro-analysis (Jockers, 2013)

“the construction of abstract models”(Jasinski, 2001)

Methods Tag clouds, heat maps, clusters, topics,network graphs

Tools GUI: Voyant, Papermachine, ITMSTUI: Mallet, Meta, R and Python packages


Day 3

Olga Scrivner

Course Info

Charts

TextVisualization

ITMS

PreprocessingData

DataVisualization

Cluster Analysis

Topic Modeling

Google Book API

Visual Analytics

Visual Analytics - “The science of analytical reasoningfacilitated by visual interactive interfaces” (Thomas et all.,2005)

I Graphs, maps and trees for literature analysis (Moretti,2005)


Day 3

Olga Scrivner

Course Info

Charts

TextVisualization

ITMS

PreprocessingData

DataVisualization

Cluster Analysis

Topic Modeling

Google Book API

Visualization Methods

I Word clouds to analyze a novel (Vuillemot et al., 2009)


Day 3

Olga Scrivner

Course Info

Charts

TextVisualization

ITMS

PreprocessingData

DataVisualization

Cluster Analysis

Topic Modeling

Google Book API


I Social network graphs of characters in Greek tragedies(Rydberg-Cox, 2011)


Day 3

Olga Scrivner

Course Info

Charts

TextVisualization

ITMS

PreprocessingData

DataVisualization

Cluster Analysis

Topic Modeling

Google Book API


I Literary fingerprint and summaries (Oelke et al., 2012)


Day 3

Olga Scrivner

Course Info

Charts

TextVisualization

ITMS

PreprocessingData

DataVisualization

Cluster Analysis

Topic Modeling

Google Book API


I Tracking emotion and sentiment in fairy tales(Mohammad, 2012)


Day 3

Olga Scrivner

Course Info

Charts

TextVisualization

ITMS

PreprocessingData

DataVisualization

Cluster Analysis

Topic Modeling

Google Book API

Topic Modeling

Discovering underlying theme of collection from Science magazine1990-2000 (Blei 2012)


Day 3

Olga Scrivner

Course Info

Charts

TextVisualization

ITMS

PreprocessingData

DataVisualization

Cluster Analysis

Topic Modeling

Google Book API

Topics - Word Term


Day 3

Olga Scrivner

Course Info

Charts

TextVisualization

ITMS

PreprocessingData

DataVisualization

Cluster Analysis

Topic Modeling

Google Book API

Topics - Word Term


Day 3

Olga Scrivner

Course Info

Charts

TextVisualization

ITMS

PreprocessingData

DataVisualization

Cluster Analysis

Topic Modeling

Google Book API

Wikipedia Topics

http://www.princeton.edu/~achaney/tmve/wiki100k/browse/topic-presence.html





Day 3

Olga Scrivner

Course Info

Charts

TextVisualization

ITMS

PreprocessingData

DataVisualization

Cluster Analysis

Topic Modeling

Google Book API

Wikipedia Topics - Assignment - 10 min

1. Language Related Topic2. Words: Dialect3. Related Document: Macedonian Language4. Related Document: Egyptian hieroglyphs5. Go to Full article:6. Find meaning:


Day 3

Olga Scrivner

Course Info

Charts

TextVisualization

ITMS

PreprocessingData

DataVisualization

Cluster Analysis

Topic Modeling

Google Book API

Voyant

http://voyant-tools.org/



Day 3

Olga Scrivner

Course Info

Charts

TextVisualization

ITMS

PreprocessingData

DataVisualization

Cluster Analysis

Topic Modeling

Google Book API

Voyant




Day 3

Olga Scrivner

Course Info

Charts

TextVisualization

ITMS

PreprocessingData

DataVisualization

Cluster Analysis

Topic Modeling

Google Book API

Voyant - 10 min


I Examine visualization charts (identify typesand properties)

I Apply various filters and queries



Day 3

Olga Scrivner

Course Info

Charts

TextVisualization

ITMS

PreprocessingData

DataVisualization

Cluster Analysis

Topic Modeling

Google Book API

Voyant Tools - Bubblelines - 7 min

http://docs.voyant-tools.org/tools/

I Delete top termsI Search for man and woman

I Make sure to have “separate lines for terms” clickedI Change search terms



Day 3

Olga Scrivner

Course Info

Charts

TextVisualization

ITMS

PreprocessingData

DataVisualization

Cluster Analysis

Topic Modeling

Google Book API

Voyant Tools - Pair Work - 10 min

http://docs.voyant-tools.org/tools/I Examine visualization methodsI Select 5 methodsI Look at the documentation and how to use them



Day 3

Olga Scrivner

Course Info

Charts

TextVisualization

ITMS

PreprocessingData

DataVisualization

Cluster Analysis

Topic Modeling

Google Book API

Interactive Text Mining Suite

I A user-friendly tool for quantitative analysis andvisualization of unstructured data

I Platform-independent

I Interactive


Day 3

Olga Scrivner

Course Info

Charts

TextVisualization

ITMS

PreprocessingData

DataVisualization

Cluster Analysis

Topic Modeling

Google Book API

ITMS Structure

1. File Uploads

I Upload files (txt, pdf, rdf and Google books API)

2. Data Preparation

I Data preprocessing (stopwords, stemming, metadata)

3. Data Visualization

I Word frequencies, Cluster analysis and topic modeling


Day 3

Olga Scrivner

Course Info

Charts

TextVisualization

ITMS

PreprocessingData

DataVisualization

Cluster Analysis

Topic Modeling

Google Book API

ITMS Structure

1. File Uploads

I Upload files (txt, pdf, rdf and Google books API)

2. Data Preparation

I Data preprocessing (stopwords, stemming, metadata)

3. Data Visualization

I Word frequencies, Cluster analysis and topic modeling


Day 3

Olga Scrivner

Course Info

Charts

TextVisualization

ITMS

PreprocessingData

DataVisualization

Cluster Analysis

Topic Modeling

Google Book API

Workshop Files

I Download 3 text files

https://iu.box.com/s/knua9af3bip7g63s3zdax9ti4z243ldz

I NY Times articles (3 documents in a plain text format)

I ITMS Web site:

http://www.interactivetextminingsuite.com



http://www.interactivetextminingsuite.com


Day 3

Olga Scrivner

Course Info

Charts

TextVisualization

ITMS

PreprocessingData

DataVisualization

Cluster Analysis

Topic Modeling

Google Book API

Upload File


Day 3

Olga Scrivner

Course Info

Charts

TextVisualization

ITMS

PreprocessingData

DataVisualization

Cluster Analysis

Topic Modeling

Google Book API

Upload File


Day 3

Olga Scrivner

Course Info

Charts

TextVisualization

ITMS

PreprocessingData

DataVisualization

Cluster Analysis

Topic Modeling

Google Book API

Upload File


Day 3

Olga Scrivner

Course Info

Charts

TextVisualization

ITMS

PreprocessingData

DataVisualization

Cluster Analysis

Topic Modeling

Google Book API

Preprocessing Data

Before performing data analysis we should preprocess data.


Day 3

Olga Scrivner

Course Info

Charts

TextVisualization

ITMS

PreprocessingData

DataVisualization

Cluster Analysis

Topic Modeling

Google Book API

Preprocessing Options

Select preprocessing options and click apply.


Day 3

Olga Scrivner

Course Info

Charts

TextVisualization

ITMS

PreprocessingData

DataVisualization

Cluster Analysis

Topic Modeling

Google Book API

Stopwords

Stopwords (e.g. the, and): select Default for English


Day 3

Olga Scrivner

Course Info

Charts

TextVisualization

ITMS

PreprocessingData

DataVisualization

Cluster Analysis

Topic Modeling

Google Book API

Manual Removal of Stopwords

Based on the need, remove any additional stopwords that youmay consider a noise, e,g, paper, shows etc

Select apply


Day 3

Olga Scrivner

Course Info

Charts

TextVisualization

ITMS

PreprocessingData

DataVisualization

Cluster Analysis

Topic Modeling

Google Book API

Stemming

To improve analytics, you can stem all your tokens, ex.instead of worked, works, working, you will have only onerelevant stem work


Day 3

Olga Scrivner

Course Info

Charts

TextVisualization

ITMS

PreprocessingData

DataVisualization

Cluster Analysis

Topic Modeling

Google Book API

Metadata Extraction

You can extract or upload metadata. You will needdatestamp (year) information for chronological topicmodeling.


Day 3

Olga Scrivner

Course Info

Charts

TextVisualization

ITMS

PreprocessingData

DataVisualization

Cluster Analysis

Topic Modeling

Google Book API

Visualization


Day 3

Olga Scrivner

Course Info

Charts

TextVisualization

ITMS

PreprocessingData

DataVisualization

Cluster Analysis

Topic Modeling

Google Book API

Word Cloud Representation


Day 3

Olga Scrivner

Course Info

Charts

TextVisualization

ITMS

PreprocessingData

DataVisualization

Cluster Analysis

Topic Modeling

Google Book API

Customization


Day 3

Olga Scrivner

Course Info

Charts

TextVisualization

ITMS

PreprocessingData

DataVisualization

Cluster Analysis

Topic Modeling

Google Book API

Cluster Analysis

You need to have at least three documentsDocuments will be grouped based on their term similaritymeasures


Day 3

Olga Scrivner

Course Info

Charts

TextVisualization

ITMS

PreprocessingData

DataVisualization

Cluster Analysis

Topic Modeling

Google Book API

Cluster Analysis


Day 3

Olga Scrivner

Course Info

Charts

TextVisualization

ITMS

PreprocessingData

DataVisualization

Cluster Analysis

Topic Modeling

Google Book API

Topic Modeling

I LDA (Latent Dirichlet allocation)

I STM (Structural Topic model)

I Chronological topic visualization (lda): requiresmetadata


Day 3

Olga Scrivner

Course Info

Charts

TextVisualization

ITMS

PreprocessingData

DataVisualization

Cluster Analysis

Topic Modeling

Google Book API

Topic Modeling Tuning

I Selection of topics (how many different themes)

I Selection of words per theme (how many words pertopic)

I Selection of iteration


Day 3

Olga Scrivner

Course Info

Charts

TextVisualization

ITMS

PreprocessingData

DataVisualization

Cluster Analysis

Topic Modeling

Google Book API

Topic Model Selection


Day 3

Olga Scrivner

Course Info

Charts

TextVisualization

ITMS

PreprocessingData

DataVisualization

Cluster Analysis

Topic Modeling

Google Book API

LDA Topic Model


Day 3

Olga Scrivner

Course Info

Charts

TextVisualization

ITMS

PreprocessingData

DataVisualization

Cluster Analysis

Topic Modeling

Google Book API

STM Topic Model


Day 3

Olga Scrivner

Course Info

Charts

TextVisualization

ITMS

PreprocessingData

DataVisualization

Cluster Analysis

Topic Modeling

Google Book API

Other Formats - Google Books

Before switching to other data formats, refresh your localbrowser.

Start with File Uploads and select Structured Data


Day 3

Olga Scrivner

Course Info

Charts

TextVisualization

ITMS

PreprocessingData

DataVisualization

Cluster Analysis

Topic Modeling

Google Book API

Other Formats - Google Books

Select your search terms and submit

Current limitation is 40 books


Day 3

Olga Scrivner

Course Info

Charts

TextVisualization

ITMS

PreprocessingData

DataVisualization

Cluster Analysis

Topic Modeling

Google Book API

Resources

http://www.rdatamining.com/examples/text-mining

https://en.wikibooks.org/wiki/R_Programming/Text_Processing

http://data.library.virginia.edu/reading-pdf-files-into-r-for-text-mining/

http://www.katrinerk.com/courses/words-in-a-haystack-an-introductory-statistics-course/schedule-words-in-a-haystack/r-code-the-text-mining-package

tm package

http://www.rdatamining.com/examples/text-mining









Visual Analytics for Linguistics - Day 3 ESSLLI

Data & Analytics