Social Computing for Linguistics & Linguistics for Social Computing

evolution

lexica

lear

ningword

NLP

model

node

network

syntax

POS

@

complex

semanti

edge

bangla

PA

DD

zulu

Social Computing for Linguistics & Linguistics for Social Computing

Monojit ChoudhuryMicrosoft Research [email protected]

evolution

lexica

learni

ngword

NLP

model

node

networ

k

syntax

POS

@

complex

semanti

edge

bangla

PA

DD

zulu

Language is shaped by social interaction patterns;emails, blogs, sms, queries – CMC’sLinguistics for SoC

SoC for LinguisticsSoC platforms for linguistic experiments; SoC Models

for linguistic theories

evolution

lexica

learni

ngword

NLP

model

node

networ

k

syntax

POS

@

complex

semanti

edge

bangla

PA

DD

zulu

Language is shaped by social interaction patterns;emails, blogs, sms, queries – CMC’sLinguistics for SoC

SoC for LinguisticsSoC platforms for linguistic experiments; SoC Models

for linguistic theories

Web Search Queries as an Emerging

Language

evolution

lexica

learni

ngword

NLP

model

node

networ

k

syntax

POS

@

complex

semanti

edge

bangla

PA

DD

zulu

Language is shaped by social interaction patterns;emails, blogs, sms, queries – CMC’sLinguistics for SoC

SoC for LinguisticsSoC platforms for linguistic experiments; SoC Models

for linguistic theories

2000 2006 20100

1

2

3

4

5

2.21

3.54

Mean length of queries

Web Search Queries as an Emerging

Language

evolution

lexica

learni

ngword

NLP

model

node

networ

k

syntax

POS

@

complex

semanti

edge

bangla

PA

DD

zulu

Language is shaped by social interaction patterns;emails, blogs, sms, queries – CMC’sLinguistics for SoC

SoC for LinguisticsSoC platforms for linguistic experiments; SoC Models

for linguistic theories

2000 2006 20100

1

2

3

4

5

2.21

3.54

Mean length of queries

Web Search Queries as an Emerging

Language

6

evolution

lexica

learni

ngword

NLP

model

node

networ

k

syntax

POS

@

complex

semanti

edge

bangla

PA

DD

zulu

Language is shaped by social interaction patterns;emails, blogs, sms, queries – CMC’sLinguistics for SoC

SoC for LinguisticsSoC platforms for linguistic experiments; SoC Models

for linguistic theories

Web Search Queries as an Emerging

LanguageUnsupervised segmentation

Distributional PatternsTaxonomy of Function units

Word co-occurrence NetworkQueries are difficult to interpret

User Behavior Studies

Basic units

Search expertise language acquisition

7

evolution

lexica

learni

ngword

NLP

model

node

networ

k

syntax

POS

@

complex

semanti

edge

bangla

PA

DD

zulu

SoC for LinguisticsSoC platforms for linguistic experiments; SoC Models

for linguistic theories

Web Search Queries as an Emerging

Language

Web Search Queries are an evolving Protolanguage

Collaborators: Rishiraj Saha Roy & Niloy Ganguly (IIT Kharagpur), Srivatsan Laxman & Kalika Bali (MSR India)

8

evolution

lexica

learni

ngword

NLP

model

node

networ

k

syntax

POS

@

complex

semanti

edge

bangla

PA

DD

zulu

Language is shaped by social interaction patterns;emails, blogs, sms, queries – CMC’sLinguistics for SoC

SoC for LinguisticsSoC platforms for linguistic experiments; SoC Models

for linguistic theories

Use of Indian languages on online

social media

Code mixing

Transliteration

Spelling Change

Indian English

Collaborators: Kalika Bali & Nimmi Rangaswamy (MSR India)

9

evolution

lexica

learni

ngword

NLP

model

node

networ

k

syntax

POS

@

complex

semanti

edge

bangla

PA

DD

zulu

Language is shaped by social interaction patterns;emails, blogs, sms, queries – CMC’sLinguistics for SoC

SoC for LinguisticsSoC platforms for linguistic experiments; SoC Models

for linguistic theories

Sociolinguistic ExperimentsOSN Games with a

Purpose

CodeGambler is an OSN Game to study the population-level emergence of categorization of a continuous space (colors) into discrete category terms (color names)

Collaborators: Animesh Mukherjee (IIT Kgp), Vittorio Loreto, (University of Rome)

10

evolution

lexica

learni

ngword

NLP

model

node

networ

k

syntax

POS

@

complex

semanti

edge

bangla

PA

DD

zulu

Language is shaped by social interaction patterns;emails, blogs, sms, queries – CMC’sLinguistics for SoC

SoC for LinguisticsSoC platforms for linguistic experiments; SoC Models

for linguistic theories

Sociolinguistic ExperimentsCrowdsourcing for

data

Flat segmentation (queries and sentences)

Nested segmentation

11

evolution

lexica

learni

ngword

NLP

model

node

networ

k

syntax

POS

@

complex

semanti

edge

bangla

PA

DD

zulu

Language is shaped by social interaction patterns;emails, blogs, sms, queries – CMC’sLinguistics for SoC

SoC for LinguisticsSoC platforms for linguistic experiments; SoC Models

for linguistic theories

SoC inspired Linguistic TheoryWord Co-occurrence

Network

Small world

word

language

in

human

treat

as

is

can

evolving

neighbori

ng

distinct

interacting

web

sentences

such

structur

e

acomplex

network

Two-regime power law degree distribution

Kernel-Periphery structure

Low rank

Collaborators: Animesh, Niloy (IIT Kgp), Chris Biemann (University of Darmstadt), Ravi Kannan (MSR India)

12

evolution

lexica

learni

ngword

NLP

model

node

networ

k

syntax

POS

@

complex

semanti

edge

bangla

PA

DD

zulu

Language is shaped by social interaction patterns;emails, blogs, sms, queries – CMC’sLinguistics for SoC

SoC inspired Linguistic TheoryWord Co-occurrence

NetworkCollaborators: Animesh, Niloy (IIT Kgp), Chris Biemann (University of Darmstadt), Ravi Kannan (MSR India)

WCN of Bollywood Lyrics is a very small world with a tiny kernel.

Learn 1000 Hindi words and you will understand every Bollywood song!

13

evolution

lexica

learni

ngword

NLP

model

node

networ

k

syntax

POS

@

complex

semanti

edge

bangla

PA

DD

zulu

Language is shaped by social interaction patterns;emails, blogs, sms, queries – CMC’sLinguistics for SoC

SoC inspired Linguistic TheoryWord Co-occurrence

NetworkCollaborators: Animesh, Niloy (IIT Kgp), Chris Biemann (University of Darmstadt), Ravi Kannan (MSR India)

WCN of Web search queries is not (yet) small world!

It has a tiny kernel but very large periphery

Thank You!http://research.microsoft.com/people/monojitc/

evolution

lexica

lear

ningword

NLP

model

node

network

syntax

POS

@

complex

semanti

edge

bangla

PA

DD

zulu