Uncovering concepts and similarities at scale

Theodore Vasiloudis, @thvasiloSwedish Institute of Computer Science (SICS)

Uncovering concepts and similarities at scale

Similarity

Objects

Objects

Users, words, artists, movies, …

Objects and correlations

Friendships, co-occurrences, transitions, …

Similarity?

You shall know a word by the company it keeps.

- J. R. Firth, 1957

You shall know a word by the company it keeps.

an object

Object with company

Object with company

Object in context

Similar objects have similar contexts


Similar objects are exchangeable


Similar objects are exchangeable

How can we discover such objects?

Calculating similarities

Similar objects are one neighbor apart


Compare edges to shared neighbors


Transform to two-hop graph


Transform to two-hop graph


Correlation graph


Correlation graph Similarity graph


edges // (i, j, correlation)pairs = edges.join(edges).on(j) // ((i, j), (corr_ik, corr_jk))similarities = pairs.mapValues((corr_ik, corr_jk)) => f(corr_ik, corr_ij))

.reduceByKey(_ + _)

First example: Natural Language

Second example: Music

Third example: Molecular biology

Discovering concepts


Concept = group of inter-similar objects


Cluster objects in the similarity graph



Cluster objects in the similarity graph

Algorithm based on SLPA by Xie et al.


Example concepts

Fine-grain artist concepts

Use for recommendation


Graph structure gives us unique possibilities

2 Dope Kgp

Soopa Villainz

Natas

Claas

Mastamind

Marz

D. Snipes

Lo-Key

Mcnastee

Jamie Madrox

Snipes

Heavy Hittaz

Castro The Savage

Astray

Violent J

Proof Halfbreed

Rukus

Level Jumpers

Mc Breed

Doomsday Productions

TwzitdProject Born

Q-StrangeBoondox

Blaze Ya Dead Homie

House Of Krazees

Obie Trice

Psychopathic Rydas

Dark Lotus

Monoxide

Royce Da 5'9"V-Sinizter

Detroit Warriors

Native Funk

Joe Bruce

The R.O.C.

Psychopathic Family

Bedlam

Shoestring

Bootleg

Mc Supernatural Ft. DilatedPe

Gza/Genius / Rza & Gza

Diabolic Diabolic Of Triple Optics

Awol One & Factor

Mr. Lif

Rza

Masta Ace

Big L

Az

CappadonnaBlack Star

Ghostface Killah

Raekwon

Nas

Method Man

Jay-Z

Gang StarrThe Notorious B.I.G.

Mobb Deep

Wu-Tang Clan

The Game

Gza/Genius Q-Unique

Jc001

Sick Since

Non PhixionJean Grae

Remo'Conscious

Ill BillGravediggaz

Akir

Immortal TechniqueSabac

Method Man & Redman

Killah Priest

Classic Wu-TangInstrumentals

Dj Muggs Vs. Gza/Genius

U-God

Ol' Dirty Bastard

Intricate Minds

Ice-T

Abk

Jumpsteady

Twiztid

Masta Killa

Inspectah Deck

Wordsworth

Jeru The Damaja

D.I.T.C.

Jedi Mind Tricks

Shyheim

Lloyd Banks

Rakim

Afu-Ra

Army Of The Pharaohs


Use case: Graph analytics


Use case: Walk the graph





Scalability

Locality: Similar objects are close in the graph

Sparseness: Most objects are unrelated

Scalability

Locality: Similar objects are close in the graph

Sparseness: Most objects are unrelated

Scalability

Scalability

Locality + Sparseness = Scalability

Billion words on laptop Google books on Amazon EC2

Scalability

Locality + Sparseness = Scalability

● Similarity calculation is done in one iteration● Runtimes in the order of minutes vs. hours/days for word2vec and collaborative

filtering.● Unsupervised learning method, ideal for implicit usage data.

Future work

Apply

Future work

Apply

as recommendation system

Future work

Apply

as recommendation system

to find higher-order dynamics …

Dark Lotus

ProofProject Born

Q-StrangeMc Breed

Obie Trice

Bootleg

Joe Bruce

Shoestring

The R.O.C.Bedlam

Psychopathic Family

Castro The Savage

Jamie Madrox

Lo-Key

Snipes

Astray

Heavy Hittaz

Mcnastee

D. Snipes

Violent JTwzitd


BoondoxRukus

Level Jumpers

Halfbreed

Psychopathic Rydas

Royce Da 5'9"

Detroit WarriorsMonoxide

Native Funk

V-SinizterBlaze Ya Dead Homie

House Of Krazees

Soopa Villainz

Natas

Kgp

Claas

Mastamind

Marz

2 Dope

Abk

Remo'ConsciousJc001

JumpsteadyIll Bill

Jay-Z

The Game

Wu-Tang Clan

Gza/Genius The Notorious B.I.G.

Mobb Deep

Gang Starr


Awol One & Factor



Mr. Lif

GravediggazBlack Star Nas

Ghostface Killah

RaekwonCappadonna


U-GodJeru The Damaja


Shyheim

WordsworthKillah Priest

Jean Grae Ice-T

Akir

Non Phixion

SabacSick Since Intricate Minds

Twiztid

Immortal Technique

Q-Unique

Lloyd Banks


Az

Big L

Rakim

Afu-Ra

Masta Ace

D.I.T.C.

Jedi Mind TricksRza

Inspectah Deck

Masta Killa

Ol' Dirty Bastard

Method Man

Method Man & Redman

Dark Lotus

ProofProject Born

Q-StrangeMc Breed

Obie Trice

Bootleg

Joe Bruce

Shoestring

The R.O.C.Bedlam

Psychopathic Family

Castro The Savage

Jamie Madrox

Lo-Key

Snipes

Astray

Heavy Hittaz

Mcnastee

D. Snipes

Violent JTwzitd


BoondoxRukus

Level Jumpers

Halfbreed

Psychopathic Rydas

Royce Da 5'9"

Detroit WarriorsMonoxide

Native Funk

V-SinizterBlaze Ya Dead Homie

House Of Krazees

Soopa Villainz

Natas

Kgp

Claas

Mastamind

Marz

2 Dope

Abk

Remo'ConsciousJc001

JumpsteadyIll Bill

Jay-Z

The Game

Wu-Tang Clan

Gza/Genius The Notorious B.I.G.

Mobb Deep

Gang Starr


Awol One & Factor



Mr. Lif

GravediggazBlack Star Nas

Ghostface Killah

RaekwonCappadonna


U-GodJeru The Damaja


Shyheim

WordsworthKillah Priest

Jean Grae Ice-T

Akir

Non Phixion

SabacSick Since Intricate Minds

Twiztid

Immortal Technique

Q-Unique

Lloyd Banks


Az

Big L

Rakim

Afu-Ra

Masta Ace

D.I.T.C.

Jedi Mind TricksRza

Inspectah Deck

Masta Killa

Ol' Dirty Bastard

Method Man

Method Man & Redman

Node 8

Node 3

Node 10Node 1

Node 9

Node 5

Node 11

Node 6

Node 7Node 4

Thank you!

@[email protected]

Code: github.com/sics-dna/concepts

Similarity calculation

Similarity calculation

Binary correlation example


Similarity:


Similarity:

i

j


Similarity:

σi,j = 4/5

i

j

Uncovering concepts and similarities at scale

Data & Analytics