Context from Big Data

Context from Big DataStartup Showcase

IEEE Big Data ConferenceNovember 1, 2015

Santa Clara, CA

Delroy Cameron, Data Scientist

@urxtech | urx.com | [email protected]

PeopleURX has 40 people: 75%

product/eng, 25% business

CustomersURX partners with the world’s top publisher & advertisers.

FundingURX raised $15M from Accel, Google Ventures, and others

Who is URX?

URX is a mobile technology platform that focuses on publisher monetization, content distribution, and user engagement.

What problem does URX solve?

URX serves contextually relevant native ads.

URX interprets page context to dynamically determine the best message & action.

How does URX affect the mobile ecosystem?

Volume (Apps)Volume (web

pages)Variety (entities)

Why is this a Big Data problem?

Rhapsody(Music)

Fansided

(Sports)

Apple(Music, TV, Books)

Source: The Statistics Portal - http://www.statista.com/statistics/276623/number-of-apps-available-in-leading-app-stores/

1.6M Apps (Android)1.5M Apps (Apple Store)

How do we collect, store, and process the data needed to build our machine learning models?

1.Data Collection and Parsing2.Data Storage

• Persistent Storage• Search Index

3.Data Processing• Dictionary Building • Vectorization (Feature Vector Creation)

Important tasks

11GB XML dump (gzip file)15M pages (but only 4M articles) Wikitext Grammar

Wikipedia Corpus (English)

1. Data collection & parsing

https://dumps.wikimedia.org/enwiki/latest/

<page> <title>AccessibleComputing</title> <ns>0</ns> <id>10</id> <redirect title="Computer accessibility"/> <revision> <id>631144794</id> <parentid>381202555</parentid> <timestamp>2014-10-26T04:50:23Z</timestamp> <contributor> <username>Paine Ellsworth</username> <id>9092818</id> </contributor> <comment>add [[WP:RCAT|rcat]]s</comment> <model>wikitext</model> <format>text/x-wiki</format> <text xml:space=“preserve">

#REDIRECT [[Computer accessibility]] {{Redr|move|from CamelCase|up}}</text> <sha1>4ro7vvppa5kmm0o1egfjztzcwd0vabw</sha1> </revision> </page>

1. Data collection & parsing

https://dumps.wikimedia.org/enwiki/latest/

1. Data collection & parsing

sax library, generator20 secs/doc, 10 years

FullWikiParser (mediawikiparser)

sax library, generator200 docs/sec, ~ 21 hours

FastWikiParser (mwparserfromhell)

hbase, lxml parser6 docs/sec, ~ one month

HTMLWikiParser (URX Index)

multithreading, generator~ 3 hours

GensimWikiCorpusParser

1. pyspark (64 cores, 8GB RAM)2. wikihadoop

(StreamWikiDumpInputFormat)• split input file

3. mwparserfromhell• parse to raw text

4. ~20 minutes

wikipedia-parser

wik

iped

ia-in

dexe

r

datanode 1

Namenode

datanode 2

datanode n

.

.

.

HDFS Elasticsearch Index

ClusterNode1

ClusterNode 2

ClusterNode m

.

.

.

2. Data storage

wik

iped

ia-p

arse

r

(0 taylor) . . . (1999995 zion)

(1 alison) . . . (1999996 dozer)

(2 swift) . . . (1999997 tank)

(3 born) . . . (1999998 trinity)

(4 december) . . . (1999999 neo)

3. Data Processor (Dictionary building)

wikihadoop, StreamWikiDumpInputFormatdictionary, tfidfmodel~ 1 hour

Pyspark (Gensim)

multithreading, generatorcorpus, dictionary, tfidfmodel~ 6 hours

GensimWikiCorpusParser

Alias Candidate Entity f1 f2 … fn

Taylor Swift wikipedia:Taylor_Swift 0.91 0.81 … 0.34

wikipedia:Taylor_Swift_(album) 0.42 0.10 … 0.42

wikipedia:1989_(Taylor_Swift_album) 0.71 0.23 … 0.31

wikipedia:Fearless_(Taylor_Swift_song) 0.13 0.22 … 0.23

wikipedia:John_Swift 0.00 0.19 … 0.56

4. Data Processor (Vectorization)

~ 350ms predict entity per alias

Gensim

~ 100ms predict entity per alias

Cython

WikipediaCorpus

corpus-parser

corpus-indexer

HDFS(Wikilinks)

WikilinksCorpus

XCorpus

Data Processor Dictionary TF-IDF Model

Machine Learning Module

HDFS (Wikipedia)

HDFS(X Corpus)

Elasticsearch1

Elasticsearch2

Elasticsearchn

1

2

3

4

56

7

Demo

Linked Entities1. http://en.wikipedia.org/wiki/Macgyver2. http://en.wikipedia.org/wiki/Neil_deGrasse_Tyson3. http://en.wikipedia.org/wiki/Richard_Dean_Anderson4. http://en.wikipedia.org/wiki/Josh_Holloway5. http://en.wikipedia.org/wiki/NBC6. http://en.wikipedia.org/wiki/CBS7. http://en.wikipedia.org/wiki/James_Wan8. http://en.wikipedia.org/wiki/Netflix9. http://en.wikipedia.org/wiki/America_America

http://zap2it.com/2015/10/5-reasons-cbs-macgyver-reboot-isnt-the-worst-idea-ever/

● Tuning pyspark jobs (64 cores, 8GB Driver RAM)

● Bringing down the elasticsearch cluster

● Rejoining the union after secession (elasticsearch

nodes)

● Text Cleaning (lowercasing, character encoding)

● Merging in Hadoop for dictionary creation

Things to watch out for

Getting started is easy.

Sign Up Download SDK Start Building

Visit http://urx.com/sign-up for more information.

Thank [email protected]