On the role of Interactivity and Data Placement in Big Data Analytics Srini Parthasarathy OSU.

On the role of Interactivity and Data Placement in Big Data Analytics

Srini ParthasarathyOSU

The Data Deluge: Data Data Everywhere

22

600$ to buy a disk drive that can store all of the

world’s music

3

[McKinsey Global Institute Special Report, June ’11]

Data Storage is Cheap

Data does not exist in isolation.

4

Data almost always exists in connection with other data – integral

part of the value proposition.

5

6

Social networks Protein Interactions Internet

VLSI networks Data dependenciesNeighborhood graphs

7

Big Data Problem: All this data is only useful if we can scalably extract useful knowledge from such complex data

THIS TALK

• THE ROLE OF DATA PLACEMENT IN BIG DATA SYSTEMS

• THE ROLE OF VISUALIZATION AND INTERACTION IN BIG DATA ANALYSIS

GLOBAL GRAPHS

GLOBAL GRAPHS

• What? – System for deploying applications processing complex data

• Why? – Seeks balance between high productivity and high performance

• How?– Built on top of PNL’s GlobalArrays– Trees (GlobalTrees, GlobalForests)– Relational Arrays (ArrayDB-GA)– Graphs (GlobalGraphs)

• Data Placement is key to high performance

Importance of Data Placement

• Locality– Placing related items close to each other so they may be

processed together

• Mitigating Impact of Data Skew– Reducing load imbalance in a parallel setting– Reducing variance in partition samples

• Generating Stratified Samples– Improving interactive performance

Key Ideas

• Pivotization– Convert data with complex structure into sets– Each element of set captures features of local topology

• Hashing into Strata: Hash related sets into similar bins– Can employ a sketch-clustering algorithm

• Partitioning: Place Strata into partitions for• Locality • Mitigating Data Skew• Samples

SK

ETCH

SORT

or S

KETC

HCL

UST

ER

S-1 : : S-4(Δ1, SK-1)(Δ5, SK-5)(Δ12,SK-12)(Δ25,SK-25) : : :

S-5 : : : S-128 : : :

PART

ITIO

NIN

G &

REP

LICA

TIO

N

P-1 : P-2 S-4 S-7 S-8 S-12 : S-128

P-3 : : : P-8 S-3 S-4 S-9S-12 : S-127

PIVO

T

T

RAN

SFO

RMAT

ION

S

A

B C

LE

A

B C

LE F

.

.

.

.

Δ1

Δ25

DATA (Δ)

A

B C

A

F C

A

E C

A

F L

B

E F

A

E L

A

B L

A

B C

A

E CA

E L

A

B L

.

.

.

.

(PS-1)

(PS-25)

PIVOT SETS (PS)

MIN

WIS

E H

ASH

ING

on

PIVO

T SE

TS

{1050, 2020,3130,1800} (SK-1)

{1050, 2020,7225, 2020} (SK-25)

.

.

.

.

.

.SKETCHES(SK) Strata (S)

Frequent Tree Mining

• Our proposed approaches shows 100X gains

WebGraph Compression

• Linear Scaleup with no loss in compression ratio

PRISM-HD -

PRobing the Intrinsic Structure and Makeup of High-dimensional Data

HD

Visualization and Interactivity are key to discovery

17

PRISM-HD• What?

– A novel mechanism for exploring complex data

• Why?– User is often overwhelmed with

characteristics of data– Befuddled on where to start

• How?– Given, similarity measure-of-interest– Compute similarity graph at threshold (t)

• Key: Graphs are dimensionless

– Provide user graph visualization cues• User determines next threshold and

repeats

HD

HD

HIGH THRESHOLD MODERATE THRESHOLD LOW THRESHOLD

Benefits of Knowledge CachingHD

Benefits of Incremental Processing on Twitter

Incremental estimates on Twitter t1 = 0.95

HD

PRISM-HD and Global Graphs in Context:Leveraging Social Media in Emergency Response

HD

Concluding Remarks

• Data is everywhere• Data is fraught with complexities

– Dimensionality, dynamics, structure, massive…• Both data placement and data interactivity

have an important role to play in big data analytics– PRISM-HD and GlobalGraphs can help!

HD

Thanks for your attentionContact: [email protected]

Mining Simulation Data

Medical Image Analysis

Protein Interaction Network (yeast)

Acknowledgements: Various NSF, NIH, DOE and industry grants