Making big data small

MAKING BIG DATA SMALLBARON SCHWARTZ

STRATA CONFERENCE NYC 2013

ME@XAPRB ON TWITTER

BARON@VIVIDCORTEX.COM

LINKEDIN.COM/IN/XAPRB

Optimization, Backups, Replication, and more

Baron Schwartz, Peter Zaitsev &

Vadim Tkachenko

High PerformanceMySQL

3rd Edition

Covers Version 5.5

STORAGETRANSFER / BANDWIDTHCOMPUTE (CPU, RAM, I/O)TIME

DATA LIFECYCLE

COMPUTERETRIEVESTORETRANSFERGATHER

THE MEGABYTE HOUR WHAT DOES 350 MILLISECONDS COST?

BIG-O NOTATION

GUIDING PRINCIPLES

DO MORE WITH LESS

ADDRESS THE AUDIENCE

GENERATE KNOWLEDGE

DATA

INFORMATION

KNOWLEDGE

SUCH KNOWLEDGE IS TOO WONDERFUL FOR ME;IT IS HIGH, I CANNOT ATTAIN IT.

- PSALM 139

ANSWER THE QUESTION

DOING MORE WITH LESS

LOSSLESS COMPRESSION

LOSSY REPRESENTATION

DATA LIFECYCLE

COMPUTERETRIEVESTORETRANSFERGATHER

THE RISK?

LESS COMPUTELESS DATA

LOSSINESS FEATURES

TECHNIQUES

LESS COMPUTELESS DATA

LOSSINESS FEATURES

TECHNIQUES

DECREASE REDUNDANCY

LESS D

ATA

DELTA COMPRESSION

LESS D

ATA

NULLS AND ZEROES

LESS D

ATA

SAMPLING

LESS D

ATA

AGGREGATION

LESS D

ATA

DISCARDING DUPLICATE SIGNALS

LESS D

ATA

COMPRESSION

LESS D

ATA

LESS COMPUTELESS DATA

LOSSINESS FEATURES

TECHNIQUES

BLOOM FILTERS

LESS C

OM

PUT

E

EXPONENTIALLY WEIGHTED MOVING AVERAGES

LESS C

OM

PUT

E

0

100

200

300

400

1 2 3 4 5

0

100

200

300

400

1 2 3 4 5

APPROXIMATION

LESS C

OM

PUT

E

LESS COMPUTELESS DATA

LOSSINESS FEATURES

TECHNIQUES

MODELS

LO

SSINESS

LO

SSINESS

LO

SSINESS

LESS COMPUTELESS DATA

LOSSINESS FEATURES

TECHNIQUES

HISTOGRAMS

FEAT

UR

ES

HISTOGRAMS

FEAT

UR

ES

PERCENTILES

FEAT

UR

ES

SPARSE OUTLIER STORAGE

FEAT

UR

ES

LOST HAYSTACKS

WHAT ABOUT THE PIONEER ANOMALY?

SMALL DATA METHOD

APPLYDISCARDISOLATEEXPLOREGATHER

SAMPLES

VISUALIZE

NEEDLES!

HAYSTACK

REVIEW

SMART-FRIENDS METHOD

CONTACT / QUESTIONS@XAPRB ON TWITTER

BARON@VIVIDCORTEX.COM

LINKEDIN.COM/IN/XAPRBPLEASE

RATE THIS TALK :-)

IMAGE CREDITSHTTP://WWW.FLICKR.COM/PHOTOS/PIPER/458847242/

HTTP://WWW.FLICKR.COM/PHOTOS/MIONG/4222013704/

HTTP://WWW.FLICKR.COM/PHOTOS/BENSTEVINSON/4908627827/

HTTP://EN.WIKIPEDIA.ORG/WIKI/FILE:BLOOM_FILTER.SVG

HTTP://WWW.FLICKR.COM/PHOTOS/JINGLESLENOBEL/4915172257/

HTTP://WWW.FLICKR.COM/PHOTOS/MAUROPPP/429231880/

HTTP://WWW.FLICKR.COM/PHOTOS/ECSTATICIST/3979554514/

HTTP://WWW.FLICKR.COM/PHOTOS/TAMBAKO/9158311021/

HTTP://WWW.FLICKR.COM/PHOTOS/FUZZYLITTLEMANPEACH/4689679368/

HTTP://WWW.FLICKR.COM/PHOTOS/FWP/49514206/

HTTP://WWW.FLICKR.COM/PHOTOS/QTHOMASBOWER/3511960429/

HTTP://WWW.FLICKR.COM/PHOTOS/JIMNISTA/7239478006/

HTTP://WWW.FLICKR.COM/PHOTOS/FLOWERS-OF-THE-SEA/6971586987/

HTTP://WWW.FLICKR.COM/PHOTOS/OLDPATTERNS/5817264314/

HTTP://WWW.FLICKR.COM/PHOTOS/BREWBOOKS/184343329/

HTTP://WWW.FLICKR.COM/PHOTOS/PAREEERICA/4235666733/

HTTP://WWW.FLICKR.COM/PHOTOS/JDHANCOCK/5177358991/

HTTP://WWW.FLICKR.COM/PHOTOS/ROSEMARY/47396922/

HTTP://WWW.FLICKR.COM/PHOTOS/ECSTATICIST/1337749333/

HTTP://WWW.FLICKR.COM/PHOTOS/PINKSHERBET/233228813/

HTTP://BOST.OCKS.ORG/MIKE/SIMPLIFY/

HTTP://WWW.FLICKR.COM/PHOTOS/PSOUP216/7540279198/

HTTPS://GITHUB.COM/BRENDANGREGG/HEATMAP/WIKI

HTTP://WWW.FLICKR.COM/PHOTOS/WALKINGSF/6221430560/

HTTP://WWW.FLICKR.COM/PHOTOS/WALKINGSF/7835295762/

HTTP://WWW.FLICKR.COM/PHOTOS/NASAMARSHALL/8615696355/

HTTP://WWW.FLICKR.COM/PHOTOS/BOOLEANSPLIT/8482641188/

HTTP://WWW.FLICKR.COM/PHOTOS/HTAKASHI/6861028238/

HTTP://WWW.FLICKR.COM/PHOTOS/ZDETO/4383899069/

JPL VIA HTTP://WWW.PLANETARY.ORG/EXPLORE/PROJECTS/PIONEER-ANOMALY-1/PIONEER_ANOMALY_STAIRS_DATA_TAPES_3.HTML

HTTP://EN.WIKIPEDIA.ORG/WIKI/FILE:PIONEER_10-11_SPACECRAFT.JPG