Predictions and other adventures - Jordi Torres · What, How and Why BigML Predictions and other adventures. observations theory predictions. problem conjecture explanation observations

What, How and Why BigML

Predictions and other adventures

observations

theory

predictions

problem

conjecture

explanation

observations

predictions

models

observations

theory

predictions

datasets

datasources

predictions

Large DatasetsLarge Datasets

ParallelismParallelism

AnytimeAnytime

DATA STREAMING

Large DatasetsLarge Datasets

ParallelismParallelism

AnytimeAnytime

Streaming Streaming HistogramsHistograms

Streaming Streaming TreesTrees

Accuracy = (TP + TN) / (TP + TN + FP + FN)

Recall = TP / (TP + FN)

Precision = TP / (TP + FP)

F1 = 2 Precison Recall / (Precision + Recall)

I program, therefore I am ...

REST

POSTGETDELETEPUT

application/json

REST+

queued finishedin-progress

REST-

queued errorin-progress

nginxnginx nginxnginx

apianapian apianapian apianapian

API service

nginxnginx

RESTish JSON over HTTPS

. . .. . .

nginxnginx nginxnginx

apianapian apianapian apianapian

nginxnginx

RESTish JSON over HTTPS

. . .. . .

nginxnginx nginxnginx

apianapian apianapian apianapian

nginxnginx

RESTish JSON over HTTPS

. . .. . .

pymongo

pymongo

nginxnginx nginxnginx

apianapian apianapian apianapian

nginxnginx

RESTish JSON over HTTPS

. . .. . .

pymongo

pymongo

Front-end webapp

RESTish JSON over HTTPS

nginxnginx nginxnginx

skysky skysky skysky

nginxnginx

HTTPS + AJAX/JSON

. . .. . .

Browser

curious yellow

Backend

RESTish JSON over HTTP

wintermutewintermute

. . .

curious yellow(front-end)

apian(middle-end, API)

wintermute(back-end)

apian(middle-end, API)

apian(middle-end, API)

apian(middle-end, API)

NoNoSQL?

wintermute wintermute

transactional eventual consistency

wintermute

eventual consistency

“It's better to ask for

forgivenessthan for

permission.”

“Better safethan sorry.”

wintermute(back-end)

storm

Not quite real-time

Not quite map-reduce-able

Too rigid

Too complex

HTTP/JSONHTTPCache-Control: max-age=300, must-revalidateLast-Modified: Tue, 2 Oct 2012 11:00:00 GMT

store

schema freeexternal JSON schema

The datasets saga

a b c d e f

columns: { a: [ ] b: [ ] c: [ ] d: [ ] e: [ ] f: [ ]}

MR

store