Using Task Queues and D3.js to build an analytics product on App Engine

Using Task Queues and D3.js to build an

analytics product on App Engine

Warren EdwardsFounder, Waizee

TODAYHow Do We Handle All The Data?!?

Look at the Product

Why App Engine? Why D3.js?

Task Queues in App Engine

Code Samples

A Quiz!

Wrap Up

All of the data stored in the world currently about 4 zettabytes according to EMC.

At current growth rates, that data will surpass a yottabyte in 2030.

If each bit was represented by a grain of sand, a yottabyte would be about five percent the mass of the Moon.

Data surpasses human readability.

Software analyzes the data so humans

can focus on [ interpretation,

judgement, action ];

[ Show Product Demo ]

AWS AE Others

Team Experience

Automated scaling / spin-up

Automated load-balancing

Automated security

Task queue functionality

Why App Engine?

Honorable mention: AppScale path from AE

Google App Engine builds on Big Table

Why D3.js?Canvas Directly D3 PolyChart Math

plotlibSuper-

conductor

Team experience ( NONE )

Development effort

Maturity of library

Developer traction

Honorable mention: xCharts builds on D3.js with prepackaged chartsPHOTO CREDIT: http://www.flickr.com/photos/cusegoyle/

HTTP Request Task Queue

Full access to data store? YES YES

Atomic transactions? YES YES

Write in Python / Java / Go? YES YES

Maximum request lifetime 30-60 sec Up to 10 min

Timing of execution Immediate Up to 30 days

Retry Can be done manually, no policy

Automatic by policy

Concurrent requests No set limit Limited by policy

TQ build on App Engine's HTTP Requestbut liberate server use from user interaction

Introducing Task Queues (TQ)

Appeal of Task Queues

Well suited to number crunching● Long-lived jobs for running analytics● Full access to data store and messaging to build

on your existing App Engine knowledge● Save state back to Data Store and call the next

task

Task Queues allow you to crunch data "while you wait"

USERUPLOADS

DATATask#1

Task#2

. . . Task#N

RESULT

Save to Data Store

Pass Parameters

Cascade Tasks in Queue for Multipass Processing

Save to Data Store

Program Flow

Data Flow

DATA

STORE

Task#1

Task#2

. . .

Task#N

USERUPLOADS

DATARESULT

Process Tasks in Parallel Queues

Program Flow

Data Flow

Let's Look at a Code Sample

taskqueue.add(queue_name='analyze', url='/work1', params={'key': keyID})

class WorkerTheFirst(webapp2.RequestHandler): def post(self): keyID = self.request.get('key')

app = webapp2.WSGIApplication([('/', StartPage), ('/work1', WorkerTheFirst), ('.*', ErrPage) ])

Add Task into Queue

Define Task with Retrieval of Parameters

Associate Task with Handler

One Task Calls Anotherclass WorkerTheFirst(webapp2.RequestHandler): def post(self): keyID = self.request.get('key') # # First pass of work here # taskqueue.add(queue_name='analyze', url='/work2', params={'key': keyID})

class WorkerTheSecond(webapp2.RequestHandler): def post(self): keyID = self.request.get('key') # # Second pass of work here #

app = webapp2.WSGIApplication([('/', StartPage), ('/work1', WorkerTheFirst), ('/work2', WorkerTheSecond), ('.*', ErrPage) ])

total_storage_limit: 120G # Max for free apps is 500M

queue:# Queue for analyzing the incoming data- name: analyze rate: 35/s retry_parameters: task_retry_limit: 5 task_age_limit: 2h

# Queue for user behavior heuristics- name: heuristic rate: 5/s # Queue for doing maintenance on the data store or site- name: kickoff rate: 5/s

Setting up Task Queues in queue.yaml

svg = d3.select('body') .append('svg') .attr('class', 'circles') .attr('width', width) .attr('height', height)

svg.append('g').selectAll('circle') .data(data) .enter() .append('circle') .attr('transform', 'translate(' + pan + ', 0)') # pan allows moving te whole graph slightly # to account for long text labels

svg.selectAll('circle') .attr('cx', function(d) {return x(d.v2)}) .attr('cy', function(d) {return y(d.v1)}) .attr('r', dot_out) # dot_out scales the size .attr('fill', function(d) {return d.v4})

Core of the D3 Code

svg = d3.select('body') .append('svg') .attr('class', 'circles') .attr('width', width) .attr('height', height)

svg.append('g').selectAll('circle') .data(data) .enter() .append('circle') .attr('transform', 'translate(' + pan + ', 0)') # pan allows moving te whole graph slightly # to account for long text labels

svg.selectAll('circle') .attr('cx', function(d) {return x(d.v2)}) .attr('cy', function(d) {return y(d.v1)}) .attr('r', dot_out) # dot_out scales the size .attr('fill', function(d) {return d.v4})

Program Writes Its Own Code !

Axes are set heuristically by server software

Titles Are Set By Heuristic Text Analysis

var title = '{{ pagetitle }}'var subtitle = '{{ pagesubtitle }}'

var title = 'Google+ Rating More Important Metric Than Star Rating'var subtitle = 'Survey of Productivity Apps in the Chrome Web Store, Nov 2012'

In Django template

Rendered in Javascript to the browser

Labels were pulled Heuristically from input - not hard coded!

QUIZ !

Choose the Correct Task Queue Call

taskqueue.add(queue_name='analyze', url='/work2', param={key: keyID})

queue.add(queue='analyze', url='/work2', params={'key': keyID})

taskqueue.add(queue='analyze', url='/work2', param={'key': keyID})

taskqueue.add(queue_name='analyze', url='/work2', params={'key': keyID})

taskqueue.add(queue='analyze', url='/work2', params={key: keyID})

A

B

C

D

E

ANSWER IS ...

Choose the Correct Task Queue Call

taskqueue.add(queue_name='analyze', url='/work2', param={key: keyID})

queue.add(queue='analyze', url='/work2', params={'key': keyID})

taskqueue.add(queue='analyze', url='/work2', param={'key': keyID})

taskqueue.add(queue_name='analyze', url='/work2', params={'key': keyID})

taskqueue.add(queue='analyze', url='/work2', params={key: keyID})

A

B

C

D

E

Correct Answer is D

TQ Open a World of Possibilities

You can send tasks to different versions of your app → Automated test of new version of app before Go Live

You can access the queue’s usage data→ Your app can monitor its own consumption of tasks through QueueStatistics class

Task Queue + Crowdsource = ???→ Software application instructing humans !

Task Queues allow "while you wait" processing● Allow server task to run autonomously● Cascade tasks for multistep processing● Flexible functionality to create great products

Task Queues provide a great tool for automating the understanding of data

D3.js offers flexible, stable tool for viz of data● Works nicely with automated scripting● Lush visualizations but not pre-packaged● Leverage huge traction in San Francisco

D3.js is best platform for visualization using automated processing of data

Questions?

Do you have a passion for analytics? Let’s talk!

warren@waizee.com@campbellwarren

We are your number cruncher in the cloud that understands your data and shows you only what is

most important.

Data SourcesThe Digital Universe in 2020: Big Data, Bigger Digital Shadows, and Biggest Growth in the Far East. IDC sponsored by EMC. December 2012

diameter and volume of the Moon: Wolfram Alpha

Yottabyte representation: Waizee calculations

2013 Gartner Magic Quadrant for Business Intelligence and Analytics Platforms

Photo Creditshttp://www.flickr.com/photos/pcoin/2066230078/sizes/m/in/photostream/

http://en.wikipedia.org/wiki/File:Visicalc.png

http://www.todayscampus.com/rememberthis/load.aspx?art=348

http://commons.wikimedia.org/wiki/File:Chocolate_chip_cookie.jpg

http://www.wpclipart.com/signs_symbol/checkmarks/checkmarks_3/checkmark_Bold_Brush_Green.png.html

http://www.flickr.com/photos/cusegoyle/