Building Integrated Applications on Google's Cloud Technologies

Building Integrated Applications on Google's Cloud TechnologiesDevFest 2011Jakarta

Christian SchalkGoogle Developer Advocateprofiles.google.com/cschalk@cschalk

Agenda

● Introduction

● App Engine Recap

● Google's new Cloud Technologies○ Google Cloud Storage○ Prediction API○ BigQuery○ Google Cloud SQL

● Summary Q&A

Introduction

Google App Engine

Introduction

Google BigQuery Google Prediction API

Google Cloud Storage

Google App Engine

Google Cloud SQL

New Google Cloud Technologies

● Google Cloud Storage○ Store your data in Google's cloud

● Prediction API○ Google's machine learning tech in an API

● BigQuery○ Hi-speed data analysis on a massive scale

● Cloud SQL○ A relational database in the cloud

Google Cloud StorageStore your data in Google's cloud

What Is Google Cloud Storage?

● Store your data in Google's cloud○ any format, any amount, any time

● You control access to your data○ private, shared, or public

● Access via Google APIs or 3rd party tools/libraries

Sample Use Cases

Static content hostinge.g. static html, images, music, video Backup and recoverye.g. personal data, business records Sharinge.g. share data with your customers Data storage for applicationse.g. used as storage backend for Android, AppEngine, Cloud based apps Storage for Computatione.g. BigQuery, Prediction API

Google Cloud Storage Benefits

High Performance and Scalability Backed by Google infrastructure

Strong Security and Privacy Control access to your data

Easy to UseGet started fast with Google & 3rd party tools

Google Cloud Storage Technical Details● RESTful API

○ Verbs: GET, PUT, POST, HEAD, DELETE ○ Resources: identified by URI○ Compatible with S3

● Buckets ○ Flat containers

● Objects ○ Any type○ Size: 100 GB / object

● Access Control for Google Accounts ○ For individuals and groups

● Two Ways to Authenticate Requests ○ Sign request using access keys ○ Web browser login

Early Google Cloud Storage Adopters

and more...

Simple Google Cloud Storage Demos

GSutil

GS (Web) Manager

Google Prediction APIGoogle's prediction engine in the cloud

Google Prediction API as a simple example

Predicts outcomes based on 'learned' patterns

Potentially endless number of applications...

CustomerSentiment

TransactionRisk

SpeciesIdentification

MessageRouting

Legal DocketClassification

SuspiciousActivity

Work RosterAssignment

RecommendProducts

PoliticalBias

UpliftMarketing

EmailFiltering

Diagnostics

InappropriateContent

CareerCounselling

ChurnPrediction

... and more ...

Prediction API Capabilities

Data● Input Features: numeric or unstructured text● Output: up to hundreds of discrete categories

Training● Many machine learning techniques● Automatically selected ● Performed asynchronously

Access from many platforms:● Web app from Google App Engine● Apps Script (e.g. from Google Spreadsheet)● Desktop app

How does it work?

"english" The quick brown fox jumped over the lazy dog.

"english" To err is human, but to really foul things up you need a computer.

"spanish" No hay mal que por bien no venga.

"spanish" La tercera es la vencida.

? To be or not to be, that is the question.

? La fe mueve montañas.

The Prediction APIfinds relevantfeatures in the sample data during training.

The Prediction APIlater searches forthose featuresduring prediction.

Using the Prediction API

1. Upload

2. Train

Upload your training data toGoogle Storage

Build a model from your data

Make new predictions3. Predict

A simple three step process...(REST Calls)

Prediction API - key features

● Multi-category prediction○ Tag entry with multiple labels

● Multiple Prediction Output○ Finer grained prediction rankings based on multiple labels

● Mixed Inputs○ Both numeric and text inputs are now supported

Can combine continuous output with mixed inputs

Prediction Demos

● Command line Demos ○ Training a model○ Checking training status○ Making predictions

● A complete Web application using the JavaScript API for Prediction

Google BigQueryInteractive analysis of large datasets in Google's cloud

Introducing Google BigQuery

● Google's large data adhoc analysis technology○ Analyze massive amounts of data in seconds

● Simple SQL-like query language

● Flexible access○ REST APIs, JSON-RPC, Google Apps Script

Many Use Cases ...

Spam TrendsDetection

Web Dashboards

Network Optimization

Interactive Tools

Key Capabilities of BigQuery

● Scalable: Billions of rows

● Fast: Response in seconds

● Simple: Queries in SQL

● Web Service○ REST○ JSON-RPC○ Google App Scripts

Using BigQuery

1. Upload

2. Import

Upload your raw data toGoogle Storage

Import raw data into BigQuery table

Perform SQL queries on table

3. Query

Another simple three step process...

(REST Calls)

Writing Queries

Compact subset of SQL○ SELECT ... FROM ...

WHERE ... GROUP BY ... ORDER BY ...LIMIT ...;

Common functions○ Math, String, Time, ...

Statistical approximations○ TOP○ COUNT DISTINCT

BigQuery via REST

GET /bigquery/v1/tables/{table name}

GET /bigquery/v1/query?q={query}

Sample JSON Reply:{ "results": { "fields": { [ {"id":"COUNT(*)","type":"uint64"}, ... ] }, "rows": [ {"f":[{"v":"2949"}, ...]}, {"f":[{"v":"5387"}, ...]}, ... ] }}Also supports JSON-RPC

BigQuery Security and Privacy

Standard Google Authentication● Client Login● AuthSub● OAuth

HTTPS support● protects your credentials● protects your data

Relies on Google Storage to manage access

What's New in BigQuery V2?

● A new REST API● A new web user interface● Support for JOIN statements● Export table or query result to a CSV file in Google

Cloud Storage● Support for ACLs on collections of tables● A new object architecture describing tables, groups of

tables, and queries. This new architecture is described under Main Concepts below.

New!

BigQuery Demo

BigQuery Web Browser

Google Cloud SQLA relational database in the cloud

Google Cloud SQL

● Developer console○ Easy to use

● Fully managed

● High availability○ Synchronous replication to multiple data centers

● Integrated with Google App Engine○ Java: JDBC, Python: DB-API○ Use with High Replication Datastore

● MySQL Compatible○ Import / export

Cloud SQL Demos!

● Cloud SQL Console● SQL via JDBC Access● SQL from PHP!

Recap

● Google App Engine○ General purpose application development platform for

the cloud

● Google Cloud Storage○ High speed cloud data storage on Google's

infrastructure

● Prediction API○ Google's machine learning technology

● BigQuery○ Interactive analysis of very large data sets

● Google SQL○ A relational SQL database in the cloud

Further info available at:● Google App Engine

○ http://code.google.com/appengine

● Google Cloud Storage○ http://code.google.com/apis/storage

● Prediction API○ http://code.google.com/apis/predict

● BigQuery○ http://code.google.com/apis/bigquery

● Cloud SQL○ http://code.google.com/apis/sql

Thank You!

Christian SchalkGoogle Developer Advocateprofiles.google.com/cschalk@cschalk

(Follow me on G+ to get the slides!)