MongoSF 2012

© 2012 VMware, Inc. All rights reserved

Activity Streams on Cloud Foundry with MongoDB and Node.jsMongoSF - 2012

By Monica Wilkinson

Friday, May 4, 12

CONFIDENTIAL

About Monica Wilkinson

2

Loves the web and data portability.

Developer Advocate @ Cloud Foundry12 years development experience.

Last 5 years in Social WebOpen Web Standards Advocate

Contact Me: @ciberchmwilkinson@vmware.com

Friday, May 4, 12

CONFIDENTIAL3

Agenda

1.Overview of Cloud Foundry2.What is Activity Streams ?3.Building an Activity Stream Engine

Friday, May 4, 12

CONFIDENTIAL

About Cloud Foundry

4

The first Open PaaS

Multi(n) Languages, Frameworks, Services & Clouds

Open Source

Friday, May 4, 12

5

Deploy and scale applications in seconds, without locking yourself

Simple, Open,

Flexible, Scalable

The Open Platform as a Service

Friday, May 4, 12

Cloud Foundry open PaaS - Choice of frameworks

Friday, May 4, 12

Cloud Foundry open PaaS - Choice of frameworks

OSS community

Friday, May 4, 12

Applica'on  Service  Interface

Data Services

Other Services

Msg Services

Cloud Foundry open PaaS - Choice of application services

Friday, May 4, 12

Applica'on  Service  Interface

Data Services

Other Services

Msg Services

Cloud Foundry open PaaS - Choice of application services

vFabric Postgres

vFabric RabbitMQTM

Additional partners services …

Friday, May 4, 12

Applica'on  Service  Interface

Data Services

Other Services

Msg Services

Cloud Foundry open PaaS - Choice of clouds

Private  Clouds  

PublicClouds

MicroClouds

Cloud  Provide

r  Interface

Friday, May 4, 12

Applica'on  Service  Interface

Data Services

Other Services

Msg Services

Cloud Foundry open PaaS - Choice of clouds

Private  Clouds  

PublicClouds

MicroClouds

Cloud  Provide

r  Interface

Avoid

Lock-in

Friday, May 4, 12

9

Multi-cloud flexibility is critical to your long-term success

Friday, May 4, 12

9

Multi-cloud flexibility is critical to your long-term success

§ Make use of both public and private clouds without rewriting your applications

Friday, May 4, 12

9

Multi-cloud flexibility is critical to your long-term success

§ Make use of both public and private clouds without rewriting your applications

§ Protect against vendor lock-in

Friday, May 4, 12

9

Multi-cloud flexibility is critical to your long-term success

§ Make use of both public and private clouds without rewriting your applications

§ Protect against vendor lock-in

§ Meet different compliance and geographical needs

Friday, May 4, 12

9

Multi-cloud flexibility is critical to your long-term success

§ Make use of both public and private clouds without rewriting your applications

§ Protect against vendor lock-in

§ Meet different compliance and geographical needs

§ Accommodate peak loads while optimizing costs

Friday, May 4, 12

9

Multi-cloud flexibility is critical to your long-term success

§ Make use of both public and private clouds without rewriting your applications

§ Protect against vendor lock-in

§ Meet different compliance and geographical needs

§ Accommodate peak loads while optimizing costs

§ Manage your growth and changing needs over time

Friday, May 4, 12

10

CloudFoundry.COM - Multi-tenant PaaS operated by VMware

Runtimes & Frameworks

Services

vCenter / vSphere

CloudFoundry.COM (beta)

Infrastructure

Friday, May 4, 12

11

Micro Cloud FoundryTM – Industry first downloadable PaaS

Runtimes & Frameworks

Services

Your Laptop/PC

Micro Cloud Foundry

Single VM instance of Cloud Foundry that runs on a developer’s MAC or PC

Friday, May 4, 12

12

CloudFoundry.ORG - Community open-source project

CloudFoundry.ORG

DownloadCode

Setup Environment

Deploy Behind Firewall

Setup Scripts

Apache2 license

Your Infrastructure

Friday, May 4, 12

Cloud Foundry: you can trade-off effort vs flexibility

13

Public PaaS

Private PaaS

Custom Private PaaS

Less

More

Less

More

.COM....

....

git clone git://github.com/cloudfoundry/

Effort Flexibility

Friday, May 4, 12

CONFIDENTIAL14

Why PaaS Matters

Friday, May 4, 12

Container (e.g. Tomcat)

Traditional web application architecture

15

Desktop Browser

Apache

Web Application

MySQL Database

developtestdeployscale

ShippingService

AccountingService

InventoryService

StoreFront

Simple to

Friday, May 4, 12

Container (e.g. Tomcat)

Traditional web application architecture

15

Desktop Browser

Apache

Web Application

MySQL Database

developtestdeployscale Or is it?

ShippingService

AccountingService

InventoryService

StoreFront

Simple to

Friday, May 4, 12

?Let’s imagine you want to run a performance test...

How long to get the servers?Who is going to set up the servers?Who is going to set up Apache, Tomcat, and MySQL?

Friday, May 4, 12

And things are changing:

This simple, monolithic architecture is inadequate

17

Friday, May 4, 12

18

Smart phones overtake PCs in Q4 2010

New kinds of clients

Friday, May 4, 12

Users expect a rich, dynamic and interactive experience on mobile devices and desktop

19

Web ApplicationDesktop Browser

HTTP Request

HTML/Javascript

Friday, May 4, 12

Users expect a rich, dynamic and interactive experience on mobile devices and desktop

19

Web ApplicationDesktop Browser

HTTP Request

HTML/Javascript

Old style UI architecture isn’t good enough

Friday, May 4, 12

Need to handle massive loads and the data explosion

20

Need scalable architectures

§ Application tier:•Replicated/clustered servers•Modular so that components can be scaled differently•Asynchronous architecture - communication via a message

broker§ Database tier:•Replication•Sharding•Polyglot persistence: Relational, NoSQL, NewSQL

databases

Friday, May 4, 12

Scaling development

21

§ Strongly coupled components makes development difficult

§ Forces multiple developers/teams to synchronize development efforts

§ Obstacle to frequent, independent deployments

§ Increases risk of failure - need to redeploy everything to change one thing

!=Scalable development

Web application

ShippingService

AccountingService

InventoryService

StoreFront

Friday, May 4, 12

Desktop Browser

NodeJS front-end application

Native Mobile Application

HTML5 mobile application

RabbitMQ

NodeJS

MySQL

Mongo

Redis

ShippingService

AccountingService

billing web application

shipping web application

InventoryService

inventory web application

StoreUI

StoreUI

StoreUI

StoreUI

22

Modern application architecture

Friday, May 4, 12

Developing and testing these applications is even more challenging

23

Friday, May 4, 12

Let’s imagine you are fixing a bug and want to run some JUnit integration tests...

Who is going to setup your sandbox: MySQL, RabbitMQ, MongoDB, ....??

Friday, May 4, 12

Let’s imagine you have fixed a bug and want to run some functional tests...

How long to purchase the servers?Who is going to set up the servers?Who is going to setup MySQL, RabbitMQ, MongoDB, ....??

Friday, May 4, 12

26

Platform-as-a-Service is the solution

Easy deployment

Application management

Easy scaling up and down

+SQL databases

NoSQL databases

Message Broker

Deployment Services

Friday, May 4, 12

CONFIDENTIAL27

Getting Started on Cloud Foundry

Friday, May 4, 12

CONFIDENTIAL

STEP 1 - Get a Cloud Foundry Account

§ https://my.cloudfoundry.com/signup/hack

28

Friday, May 4, 12

29

STEP 1 - Fetch password from email

Friday, May 4, 12

CONFIDENTIAL

STEP 2 - Login to Cloud Foundry

•sudo gem install vmc

•vmc login <username>

30

Friday, May 4, 12

CONFIDENTIAL

STEP 3 - Push your code to Cloud Foundry

vmc push

• Detects runtime automatically. • In addition if you have manifest.yml vmc will read:

• App Name • App Instances• Services to bind to App

•Demo

31

Friday, May 4, 12

CONFIDENTIAL32

Services on Cloud Foundry

Friday, May 4, 12

Current Services Available on Cloud Foundry

33

Friday, May 4, 12

Creating a service instance$ vmc create-service mysql --name mysql1

Creating Service: OK

$ vmc services

============== System Services ==============+------------+---------+---------------------------------------+| Service | Version | Description |+------------+---------+---------------------------------------+| mongodb | 1.8 | MongoDB NoSQL store || mysql | 5.1 | MySQL database service || postgresql | 9.0 | PostgreSQL database service (vFabric) || rabbitmq | 2.4 | RabbitMQ messaging service || redis | 2.2 | Redis key-value store service |+------------+---------+---------------------------------------+

=========== Provisioned Services ============

+-------------+---------+| Name | Service |+-------------+---------+| mysql1 | mysql |+-------------+---------+

Friday, May 4, 12

$ vmc push cer-spring --path web/target/Application Deployed URL [cer-spring.cloudfoundry.com]: Detected a Java SpringSource Spring Application, is this correct? [Yn]:

Memory Reservation (64M, 128M, 256M, 512M, 1G) [512M]:

Creating Application: OKWould you like to bind any services to 'cer-spring'? [yN]: y

Would you like to use an existing provisioned service? [yN]: yThe following provisioned services are available

1: mysql1

2: mysql-135e0Please select one you wish to use: 1

Binding Service [mysql1]: OKUploading Application:

Checking for available resources: OK

Processing resources: OK Packing application: OK

Uploading (12K): OK

Binding a service to an application

Friday, May 4, 12

$ vmc push cer-spring --path web/target/Application Deployed URL [cer-spring.cloudfoundry.com]: Detected a Java SpringSource Spring Application, is this correct? [Yn]:

Memory Reservation (64M, 128M, 256M, 512M, 1G) [512M]:

Creating Application: OKWould you like to bind any services to 'cer-spring'? [yN]: y

Would you like to use an existing provisioned service? [yN]: yThe following provisioned services are available

1: mysql1

2: mysql-135e0Please select one you wish to use: 1

Binding Service [mysql1]: OKUploading Application:

Checking for available resources: OK

Processing resources: OK Packing application: OK

Uploading (12K): OK

Binding a service to an application

Would you like to bind any services to 'cer-spring'? [yN]: yWould you like to use an existing provisioned service? [yN]: yThe following provisioned services are available1: mysql12: mysql-135e0Please select one you wish to use: 1Binding Service [mysql1]: OK

Friday, May 4, 12

$ vmc push cer-spring --path web/target/Application Deployed URL [cer-spring.cloudfoundry.com]: Detected a Java SpringSource Spring Application, is this correct? [Yn]:

Memory Reservation (64M, 128M, 256M, 512M, 1G) [512M]:

Creating Application: OKWould you like to bind any services to 'cer-spring'? [yN]: y

Would you like to use an existing provisioned service? [yN]: yThe following provisioned services are available

1: mysql1

2: mysql-135e0Please select one you wish to use: 1

Binding Service [mysql1]: OKUploading Application:

Checking for available resources: OK

Processing resources: OK Packing application: OK

Uploading (12K): OK

Binding a service to an application

Friday, May 4, 12

Binding a service to an application

$ vmc bind-service mysql1 myappBinding Service [mysql1]: OKStopping Application 'myapp': OKStaging Application 'myapp': OK Starting Application 'myapp': OK$

Friday, May 4, 12

CONFIDENTIAL37

Every day development on Cloud Foundry

Friday, May 4, 12

CONFIDENTIAL

Every day development§ Debugging and accessing the data locally

• Caldecott --> Service tunneling. Access your Cloud Foundry service as if it was local.

38

Friday, May 4, 12

CONFIDENTIAL

Tunneling

39

gem install caldecott

vmc tunnel <mongodb>

Friday, May 4, 12

CONFIDENTIAL

Using your favorite tools

40

Friday, May 4, 12

CONFIDENTIAL41

Friday, May 4, 12

CONFIDENTIAL42

What is Activity Streams ?

Friday, May 4, 12

CONFIDENTIAL43

http://activityStrea.ms§ Activity Streams is a simple open specification used to describe social

actions around the web.

§ The goal of Activity Streams is data portability

§ Some publishers of Activity Streams: Socialcast, MySpace, Facebook and G+

§ The default format is JSON but Atom is also supported. The JSON format was added after we saw some initial limitations with JSON

§ Core concepts are: Actor, Verb, Object and Target

Friday, May 4, 12

© 2012 VMware, Inc. All rights reserved

Actor -> Monica

Verb -> Pinned(Bookmarked)

Object -> Rosa Parks’ Bio Page

Target -> Female Leaders Board

Source -> Pinterest

Published -> MLK Day 2012

Friday, May 4, 12

CONFIDENTIAL45

Example Activity

John posted “SCALA 2012 Recap” on his blog “John OSS”10 minutes ago

Friday, May 4, 12

CONFIDENTIAL

Activity in JSON

46

{    "published": "2011-02-10T15:04:55Z",    "actor": {      "url": "http://example.org/john",      "objectType" : "person",      "id": "tag:example.org,2011:john",      "image": {        "url": "http://example.org/john/image",        "width": 250,        "height": 250      },      "displayName": "John Smith"    },    "verb": "post",    "object" : {      “objectType” : “blog-entry”, “url": "http://example.org/blog/2011/02/entry",      "id": "tag:example.org,2011:abc123/xyz"    },    "target" : {      "url": "http://example.org/blog/",      "objectType": "blog",      "id": "tag:example.org,2011:abc123",      "displayName": "John OSS"    }  }

Friday, May 4, 12

CONFIDENTIAL47

What is an Activity Stream Engine ?§ An activity stream engine allows you to publish events and subscribe to

events.

§ Many activity stream engine support aggregation of events via Streams.

§ Many activity stream engines support fanning out activities to subscribers.

§ Many activity stream engines tailor the stream to the reader providing Key Stats and Contextual Info. Example: Monica with 1000 followers or Bank of America established in 1989

Friday, May 4, 12

CONFIDENTIAL48

Building an Activity Streams Engine

Friday, May 4, 12

CONFIDENTIAL49

What should we use to store Activities ?

ColumnKey-Value Document Graph

Redis, Riak

Cassandra,HBase

MongoDB Neo4J

Friday, May 4, 12

CONFIDENTIAL50

ColumnKey-Value Document Graph

MongoDB (so easy)

Mongo DB schema-less documents are perfect for Activities

Friday, May 4, 12

CONFIDENTIAL

About MongoDB

51

1. No-SQL database

2. Stores JSON-style documents with embedded documents

3. Horizontally scalable

4. Full Indexing Support

5. Open Source so great ORMs and drivers.

development : {

tools: many,

language_support: superb,

agility: high

},

production: {

speed: fast

fault_tolerance: true

scalability: high

=

Friday, May 4, 12

CONFIDENTIAL

Step 1- Build the persistence Modelvar MediaLinkHash = {

        duration: Number,        height: Number,        width: Number,        url: String    };

    var ActivityObjectHash = {       id: {type: String},       image: MediaLinkHash,       displayName: {type: String},       summary: {type: String},       content: {type: String},       url: {type:String},       author: {type: ObjectId, ref: 'activityObject'},       published: {type: Date, default: Date.now},       updated: {type: Date, default: Date.now},       objectType: {type: String},       attachments: [{type: ObjectId, ref: 'activityObject'}],       upstreamDuplicates: [String],       downstreamDuplicates: [String]    };

    var ActivityObjectSchema = new Schema(ActivityObjectHash)    this.ActivityObject = mongoose.model('activityObject', ActivityObjectSchema);

52

Friday, May 4, 12

CONFIDENTIAL

Step 1- Persistence Model    var ActivitySchema = new Schema({        id: {type: String},        verb: {type: String},        url: {type: String},        title: {type: String},        content: {type: String},        icon: MediaLinkHash,        object: ActivityObjectHash,        actor: ActivityObjectHash,        target: {type: ObjectId, ref: 'activityObject'},        published: { type: Date, default: Date.now},        updated: { type: Date, default: Date.now},        inReplyTo: {type: ObjectId, ref: 'activity'}    });

    this.Activity = mongoose.model('activity', ActivitySchema);

    return this;};

53

Friday, May 4, 12

CONFIDENTIAL

Step 2 - Expose helpers for the queries§ The most important is to be able to list the activities in

descending order by published time.§ It is also important to hydrate any objects for which we

have references

§ With Mongoose you can do: § this.getActivityStream = function(n, fx) { Activity.find().sort('published', 'descending').limit(n).populate('target').run(fx);

    }

§ More at: https://github.com/cloudfoundry-samples/activity-streams-mongoose

54

Friday, May 4, 12

CONFIDENTIAL

How do we make our engine faster and more scalable ?§ Add support for PubSub§ In a PubSub model as soon as an event is published it

gets sent to all the subscribers.§ The Publisher doesn’t know who the subscribers are, it

simply knows where to publish.§ Redis has nice support for Pub Sub http://redis.io/topics/

pubsub

55

Friday, May 4, 12

CONFIDENTIAL56

Exposing the Engine as Service

Friday, May 4, 12

CONFIDENTIAL

Using Socket.io

§ Server Side:• Subscribe to a Stream• When there is a new event send it to the client as JSON

§ Client Side• Add support to publish activities to the Stream• Change the client rendering to use ActivityStrea.ms

§ Benefits• Richer messaging• Activity Syndication

57

Friday, May 4, 12

CONFIDENTIAL58

Demo

http://asms.cloudfoundry.com

Friday, May 4, 12

© 2012 VMware, Inc. All rights reserved

Thank You - Questions ?Cloud Foundry Activity Streams Libraries are at:

https://github.com/organizations/cloudfoundry-samples

http://cloudfoundry.com

Questions: @cloudfoundry - @ciberch or mwilkinson@vmware.com

Friday, May 4, 12