Open stack ocata summit enabling aws lambda-like functionality with openstack swift and openwhisk

Enabling AWS Lambda-like functionality with OpenStack Swift and OpenWhiskShaun Murakami, Senior Cloud Architect, @stmurakaAndrew Bodine, Software Engineer, @bodine_andrew

Development in the Cloud

Bare metal

Virtual machines

ContainersFunctions

Decreasing concern (and control) over stack implementation

Incr

easi

ng fo

cus

on b

usin

ess

logi

c

Serverless can handle many cloud native 12 Factors appI Codebase Handled by developer (Manage versioning of functions on their own)

II Dependencies Handled by developer, facilitated by serverless platform (Runtimes and packages)

III Config Handled by platform (Environment variables or injected event parameters)

IV Backing services Handled by platform (Connection information injected as event parameters)

V Build, release, run Handled by platform (Deployed resources are immutable and internally versioned)

VI Processes Handled by platform (Single stateless containers often used)

VII Port binding Handled by platform (Actions or functions are automatically discovered)

VIII Concurrency Handled by platform (Process model is hidden and scales in response to demand)

IX Disposability Handled by platform (Lifecycle is hidden from the user, fast startup and elastic scale is prioritized)

X Dev/prod parity Handled by developer (The developer is the deployer. Scope of what differs is narrower)

XI Logs Handled by platform (Developer writes to console.log, platform handles log streaming)

XII Admin processes Handled by developer (No distinction between one off processes and long running)

Event driven programming

Execute app logic in response to database triggers

Execute app logic in response to sensor data

Execute app logic in response to cognitive trends

Execute app logic in response to scheduled tasks

Provide easy server-side backend for mobile app

Amazon Lambda• An event-driven compute service where AWS Lambda runs

your code in response to events, such as changes to data or in response to HTTP requests.

http://docs.aws.amazon.com/lambda/latest/dg/welcome.htmlhttp://docs.aws.amazon.com/lambda/latest/dg/lambda-introduction.html

OpenWhisk

An open source cloud platformthat executes code

in response to events

https://github.com/openwhisk/openwhisk

OpenWhisk supports many growing workloads

OpenWhisk can help power various mobile, web and IoT app use cases by simplifying the programming model of orchestrating various services using events without a dedicated backend.

Digital app workloads Big Data/Analytics pipelineComplex data pipelines for Big Data/Analytics tasks can be scripted using changes in data services or streams for near real-time analytics and feedback.

DevOps and infrastructure as codeOpenWhisk can be used to automate DevOps pipelines based on events triggered from successful builds, or completed staging, or a go-live event.

Microservices builderOpenWhisk can be used to easily build microservices given the footprint and programming model desired by microservices

OpenWhisk Programming model

Developers use Rules to…associate how Actions handle events emitted by Triggers generated by services

R T A:=

riggersA class of events that can occurT

Social events

Data changes

Device readings Location updates

User input

ctionsCode that runs in response to an event (i.e., an event handler)

A

Supports a variety of languages such asJavaScript, Python, Java, and Swift ...

function main(params) {return { message: 'Hello, ' +

params.name + ' from ' + params.place };};

... even as a Docker container

Actions can be chained together

AA := A1 + A2 + A3

AB := A2 + A1 + A3

AC := A3 + A1 + A2

OpenWhisk can implement RESTful microservices

Send HTTP requestHTTP GET

app.com/customers

1

Invoke OpenWhiskaction get-customers

Browser

Mobile App

Web App

2

JS Swift Docker …

OpenWhisk

API Proxy

OpenStack Swift Middleware• Swift uses middleware to add custom behaviors.• The majority of Swift middleware is applied to the

Proxy Server.• Python WSGI Middleware (or just “middleware”)

can be used to “wrap” the request and response of a Python WSGI application (i.e. a webapp, or REST/HTTP API), like Swift’s WSGI servers (proxy-server, account-server, container-server, object-server)

http://docs.openstack.org/developer/swift/development_middleware.html

http://docs.openstack.org/admin-guide/objectstorage-arch.html

OpenStack Swift System Metadata• Provides a means to store potentially private custom metadata

with associated Swift resources in a safe and secure fashion.• May be set on accounts and containers by including headers

with a PUT or POST request. • Maybe set on objects using only PUT requests.• Takes the form of X-<type>-Sysmeta-<key>: <value>

• <type> depends on the resources type (i.e. Account, Container, Object)

• <key> and <value> are set by trusted code running in a Swift WSGI Server

http://docs.openstack.org/developer/swift/development_middleware.html#sysmeta

Demo Overview

AWS S3 + Lambda

1. A user uploads an object to the source bucket in Amazon S3 (object-created event).

2. Amazon S3 detects the object-created event.3. Amazon S3 publishes the s3:ObjectCreated:* event

to AWS Lambda by invoking the Lambda function and passing event data as a function parameter.

4. AWS Lambda executes the Lambda function by assuming the execution role that you specified at the time you created the Lambda function.

5. From the event data it receives, the Lambda function knows the source bucket name and object key name. The Lambda function reads the object and creates a thumbnail using graphics libraries, and saves it to the target bucket

http://docs.aws.amazon.com/lambda/latest/dg/with-s3-example.html

OpenStack Swift + OpenWhisk

JS Swift Docker …

OpenWhisk

API P

roxy 1. A user uploads an object to a OpenStack Swift

Container.2. OpenStack Swift detects an object has been

modified in the Container.3. The Swift Webhook middleware triggers an

OpenWhisk Action by invoking the OpenWhisk API specified as the container’s webhook URL.

4. The OpenWhisk credentials are validated against the OpenWhisk authorization service.

5. From the request payload, the OpenWhisk action knows the source container and object name. The OpenWhisk Action creates a thumbnail using graphics libraries and saves it to a target container

Swift

Mid

dlew

are

SourceContainer

TargetContainer

2

3

4

WhiskAction

1

5

Let’s take a look at how it’s implemented• OpenStack Swift Middleware Code• OpenWhisk Action Code

OpenStack Swift Middleware (webhook.py)class WebhookMiddleware(object):

@wsgifydef __call__(self, req):

req.headers[SYSMETA_WEBHOOK] = req.headers['x-webhook']req.headers[SYSMETA_WEBHOOK_AUTH] = req.headers['x-webhook-auth']

resp = req.get_response(self.app)

if obj and is_success(resp.status_int) and req.method in [’PUT’,’DELETE’]:webhook_req = urllib2.Request(container_info['sysmeta'].get('webhook'))

webhook_req.add_header('Authorization', base64.b64encode(container_info['sysmeta'].get('webhook-auth')))webhook_req.add_data(json.dumps({

’url’: swiftUrl, ‘token’: req.headers[‘x-auth-token’], ‘container’: container, ‘object’: obj, ‘method’: req.method}))

...try:

webhook_resp = urllib2.urlopen(webhook_req).read()….

return resp

Install Webhook Middleware1. Place webhook.py in middleware of swift-proxy…/swift/common/middleware/webhook.py

2. Update proxy-server.conf…[pipeline:main]

pipeline = catch_errors gatekeeper healthcheck ... webhook proxy-server

[filter:webhook]

paste.filter_factory = swift.common.middleware.webhook:webhook_factory

...

OpenWhisk Action DockerfileFROM ubuntu:14.04

ENV DEBIAN_FRONTEND noninteractive

RUN apt-get update \&& apt-get install -y curl

# Node.jsRUN curl -sL https://deb.nodesource.com/setup_4.x | bash - \

&& apt-get install -y nodejs

WORKDIR /root/server

COPY server .RUN npm install

EXPOSE 8080

CMD node ./app.js

OpenWhisk Action

DemoPutting it all together…

Future workSwift Asynchronous Event Notification Framework• Support Multiple Notification Use Cases

• OpenWhisk / Serverless computing• Cloud inter-service notifications• Metadata search indexing• Resource / traffic monitoring

• Common Notification APIs• Enable / disable notifications on an account or container• Allow for user / system level policies• Desire to become a de-facto standard in Swift for notification management

• Deployment Flexibility• Can be adapted to multiple message queue technologies• Initial submission will include Kafka and Zaqar

• Patch WIP• https://review.openstack.org/388393

Resources• AWS Lambda

• Example: http://docs.aws.amazon.com/lambda/latest/dg/with-s3-example.html

• OpenWhisk• Code: https://github.com/openwhisk/openwhisk• On IBM BlueMix: http://www.ibm.com/cloud-computing/bluemix/openwhisk/

• OpenStack Swift Middleware• http://docs.openstack.org/developer/swift/development_middleware.htm

• Demo Code• https://github.ibm.com/stmuraka/OpenStackSwift-OpenWhisk