Building TweetEngine

Building TweetEngine

Ikai LanDeveloper Relations, Google App Engine

[email protected]: @ikai

Goals of this talk

Using TweetEngine to explain key concepts of building cloud applicationsExplaining App Engine as a development environmentWhet your appetite!

What is App Engine?

App Engine is a platform

You build & test your app

Then upload your app to Google

App Engine runs everything

No need to worry about machines, network, storage, scalability, etc.

The king of ease-of-use

Extremely rapid developmentVery low barrier of entrySimple yet robust syntaxRich library of packages/modulesApp Engine's first language API

Components

Getting TweetEngine

Application: http://tweetengine.net/

Code: http://github.com/Arachnid/tweetengine

Prerequisites:Python 2.5buildoutlxml (for i18n features)

OAuth Secure method for granting permissions to third-party integrationsAllows sites to use a trusted identity provider (Google, Twitter)"Valet key" for the internet

Image and "valet key" source: (http://hueniverse.com/oauth/guide/intro/)

Standard OAuth flow (User)What does the User see?

1. Link to login with Twitter2. User is redirected to an http:

//twitter.com URL3. User logs in, grants access4. User is redirected back to site

Standard OAuth flow (Server)What does the server do?

1. Acquire request token2. Generate login URL3. Redirect or link user to this

URL4. After user has logged in,

Twitter will redirect to a callback URL on server

5. Server saves token passed to callback URL

TweetEngine + OAuth

OAuth request token saved in TwitterAccount modelSaved per Twitter Accountoauth.py handles signingapis.py handles proxied Twitter API calls

Logging in to Twitter using OAuth

# src/tweetengine/add.pyauth_token = self.request.get("oauth_token")auth_verifier = self.request.get("oauth_verifier")user_info = client.get_user_info(auth_token, auth_verifier=auth_verifier)

# Create the twitter account account = model.TwitterAccount.get_or_insert( user_info["username"], oauth_token=user_info["token"], oauth_secret=user_info["secret"], name=user_info["name"], picture=user_info["picture"])

Initializing an OAuth client

# src/oauth.py

OAuthClient.__init__(self, "twitter", consumer_key, consumer_secret, "http://twitter.com/oauth/request_token", "http://twitter.com/oauth/access_token", callback_url)

Twitter API calls- Web handlers user AJAX- Backend calls go to TwitterApiHandler# src/tweetengine/handlers/apis.pyfor k,v in params.items(): if isinstance(v, unicode): params[k] = v.encode('utf8')

# Join all of the params together. params_str = "&".join(["%s=%s" % (encode(k), encode(params[k])) for k in sorted(params)]) # Join the entire message together per the OAuth specification. message = "&".join(["GET" if method == urlfetch.GET else "POST", encode(url), encode(params_str)])

# Create a HMAC-SHA1 signature of the message.key = "%s&%s" % (self.consumer_secret, secret)# Note compulsory "&".signature = hmac(key, message, sha1)digest_base64 = signature.digest().encode("base64").strip() params["oauth_signature"] = digest_base64 return urlencode(params)

OAuth

The "de facto" standard for authentication on the web against a third party APITweetEngine's support can be easily generalized to be used on any OAuth supporting siteHighly recommend: New sites, do not build identity system, use OAuth or OpenID

i18n - internationalization

Problem: maintaining localized versionsLocalization is more than translationsTweetEngine already supports English, German and Italian!

Image source: http://www.flickr.com/photos/natematias/22132919

Workflow

1. Developer marks strings as requiring translation2. Translators work on .PO files3. These are compiled into efficient .POT files4. At runtime, translations are looked up when pages rendered

using "Accepts-Header"

What does a .po file look like?

#. Default: "Collaborative tweeting"#: tweetengine/templates/base.pt:36msgid "subline"msgstr "Tweeting collaborativo"

#. Default: "Welcome!"#: tweetengine/templates/index.pt:3msgid "title-welcome"msgstr "Benvenuto!"

Sample template code

tweetengine/templates/index.pt:

<metal:title fill-slot="title" i18n:translate="title-welcome">Welcome!</metal:title>

tweetengine/templates/base.pt:<h2><a href="/" i18n:translate="subline">Collaborative tweeting</a></h2>

Run from TweetEngine root: bin/i18nize

src/tweetengine/handlers/base.py # Instantiate Chameleon template loaderfrom chameleon.zpt.loader import TemplateLoaderfrom tweetengine import model from tweetengine.menu import mainmenu

tpl_path = os.path.join(os.path.dirname(__file__), "..", "templates")tpl_loader = TemplateLoader(tpl_path)

# Later on in the file template_vars['target_language'] = self.request.headers.get('Accept-Language', None)tpl = tpl_loader.load('base.pt') template_vars['master'] = tpl.macros['master']tpl = tpl_loader.load('macros.pt')template_vars['macros'] = tpl.macrostpl = tpl_loader.load(template_file)self.response.out.write(tpl(**template_vars))

Remember: i18n > translations

Resizing UI elements for languages with long/short wordsRight-to-left languages (Hebrew, Arabic) and layoutsTime and date formatsMetric/English systemMuch more ... this just a place to start!

App Engine is also about tooling

Image source: http://www.flickr.com/photos/sparr0/4584964212/

Two App Engine toolsAppStats

Application profile that is App Engine aware

Task QueuesBackground queueing and worker mechanism

Image source: http://www.flickr.com/photos/spbatt/3928384579/

AppStatsTrack expensive requestsExpose bottlenecks in your codeShow you what code is being executed the most for "hotspot optimization"Run in production with minimal overhead

AppStats example

AppStats example (con't)

How does it work?Uses Call Hooks

pre‐call hook records start .me, request, stackpost‐call hook records end .me, mcycles, response

Measuring times:real time using Python’s wall clock APIAPI megacycles using rpc.cpu_usage_mcyclesCPU megacycles using quota.get_request_cpu_usage()

Get stack contents from Python’s sys._getframe()Recording accumulates in memory objectWritten to memcache at end of each request

AppStats for TweetEngine

Add AppStats as WSGI Middleware: app = recording.appstats_wsgi_middleware(app)

Add AppStats GUI to URL mapping (app.yaml): - url: /stats.* script: appstats/ui.py

Dev: http://localhost:8080/stats Prod (as admin): http://APPID.appspot.com/stats

That's it!

Task Queues

Execute background tasksThrottle controlETACan be unique!Invoked using a web handler

Standard async worker architecture

Task Queues in App Engine# Creating a task # src/tweetengine/model.py taskqueue.Task(eta=send_time, name=task_name, url=twitter.ScheduledTweetHandler.URL_PATH).add()

# Task Handler # src/tweetengine/handlers/twitter.pyclass ScheduledTweetHandler(webapp.RequestHandler):

def post(self): publishApprovedTweets()

def publishApprovedTweets(): # Iterate through unsent OutgoingTweets and send them

Task Queues in TweetEngine

Used to schedule TweetsETA parameter used to set an approximate execution timeTasks are named - this allows us to set times to check for OutgoingTweets that should be sent at this time and not duplicate work (we don't have to use cron)

Task Queues

Allow developers to build asynchronous workers without hassle of building out queueing/polling systemHandles task uniqueness and transactional task creationCan be scheduled, throttled

Summary: TweetEngine technologies

1. OAuth - "Valet key" for internet applications2. i18n - Using templates and message bundles in web

applications3. AppStats - App Engine request profiling toolkit4. Task Queues - Background work for App Engine

applications

Questions?

Application: http://tweetengine.net/

Code: http://github.com/Arachnid/tweetengine

[email protected]: @ikai