AWS Lambda at JUST EAT

AWS Lambda at JUST EAT

ANDREW BROWN - Senior Platform Engineer

Overview

Just Eat

Early Adoption

Use Cases

Serverless

• The world’s leading digital marketplace for takeaway food delivery• Founded in 2001• Operate in 13 markets around the globe

JUST EAT

• Platform peak has processed 2,500 orders per minute

• Thats a lot of Food!

TECHNOLOGY

• Run hundreds of EC2 Instances at peak / dinner time

• Scheduled Scaling of Resources - EC2, Dynamo Capacity

• Heavily utilise CloudFormation

• Just Eat Migrated to AWS 5 years ago - All In!

AWS - Setup

• Multiple AWS Accounts

• QA Daily Launch and Teardown

• On average an instance less than 3 days old

•

• Approx 150 “features” - Each Feature has a Stack

What’s good about our Setup?Developers can concentrate on writing code

• Packer, CodeDeploy, Consul, Cloudformation

• Developers create package and deploy!

• PaaS, IaaS

What’s good about our Setup?

Consistency

• Many accounts, environments, countries, features

• All get the same AWS resources - e.g. Security Groups

• Templated Cloudformations

AWS - Lambda

• Re:Invent 2014 announcement

• AWS Lambda runs your code in response to events

•

• “we could use that for x, y, and z!”

So Why Use Lambda?

• Lightweight

• Cheaper

• Less to manage

• Quicker to Deploy

But Why Should JE Use Lambda?

Let’s review One Feature

• CloudFormation• EC2 Instances x 3 - Multi AZ• EBS Volumes• ELB Endpoint and Health Check• IAM Roles• DNS• Security Groups, Subnets, VPC• ASG settings, LaunchConfiguration

~ 1600 lines of JSON

$0.246 x 3 x 24 x 365 (m4.large)

$0.10 x 30 x 3 x 12 (gp2)

$0.028 x 24 x 365 (classic)

~= $7k

So Why Use Lambda?

What does this Feature do?

• Resets Delivery Time every morning

• Publishes a SNS message about an order

• Provision access for an instance

• Tidy up resources

Early Lambdas● New Technology!

● Multiple languages and frameworks

● Seen as additional parts of a feature

● Using the Console….

Early LambdasLimitations / Problems

● Consistency (that’s Just Eat not AWS!)

● No VPC Support (Added Feb 2016)

● No Cron in Cloudformation

● Cloudformation Network Still Attached

Use Cases - PaaSBot

• CloudFormation Stack per Environment

• Windows EC2 Instance

• Ran a bunch of Scheduled Tasks

Use Cases - PaaSBotClean up EBS Volumes

• Cron build in Teamcity - one per env• Scheduled Windows Task (one per account, assumed roles)• Lambda?

Report Metrics

• T2 Credit Balances• AWS Limits• Lambda?

Use Cases - PaaSBotSecurity Audit

• Check Security Group rules• Check User access

Use Cases - PaaSLambdaLambdas

• No Windows Instance(s)• Micro-Scheduled Tasks• Easier to update• Better visibility

Use Cases - Instance Access

• Recently started using HashiCorp Consul and Vault

• Concerns : “What can my instance access?”

• Consul ACLS and Vault Tokens - limit what key/values you can access

• ACLs Initially provisioned via Deployment Tool

Use Cases - Instance Launch

• Lambdas Created with Consul and Vault Cloudformations - nodejs

• ASG Event -> CloudWatch Events -> Trigger Lambda

• Provisions ACL and AppId login

• Removes Access when Instance terminated

Use Cases - Instance LaunchLaunch Stack

i-12345i-12345

i-12345

ASG

CloudWatch Event

{ “acl” : a1a1a1-b2b2b2 }

{ “token” : secrettoken }

Use Cases - Instance TerminateDelete Stack / Scale Down

i-12345i-12345

i-12345

ASG

CloudWatch Event

Different Approach• Starting to create Lambda Features

• Needed a consistent approach

• Recently Implemented Serverless

Serverless• Still in “Beta”

ServerlessGood Points

• Brings Back Consistency• Ease to deploy• Defined Dependencies• Being Quickly Adopted

• Track signal strength • Calculate cost• Check capacity• Start Ad-hoc backup / Prune Log files

ServerlessNot so Good Points

• Currently each Feature includes their AWS resources - previously templated

• Code repeated

• Monitoring & Logging Immature

Lessons Learnt & Future Plans• Difficult to integrate into our Platform - Versatility • Still a few Bugs -

• Serverless - Based on Initial Uptake• AWS Lambda & API Gateway• Will it replace our APIs? - Maybe

Any questions?

Thank you