Obama For America on AWS Younjin Jeong Solutions Architect
May 06, 2015
Obama For America on AWS
Younjin Jeong
Solutions Architect
What am I talking about today?
What was OFA? Why is this relevant? • Who did it?
• What did they build?
How did they do that? • Technologies and Tradeoffs
• Services vs. Software
What did they learn from building something so big?
Full Disclosure
I work for AWS
AWS does not endorse
political candidates
Yes, I talk too much
So here’s the Idea
~30th biggest E-commerce operation, globally
~200 distinct new applications, many mobile
Hundreds of new, untested analytical approaches
Processing hundreds of TB of data on thousands of servers
Spikes of hundreds of thousands of concurrent users
FUN FUN FUN
a few constraints…
~30th biggest E-commerce operation, globally
~200 distinct applications, many mobile
Hundreds of new, untested analytical approaches
Processing hundreds of TB of data on thousands of servers
Spikes of hundreds of thousands of concurrent users
Critically compressed budget
Less than a year to execute
Volunteer and near-volunteer development team
Core systems will be used for a single critical day
Constitutionally-mandated completion date
NOT FUN NOT FUN
NOT FUN
CHALLENGE ACCEPTED !
Built by guys and gals like these: Obama For America
Business as usual..
…for a technology startup
Election Day – OFA Headquarters
So they built it all, and it worked
Typical Charts
How?
The old approach, even from Amazon
The old approach.. Might have some problems..
No Up-Front
Capital Expense Pay Only for
What You Use
Self-Service
Infrastructure Easily Scale
Up and Down
Improve Agility &
Time-to-Market
Low Cost
Cloud Computing Benefits
Deploy
OFA’s Infrastructure
awsofa.info
Web-Scale Applications
500k+ IOPS DB Systems
Services API
Ingredients
Ubuntu nginx boundary Unity jQuery SQLServer hbase
NewRelic EC2 node.js Cybersource hive ElasticSearch
Ruby Twilio EE S3 ELB boto Magento PHP EMR SES
Route53 SimpleDB Campfire nagios Paypal CentOS
CloudSearch levelDB mongoDB python securitygroups
Usahidhi PostgresSQL Github apache bootstrap SNS
cloudformation Jekyll RoR EBS FPS VPC Mashery
Vertica RDS Optimizely MySQL puppet tsunamiUDP R
asgard cloudwatch ElastiCache cloudopt SQS cloudinit
DirectConnect BSD rsync STS Objective-C dynamoDB
Data Stores
Ubuntu nginx boundary Unity jQuery SQLServer hbase
NewRelic EC2 node.js Cybersource hive ElasticSearch
Ruby Twilio EE S3 ELB boto Magento PHP EMR SES
Route53 SimpleDB Campfire nagios Paypal CentOS
CloudSearch levelDB mongoDB python securitygroups
Usahidhi PostgresSQL Github apache bootstrap SNS
cloudformation Jekyll RoR EBS FPS VPC Mashery
Vertica RDS Optimizely MySQL puppet tsunamiUDP R
asgard cloudwatch ElastiCache cloudopt SQS cloudinit
DirectConnect BSD rsync STS Objective-C dynamoDB
Development Frameworks
Ubuntu nginx boundary Unity jQuery SQLServer hbase
NewRelic EC2 node.js Cybersource hive ElasticSearch
Ruby Twilio EE S3 ELB boto Magento PHP EMR SES
Route53 SimpleDB Campfire nagios Paypal CentOS
CloudSearch levelDB mongoDB python securitygroups
Usahidhi PostgresSQL Github apache bootstrap SNS
cloudformation Jekyll RoR EBS FPS VPC Mashery
Vertica RDS Optimizely MySQL puppet tsunamiUDP R
asgard cloudwatch ElastiCache cloudopt SQS cloudinit
DirectConnect BSD rsync STS Objective-C dynamoDB
Sites
Communications
Ad Targeting
Ops Tools
Analytics
Apps
Micro-targeting
Micro-listening
Reporting
Registrations
Volunteer
Coordination
Etc, etc, etc.
Technology Choice
Polyglot Development
Cloud Hosting
Diverse, App-centered Databases
SOA, queue-based system integrations
Expected Tradeoff
More Complex Ops
Less Infra Control, performance
More Complex Ops, Fragility, Data Corruption
Dev Complexity, slower system performance
Technology Choice
Polyglot Development
Cloud Hosting
Diverse, App-centered Databases
SOA, queue-based system integrations
Expected Tradeoff
More Complex Ops
Less Infra Control, performance
More Complex Ops, Fragility, Data Corruption
Dev Complexity, slower system performance
Upside
Build as little as possible, rev-1 faster, reuse dev skills
Scale, Speed, Cost
Heterogeneous Resilience, right tools for the job
Scalability, serviceability, operational flexibility, and substantially faster in aggregate
No time to waste
This applies to lots of services!
ELB
ElastiCache
RDS
CloudSearch
Route53
S3
CloudFront
DynamoDB
You can mostly do
these on your own…
But do you have extra:
focus, expertise, time, research,
money, risk-tolerance, staff, dedication to
innovate, operations coverage, scalability in design...
Looks pretty simple.
Inserts 7.5m records in DynamoDB, in 8 minutes
One thing that is difficult to prepare for…
No pressure…
They had this built for the previous 3 months, all on the East Coast.
They had this built for the previous 3 months, all on the East Coast.
We built this part in 9 hours
to be safe.
AWS + Puppet +
Netflix Asgard + CloudOpt + DevOps =
Cross-Continent Fault-Tolerance On-Demand
Replication across the continent..
http://tsunami-udp.sourceforge.net/
So what did they learn?
Game Day: Practice failures so you know what to do.
Loose-Coupling: Ops easy, scale easy, test easy, fix easy…
Fail-Forward: features, quality, and focus are all critical.
HA in Depth: S3 static pages, de-coupled UI, jekyll/hyde
Cloud works.
What will you do next?
Maybe look at some of their Ruby code?
https://github.com/democrats/voter-registration
AMAZON REDSHIFT
AMAZON REDSHIFT Redshift runs on HS type instances
HS1.8XL: 128 Go RAM, 16 Coeurs, 16 To de contenu compressé, 2 Go/sec en lecture
HS1.XL: 16 Go RAM, 2 Coeurs, 2 To de contenu compressé
Extra Large Node
(HS1.XL)
Single node
Cluster 2-32 Nodes (4 To – 64 To)
Eight Extra Large Node (HS1.8XL) Cluster 2-100 Nodes (32 To – 1.6 Po)
10 GigE
(HPC)
Ingestion
Backup
Restoration
JDBC/ODBC
Amazon DynamoDB
AMAZON ELASTIC
MAPREDUCE
AMAZON
DYNAMODB
AMAZON EC2
AWS STORAGE
GATEWAY
AMAZON S3
DATA CENTER
AMAZON RDS
AMAZON
REDSHIFT
Younjin Jeong - AWS [email protected]
Thank you!