Mobile apps have different service requirements from their desktop and web-based analogs. Bandwidth, client processing, and other considerations can impose significant extra demands on a scalable service. This session is a technical discussion of the challenges Flipboard met while scaling a data-intensive mobile app from 0 to 100 million clients and how they are working on scaling 10x using AWS. At each major step, Flipboard has encountered many challenges. Learn about how they handled those challenges and the evolution of their systems architecture, design choices, and software selection.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Architecture Changes• Different services have different scale profiles — began
the shift towards microservices• Image content moved to CloudFront• Moved primary data store to MySQL via Amazon RDS• Home grown bash scripts for deploys• Focus on instrumentation
• Logging, Metrics, Monitoring followed suit
Friday, November 15, 13
Host[i-76e33611] - Amazon Instance ID name[tsd01] - Name of the instance owner[ops] - Who owns the instance? service[OOS] - IS for in service, OOS for out of service ami[ami-3f622f56] - What AMI was used type[m1.xlarge] - Type of EC2 instance loc[us-east-1a] - Region and Availability Zone role[flops] - Type of role subclass[opentsdb] - Subclass of role group[0] - Group of node pool[production] - Production, staging, dev public[50.16.58.220] - Public IP address private[10.60.43.18] - Private IP Address
SimpleDB for CMDB
Friday, November 15, 13
# fl-inst-describe -r flip -p production -g 0 -s IS -o ops
Domain[flipboard.prod.instances] has count[1] hosts meeting criteria=======================================Servers of role flip=======================================
Architecture Changes• Heavy focus on instrumentation of all services• Pipeline of batch processing using Hadoop• Pipeline of real-time processing using Storm + Kafka• Keen focus on using appropriately sized EC2
instances• Moving off of bash scripts, moving to puppet
Friday, November 15, 13
Mobile application instrumentation
Friday, November 15, 13
All at once?
fl-inst-upgrade -r flip -p production-q
… or …
By group?
fl-inst-upgrade -r flip -p production -g 0 -q
Deploy by groups
Friday, November 15, 13
Using CloudWatch metrics for errors
Friday, November 15, 13
fl-inst-upgrade -r flip -p production -g 1 -q
Continued your deploy
Friday, November 15, 13
Graphite for all metrics
Friday, November 15, 13
Millions of metrics with Graphite
Friday, November 15, 13
d3.js + cubism.js
Friday, November 15, 13
Monitoring via CloudWatchAlarm in PagerDuty
Details available in PagerDuty
Friday, November 15, 13
Lessons Learned• Use Amazon services when possible (Amazon RDS,
Amazon Redshift, Amazon Route 53)• Use SSDs where applicable• Understand your scale and your needs going forward
and invest in Reserved Instances (3 years!)• But, allow flexibility for changing needs and instance
What’s next?• Better use of Auto Scaling groups• Predictive analytics — lots of signals• Automated remediation• Heavy focus on using the right instance types for each