C2S Tech Tips: Rapid Prototyping

1. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 20152015, Amazon Web Services, Inc. or its affiliates. All rights reserved. C2S Tech Tips: Rapid Prototyping with C2S Solutions Architects 2. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 Agenda Goal Concept Design Data Flow Scaling Security Metrics Cost Optimization **Lets Build Something!!** 3. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 Goal Build a rapid prototype using native features available in C2S today to ingest, process, and analyze a large data set 4. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 Disclaimer The purpose of this prototype is to display the ease and speed at which new capabilities can be created using AWS The prototype displayed is not an AWS concept product this is a simple demo Some AWS partners offer products with similar (and more extensive) capabilities; these can be found in the AWS Marketplace 5. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 Concept Ingest a public data set from news media outlets (e.g., the GDELT project) Perform some processing/analysis on the data (e.g., a word cloud) Display the end-product to the customer in an easily consumable format (e.g., a map to the geographic coordinates of the article) 6. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 Design Your mileage may vary (YMMV) Internet gateway with public IP addresses for access to AWS endpoints (e.g., Amazon S3, Amazon SQS, Amazon SNS, etc.) Baked AMIs with boot scripts downloaded from S3 Amazon EC2 instance roles for resource access 7. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 Design Availability Zone SNS S3 Availability Zone SQS CloudWatch Auto Scaling group [Monitor - 1:1] Auto Scaling group [Worker Fleet - 0:n] 8. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 Workflow Add GDELT article dump to S3 bucket SNS notification sent to SQS queue on new object addition Monitor instance parses each article into an SQS message SQS queue size triggers Auto Scaling group for worker fleet Workers begin polling SQS queue, downloading articles, generating word clouds, writing to DynamoDB table Work completes, queue size decreases, Auto Scaling group workers are terminated by Auto Scaling group 9. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 Data Flow S3 DynamoDBBucket Monitor SQSSNS ASG 0:n S3 ASG 1:1 Worker Fleet SQS 10. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 Data Flow What does an SQS message look like? 11. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 Scaling Use of AWS-managed services S3, SQS, & SNS are distributed across multiple Availability Zones out of the box No infrastructure to maintain Use of Auto Scaling groups Scale on metrics, hands-off 12. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 Scaling Two Auto Scaling groups Min 1, Max 1 ensures that our monitor instance is available to process event notifications when new data appears in our S3 bucket in case of an instance loss or AZ outage Our worker fleet scales up/down based on the number of messages waiting to be processed in our SQS queue 13. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 Security Amazon EC2 instance roles Amazon S3 bucket policies Amazon SQS queue access policies 14. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 Demo Lets go to the video tape 15. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 Some Metrics Attribute Short Run Long Run Date range of articles 2015/06/08 2015/06/04-09 Total number of objects processed 183,064 867,393 Cumulative size of downloaded articles 15.1 GB 80.2 GB 16. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 Cost Optimization Generate an AWS CloudFormation template to set up and execute test runs Use a combination of resource tags and detailed billing reports (DBRs) to capture costs per run Use Amazon CloudWatch metrics to ensure adequate resource utilization 17. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 Cost Optimization Establish 4 lanes using different instance types: r3.8xlarge, m3.xlarge, m3.medium, and t2.micro. Aim for 80-100% average CPU utilization across instance types Look for bottlenecks as you move things around 18. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 Cost Optimization 19. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 Data Flow one lane S3 DynamoDBBucket Monitor SQSSNS ASG 0:n S3 ASG 1:1 Worker Fleet SQS 20. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 SQS Queue & Instances t2.micro 21. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 CPU Utilization t2.micro 22. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 Data Flow all lanes ASG 0:n ASG 1:1 ASG 0:n ASG 1:1 ASG 0:n ASG 1:1 ASG 0:n ASG 1:1 23. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 SQS Queue & Instances all lanes 24. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 CPU Utilization all lanes 25. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 Cost Comparison Short Run Instance Type Total Instances Time to Complete Amazon EC2 Cost t2.micro 100 0:36 $1.30 m3.medium 100 0:59 $7.00 m3.xl 25 0:41 $7:00 r3.8xl 1 1:32 $5.60 Prices based on us-west-2 region 26. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 Cost Comparison Long Run Instance Type Total Instances Time to Complete Amazon EC2 Cost t2.micro 100 22:53 $29.90 m3.medium 100 17:14 $120.60 m3.xl 25 16:29 $113.90 r3.8xl 1 19:53 $56.00 Prices based on us-west-2 region 27. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 Other Cost Factors Amazon DynamoDB Amazon SQS Storage Bandwidth 28. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 Final Thoughts The core of this demo took one person- week to implement Using native features of C2S services to implement applications will save time and money 29. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 References The GDELT Project http://gdeltproject.org/ A simple Python word cloud https://github.com/amueller/word_cloud Keyhole Markup Language (KML) http://www.opengeospatial.org/standards/kml 30. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 Thank You. This presentation will be loaded to SlideShare the week following the Symposium. http://www.slideshare.net/AmazonWebServices AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015