2013 Trend Micro 25th Anniversary 2014/6/11 Amazon Web Service AWS Technical Professional.

2013 Trend Micro 25th Anniversary

112/04/10 1

Amazon Web ServiceAWS Technical Professional


112/04/10 2

AWS Technical Professional

Batch Processing


AWS Technical ProfessionalBatch Processing | Module Overview

112/04/10 3

• Describe the services provided by AWS that facilitate batch processing.

• Describe how to create auto scaling groups that are triggered by items in a queue.

•Describe the Amazon Simple Queue Service (SQS) delivery guarantees.

•Describe the performance and durability characteristics of SQS.

Module Objective


Batch Processing

• Overview

• Using AWS to Process Heavy Computational Loads

• Simplified Architecture walkthrough

112/04/10


112/04/10 5

Batch Processing| Overview

The execution of a group tasks by a computer, usually without manual intervention.

Batch Processing

We run a website that showcases independent graphic artists and displays a library of images.

Artists submit artwork in the form of digital images and we generate a watermarked version of the image to prevent image theft along with thumbnail images to display in the online catalog.Image processing requests will likely be submitted in chunks so we will store work requests in a queue and scale our computational resources (Amazon EC2 instances) automatically using Auto Scaling to meet the demand.

Our Scenario


Processing Heavy Computational Loads

• Scenarios requiring large computational capacity

• Real world examples

• Generalized architecture

112/04/10


112/04/10 7

Batch Processing| Heavy Computational Loads

• Analysis of large datasets

• Scientific simulations

• Analysis of financial markets

• Batch processing

Scenarios

• Data is easily segmented into multiple datasets that can all be processed independently and at the same time.

• Application can generally consume all compute or processing resources available.

• Pay only for resources you use.

Application Characteristics


112/04/10 8

Batch Processing| Heavy Computational Loads

• Item Classification: Define risk and compliance profile for 9 million units per day across 1+ billion item catalog

• Affiliates Program: Calculate monthly payments to millions of Amazon affiliates

Real-world examples at Amazon.com

• NASA: Uses the Amazon Simple Workflow Service (SWF) to process images received from the Mars rover Curiosity.

• DigiChalk: Uses the Amazon Simple Queue Service (SQ) to manage and transcode media files.

Real-world example outside Amazon.com


112/04/10 9

Batch Processing| Generalized Architecture

This diagram shows a generalized batch processing architecture and identifies which AWS services can be used for the various roles.


112/04/10 10

Batch Processing| Generalized Architecture – Orchestration Services

The orchestration services guide the flow of data or information through the system.


112/04/10 11

Batch Processing| Generalized Architecture – Data Services

The data and storage services serve as a repository for the data that is consumed and emitted by the batch processing application


112/04/10 12

Batch Processing| Generalized Architecture – Compute Services

The compute services are responsible for performing the work on the incoming data.


112/04/10 13

Batch Processing| Generalized Architecture – Notification Services

Notification services are used to provide notifications to observers when the work has been completed.


Batch Processing Scenario

• Simplified architecture

• Operational considerations

112/04/10


112/04/10 15

Batch Processing| Simplified Architecture

This diagram illustrates how a batch processing scenario can be implemented with AWS and how the component services would interact.


112/04/10 16

Batch Processing| Simplified Architecture – Ingestion Stage

Stage 1 –User submits work to the job manager.


112/04/10 17

Batch Processing| Simplified Architecture – Ingestion Stage

Stage 2 – Job manager queues the work request


112/04/10 18

Batch Processing| Generalized Architecture – Processing Stage

Stage 3 – Instances pull work from the queue


112/04/10 19

Batch Processing| Simplified Architecture – Processing Stage

Stage 4 – Grab work details and do the work


112/04/10 20

Batch Processing| Simplified Architecture – Storage and Publishing Stage

Stage 5 – Output the work


112/04/10 21

Batch Processing| Operational Considerations – Amazon Simple Queue Service

Preparing the Queue


112/04/10 22


Preparing the Queue


112/04/10 23


Preparing the Queue


112/04/10 24


Queue Details


112/04/10 25

Batch Processing| Operational Considerations – Auto Scaling

• Name the configuration

• Specify the EC2 instance type

• Specify the ID of the AMI to use when launching a new instance

Auto Scaling Step 1 – Create the Launch Configuration

as-create- launch-config myLaunchConfig--image-id ami-2a31bf1a--instance-type m1.smaLL

Command Line Example


112/04/10 26


• Name the group

• Specify the launch configuration (created previously)

• Specify the minimum and maximum sizes of the group

Auto Scaling Step 2 – Create the Auto Scaling Group

as-create- auto-scaling-group myAutoScalingGroup--launch-configuration myLaunchConfig--min-size 1--max-size 20



112/04/10 27


• Name the policy

• Specify the policy type

• Specify the associated auto scaling group

• Specify the auto scaling adjustment

Auto Scaling Step 3 – Create the Auto Scaling Group Policies

as-put-scaling-policy myScaleUpPolicy--type PercentChangeInCapacity--auto-scaling-group myAutoScalingGroup--adjustment 50

as-put-scaling-policy myScaleDownPolicy--type ChangeInCapacity--auto-scaling-group myAutoScalingGroup--adjustment =-1



112/04/10 28


Auto Scaling profile can be executed by:

• A manual action (via the command line tools or API)

• A time-based schedule

• Responding to an Amazon CloudWatch alarm

Auto Scaling Step 4 – Create the Auto Scaling Triggers


112/04/10 29

Batch Processing| Operational Considerations – CloudWatch

Creating the Alarm


112/04/10 30


Defining the Alarm Properties


112/04/10 31


Configuring the Alarm Actions

2013 Trend Micro 25th Anniversary 2014/6/11 Amazon Web Service AWS Technical Professional.

Documents

trend micro

queue slide

work slide

scenario slide

image processing requests

processing resources

work request slide

aws services