Copyright (c) 2018 by Amazon.com, Inc. or its affiliates. AWS Ops Automator is licensed under the terms of the Amazon Software License available at https://aws.amazon.com/asl/ AWS Ops Automator AWS Implementation Guide Arie Leeuwesteijn Mahmoud ElZayet Ruald Andreae July 2017 Last updated: April 2018 (see revisions)
28
Embed
AWS Ops Automator - Amazon S3 · Amazon Web Services – AWS Ops Automator April 2018 Page 4 of 28 scheduling, and configuration management. Automating these tasks can help increase
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Copyright (c) 2018 by Amazon.com, Inc. or its affiliates.
AWS Ops Automator is licensed under the terms of the Amazon Software License available at
Amazon Web Services – AWS Ops Automator April 2018
Page 9 of 28
AWS CloudFormation templates, AWS Identity and Access Management roles, an Amazon
Simple Storage Service (Amazon S3) bucket, and an Amazon Simple Notification Service
(Amazon SNS) topic.
Automated Deployment Before you launch the automated deployment, please review the architecture, configuration,
security, and other considerations discussed in this guide. Follow the step-by-step
instructions in this section to configure and deploy an AWS Ops Automator into your
account.
Time to deploy: Approximately 30 minutes
What We’ll Cover The procedure for deploying this architecture on AWS consists of the following steps. For
detailed instructions, follow the links for each step.
Step 1. Launch the AWS Ops Automator Stack in the Primary Account
• Launch the AWS CloudFormation template into your primary AWS account.
• Enter values for required parameters: Stack Name.
• Review the other template parameters, and adjust if necessary.
Step 2. Launch the Task Template in the Primary Account
• Launch the applicable task-configuration AWS CloudFormation template into the
primary account.
• Review the template parameters, and adjust if necessary.
Step 3. Launch the Role Template in the Secondary Account
• Launch the applicable role AWS CloudFormation template into the secondary account
with applicable resources.
• Enter values for required parameters: Stack Name.
Step 4. Tag Your Resources
• Apply the custom tag to applicable resources.
Amazon Web Services – AWS Ops Automator April 2018
Page 10 of 28
Step 1. Launch the AWS Ops Automator Stack in the Primary Account This automated AWS CloudFormation template deploys the AWS Ops Automator in your
primary account. Launch this template using an AWS Identity and Access Management
(IAM) role specifically created for this purpose. For more information, see the Security
section.
Note: You are responsible for the cost of the AWS services used while running this solution. See the Cost section for more details. For full details, see the pricing webpage for each AWS service you will be using in this solution.
1. Sign in to the AWS Management Console and click the button to
the right to launch the ops-automator AWS CloudFormation
template.
You can also download the template as a starting point for your own implementation.
2. The template is launched in the US East (N. Virginia) Region by default. To launch this
solution in a different AWS Region, use the region selector in the console navigation bar.
Note: This solution uses AWS Lambda, Amazon DynamoDB, and Amazon CloudWatch, which are currently available in specific AWS Regions only. Therefore, you must launch this solution an AWS Region where these services are available. 2
3. On the Select Template page, keep the default setting for Choose a Template and
select Next.
4. Under Parameters, review the parameters for the template and modify them as
necessary. This solution uses the following default values.
Parameter Default Description
Ops Automator Tag
Name
OpsAutomatorTaskList The tag key (name) that identifies applicable resources. The
tag value will contain the list of tasks to be performed on
tagged resources. See Step 4 for detailed information.
Clean up task
tracking table?
Yes Choose whether to clean the task tracking table.
Keep failed tasks? Yes Choose whether to store failed tasks in the Amazon
DynamoDB table.
Schedule active? Yes Choose whether to activate the scheduling task feature.
2 For the most current AWS service availability by region, see https://aws.amazon.com/about-aws/global-infrastructure/regional-
Amazon Web Services – AWS Ops Automator April 2018
Page 16 of 28
Appendix A: Task-Configuration Templates The AWS Ops Automator automatically generates a separate AWS CloudFormation
template for each action. Review the template parameters and modify them as necessary.
Common Template Parameters Each action-specific template has a set of common parameters, as well as parameters that
are specific to the applicable action. Review the common parameters and modify them as
necessary.
Parameter Default Description
Task description <Optional input> Description of the task. For example, Create a snapshot
every 30 minutes.
Task interval <Requires input> Enter the scheduled expression (Cron syntax) that specifies when
to run the task. For example, enter 0/30**** to run the task
every 30 minutes.
Tag filter <Optional input> Filter used to select resources. You can use this instead of adding
the task name to the list of values in the
OpsAutomatorTaskList tag. For example, Owner=DBAdmin,
Stack=Test. For more information on tag filters, see Appendix
D.
Note: For actions that can delete or terminate resources, you cannot use “*” as the name of the tag in the filter.
Timeout 60 The number of minutes the solution waits for a task to complete
before reporting timing out.
Note: This parameter will only show for actions that the solution checks for completion.
Regions <Default region> List of regions where the task will run. For example, us-east-
1, eu-west-1.
Note: Use this parameter for actions that use regional resources. This parameter will not show for actions that use global resources. For example, IAM and Amazon S3.
This account Yes Select Yes to allow the task to select resources in this account.
Amazon Web Services – AWS Ops Automator April 2018
Page 17 of 28
Parameter Default Description
Cross account roles <Optional input> Comma-delimited list of cross-account roles used by the task. For
example,
arn:aws:iam::111122223333:role/CreateSnaphotRole.
Note: Enter the secondary account CrossAccountRoleArn value(s) in this parameter. For customers who use a cross-account role ARN file, leave this parameter blank. For more information on the cross-account ARN file, see Appendix B.
Timezone UTC The time zone used for scheduling the task
Task enabled Yes Select No to temporarily disable execution of the task
Enable debugging No Choose whether to log detailed debugging information
Amazon EC2 Create Snapshot Template The create snapshot template enables the solution to automatically create snapshots of
Amazon Elastic Block Store (Amazon EBS) volumes attached to Amazon Elastic Compute
Cloud (Amazon EC2) instances.
Review the action-specific parameters for the template and modify them as necessary. This
action template uses the following default values.
Parameter Default Description
Copy root volume Yes Create a snapshot of a root Amazon EC2 instance volume
Copy data volumes Yes Create snapshots of Amazon EC2 instance data volumes
Copied instance tags <Optional input> Enter a tag filter to copy tags from the instance to the snapshot. For
example, enter * to copy all tags from the instance to the snapshot.
For more information on tag filters, see Appendix D.
Copied volume tags <Optional input> Enter a tag filter to copy tags from the volume to the snapshot. For
example, enter * to copy all tags from the volume to the snapshot.
For more information on tag filters, see Appendix D.
Snapshot tags <Optional input> Tags that will be added to snapshots. Use a list of
tagname=tagvalue pairs. For example, if you create a task called
DeleteEC2Snapshots, you can enter a value of
OpsAutomatorTaskList=DeleteEC2Snapshots in this
parameter to allow the AWS Ops Automator to delete the snapshot
based on the parameters specified in the DeleteEC2Snapshots
task.
Set snapshot name Yes Set the name of the snapshot
Snapshot name prefix <Optional input> Prefix of the snapshot name
Amazon Web Services – AWS Ops Automator April 2018
Page 18 of 28
Amazon EC2 Delete Snapshot Template The delete snapshot template enables the solution to automatically delete snapshots of
Amazon Elastic Block Store (Amazon EBS) volumes attached to Amazon Elastic Compute
Cloud (Amazon EC2) instances older than a customer-defined number of days. Or, customers
can configure this action to keep only the latest snapshots.
Review the action-specific parameters for the template and modify them as necessary. This
action template uses the following default values.
Parameter Default Description
Retention days <Requires
input>
The retention period in days. Set this parameter to 0 to use
Retention count.
Retention count <Requires
input>
The number of snapshots to retain for a volume. The maximum
allowed value is 1000. Set this parameter to 0 to use Retention
days.
Note: If both Retention days and Retention count are set to 0, the solution will return an error. You must only specify one.
Amazon EC2 Copy Snapshot Template The copy snapshot template enables the solution to automatically copy snapshots of Amazon
Elastic Block Store (Amazon EBS) volumes attached to Amazon Elastic Compute Cloud
(Amazon EC2) instances from one AWS Region to another.
Review the action-specific parameters for the template and modify them as necessary. This
action template uses the following default values.
Parameter Default Description
Copied tags <Optional input> Enter a tag filter to copy tags from the source
snapshot to the copied snapshot. For example,
enter * to copy all tags from the source snapshot
to the copied snapshot. For more information on
tag filters, see Appendix D.
Snapshot tags <Optional input> Tags to add to the copied snapshot. Use a list of
tagname=tagvalue pairs.
Destination
region
<Optional input> AWS Region to copy the snapshot to
Tag name for
copied snapshots
Ec2CopySnapshot:SnapshotCopied Tag to add to the copied snapshot to show that it
was copied. We recommend that you use the
default tag name unless you already use the same
name for other purposes.
Amazon Web Services – AWS Ops Automator April 2018
Page 19 of 28
Parameter Default Description
Encrypted No To enable encryption of the copied snapshot, select
Yes.
KMS Key ID <Optional input> The full ARN of an AWS Key Management Service
(AWS KMS) customer master key (CMK) to use
when creating the snapshot copy. If you do not
specify a CMK, the solution will use the default
CMK for Amazon EBS.
For instructions on how to find a key ARN, see the
AWS KMS Developer Guide.
Note: If you specify an alternative CMK, it must exist in the same AWS Region that the snapshots are copied to. Also, the account or role that the Ops Automator uses, or an applicable cross-account role, must have permission to use the key. For more information, see Appendix C.
Amazon Redshift Copy Snapshot Template The Amazon Redshift copy snapshot template enables the solution to automatically copy
snapshots of Amazon Redshift clusters from one AWS Region to another.
Review the action-specific parameters for the template and modify them as necessary. This
action template uses the following default values.
Parameter Default Description
Copied cluster tags <Optional input> Enter a tag filter to copy tags from the instance to the snapshot.
For example, enter * to copy all tags from the instance to the
snapshot. For more information on tag filters, see Appendix D.
Snapshot tags <Optional input> Tags to add to snapshot. Use a list of tagname=tagvalue
pairs. For example, if you create a task called DeleteRedShiftSnapshots, you can enter a value of
OpsAutomatorTaskList=DeleteRedShiftSnapshots in
this parameter to allow the AWS Ops Automator to delete the copied snapshot based on the parameters specified in the DeleteRedShiftSnapshots task.
Grant restore access
to accounts
<Optional input> Comma-delimited list of accounts that can restore the snapshot.
For example, 777788889999, 000000000000.
Amazon Redshift Delete Snapshot Template The Amazon Redshift delete snapshot template enables the solution to automatically delete
snapshots of Amazon Redshift clusters older than a customer-defined number of days. Or,
customers can configure this action to keep only the latest snapshots.
Review the action-specific parameters for the template and modify them as necessary. This
action template uses the following default values.
Amazon Web Services – AWS Ops Automator April 2018
Page 23 of 28
Tag filter format Description
Note: For actions that can delete or terminate resources, you cannot use “*” as the name of the tag in the filter.
A* Tag keys that start with “A”
*A* Tag keys that contain “A”
*A Tag keys that end with “A”
\.*\d$ Tag keys that end with a digit
A=B Tag keys “A” with value “B”
A=B* Tag keys “A” with a value that starts with “B”
*=B Any tag with a value “B”
*=B* Any tag with a value that starts with “B”
*=\.*\d$ Any tag with a value that ends with a digit
A=B,C=D Tag keys “A” with value “B” or tag keys “C” with a value “D”
The following table gives examples of different tag filters and the resulting AWS Ops
Automator action.
Tag filter AWS Ops Automator Action
Owner=DBAdmin Perform the task on resources with the Owner tag key with a value of
DBAdmin.
Owner Perform the task on resources with the Owner tag key with any value.
*=DBAdmin Perform the task on resources with any tag key with a value of DBAdmin.
*test Perform the task on resources with a tag key that ends with test.
test* Perform the task on resources with a tag key that starts with test.
*=*test Perform the task on resources with any tag key with a value that ends
with test.
*=test* Perform the task on resources with any tag key with a value that starts
with test.
Owner=DBAdmin,Stack=Test Perform the task on resources with the Owner tag key with a value of
DBAdmin or resources with the Stack tag key with a value of Test.
Appendix E: Log Files The AWS Ops Automator creates a log group that contains the default AWS Lambda log
files and a log group that contains the following log files:
Amazon Web Services – AWS Ops Automator April 2018
Page 24 of 28
• AutomatorMain-yyyymmdd - Logs the output from the Lambda function that handled
the event.
• ScheduleHandler-yyyymmdd - Logs the output from Lambda functions that handle
time-based tasks.
• TaskTrackingHandler-yyyymmdd - Logs the output from Lambda functions that
handle event-based tasks.
• SelectResourcesHandler-<taskname>-yyyymmdd – Logs the output from the
Lambda function that selects resources for task execution.
• <Taskname>-yymmddhhmm-<unique task id> - Logs the output of the Lambda
function that executes the action. The unique task ID correspondents to the unique id for each executed task in the task tracking Amazon DynamoDB table.
• CompletionHandler-yyyymmdd – Logs the output from the Lambda function that
checks whether the action completed.
Messages This solution also logs error, warning, and debugging messages. Each message has the format: yyyy-mm-dd – hh:mm:ss.mmm - <type> - <text>. The type can have the
value INFO for informational messages, DEBUG for detailed debugging messages, WARNING for warning messages, or ERROR for error messages.
Warning and error messages are sent to an email address using Amazon Simple Notification
Service (Amazon SNS). The Amazon SNS topic is located in the AWS Ops Automator stack
output named IssueSNSTopic.
Appendix F: Sample Deployment Configuration The AWS Ops Automator enables customers to perform a sequence of tasks on resources in
their accounts. The following section shows how to configure a sequence of tasks that will
take snapshots of all Amazon Elastic Block Store (Amazon EBS) volumes at 1:00 AM daily
and retain the snapshots for seven days.
First, deploy the ops-automator template in your primary account. For more information,
see Step 1. Once the template is deployed, launch the Ec2DeleteSnapshot and
Ec2CreateSnapshot role templates in any secondary account(s) with applicable EBS
volumes. For more information, see Step 3. Copy the cross-account role Amazon Resource
Name(s). To perform tasks on resources in a large number of secondary accounts, save the
ARNs in a text file. For more information, see Appendix B.
Amazon Web Services – AWS Ops Automator April 2018
Page 25 of 28
Next, deploy the Ec2DeleteSnapshot configuration template in the primary account
using the following values:
Parameter Value
Stack name Delete7
Task description Delete a snapshot after 7 days
Task interval 0 2 * * ?
Tag filter (Leave blank)
Regions (Enter the applicable AWS Region(s). For example, us-
east-1, eu-west-1)
This account Yes
Cross-account roles (Enter the cross-account ARNs or leave blank to use a text
file.)
Timezone UTC
Task enabled Yes
Enable debugging No
Retention days 7
Retention count 0
Note: Set this parameter to 0 to retain snapshots
using Retention days.
Then, deploy the Ec2CreateSnapshot configuration template in the primary account
using the following values:
Parameter Value
Stack name BackupDaily
Task description Create a snapshot at 1 am daily
Task interval 0 1 * * ?
Tag filter (Leave blank)
Regions (Enter the applicable AWS Region(s). For example, us-
east-1, us-west-2)
This account Yes
Cross-account roles (Enter the cross-account ARNs or leave blank to use a text
file.)
Timezone UTC
Task enabled Yes
Amazon Web Services – AWS Ops Automator April 2018
Page 26 of 28
Parameter Value
Enable debugging No
Copy root volume Yes
Copy data volumes Yes
Copied instance tags *
Copied volume tags *
Snapshot tags OpsAutomatorTaskList=Delete7
Set snapshot name Yes
Snapshot name prefix ops-auto
When completely deployed using the configuration above, the AWS Ops Automator will do
the following:
1. Create a snapshot of any EBS volumes attached to Amazon Elastic Cloud Compute
(Amazon EC2) instances with the BackupDaily tag at 1 am. If a snapshot already exists
for the volume, the solution will take an incremental snapshot.
2. Copy the Amazon EC2 instance and volume tags to the snapshot.
3. Attach a new tag (OpsAutomatorTaskList=Delete7) to the snapshot. This tag is used
to identify applicable snapshots for deletion after the retention period (seven days).
4. After seven days, the solution will delete the snapshot.
Appendix G: Troubleshooting Capacity Issues When the automated process is triggered, an instance of the solution’s main AWS Lambda
function must select all applicable resources within a five-minute period. If the solution fails
to process a task for a large number of resources, such as copying or deleting a large amount
of Amazon Elastic Block Store (Amazon EBS) snapshots, it is often because the Lambda
function does not have enough memory to select those resources within the timeout period.
The primary solution template includes a Lambda size (MB) parameter, which allows you
to increase the memory of the main AWS Ops Automator Lambda function.
You can review the logs for your Lambda function in Amazon CloudWatch to see if
insufficient memory is causing execution issues. In the AWS Lambda console, choose the
function named <stackname>-SchedulerDefault. In the Monitoring tab under
Invocation duration, choose Jump to Logs to view the function’s log files directly in the
Amazon CloudWatch console. If there are entries that show execution times close to five
minutes, or executions that were timed out by the five-minute limit, then you need to increase
Amazon Web Services – AWS Ops Automator April 2018
Page 27 of 28
the memory size of the Lambda function. Use the Lambda size (MB) parameter in the
primary solution template to adjust this value.
Appendix H: Collection of Anonymous Data This solution includes an option to send anonymous usage data to AWS. We use this data to
better understand how customers use this solution and related services and products. When
enabled, the following information is collected and sent to AWS:
• Solution ID: The AWS solution identifier
• Unique ID (UUID): Randomly generated, unique identifier
• Timestamp: Data-collection timestamp
• Copy Snapshot Task Data: The number of snapshots copied
• Create Snapshot Task Data: The number of snapshots taken and the total size
• Delete Snapshot Task Data: The number of snapshots deleted
• Amazon DynamoDB Set Capacity Task Data: The number of old and new read
and write capacity units and global secondary indexes for DynamoDB tables
Note that AWS will own the data gathered via this survey. Data collection will be subject to
the AWS Privacy Policy. To opt out of this feature, set the Send Anonymous Usage Data
parameter to No.
Send Us Feedback We welcome your questions and comments. Please post your feedback on the AWS
Solutions Discussion Forum.
You can visit our GitHub repository to download the templates and scripts for this solution,