FME World Tour 2015 - Cloud Building Blocks and FME - Ken Bragg

Cloud Building Blocks and FME

Ken BraggEuropean Services Manager

What is the cloud?

Overview

Cloud data storage

AWS Simple Storage (S3)

AWS Aurora

AWS Relational Database Service (RDS)

AWS Elastic File System (EFS)

Other critical AWS Services

Simple Queuing Service (SQS)

AWS Lambda

Make your own Data Download Service in 10 min.

Data Considerations

● Uploading data to AWS is as slow as your internet connection.○ Aspera - 10 Mbps connection: 23 hours 100GB

● You can use AWS Import/Export to move large amounts of data into the cloud.

● Moving data around once it is in the cloud is fast.○ Within region: ~ 20MB/s○ Between regions: ~ 2 MB/s

Cloud data storage

AWS Simple Storage Service (S3) - secure, durable, highly-scalable object storage

AWS Elastic File Service – provides sharable disk storage across multiple EC2 instances.

AWS RDS - a web service that makes it easy to set up, operate, and scale a relational database in the cloud.

AWS Aurora - provides up to 5x better performance than MySQL at a price point one tenth that of a commercial database.

Secure, durable, highly-scalable object storage.

Reliability is 99.999999999%

Over 3 million request per second.

S3 is highly performant and applications can get 100’s of requests a second via S3.

$0.0330 per GB of storage. 5TB of data $160 a month.

Can store single objects up to 5TB.

AWS S3 - Overview

A simple key/value persistent object store.

Every object is identified with a key.

It is not a disk and doesn’t have a directory or folder structure. It can be presented as a folder structure though.

AWS S3 - How it works

Desktop transformers

S3Uploader

S3Downloader

S3Deleter

S3ObjectLister

FME Server S3 watcher :- Watch public S3 buckets that you do not have control of.

FME Server S3 subscriber :- Send data to S3 once a workspace has completed.

** When running workspaces on FME Cloud, expect transfer speeds > 20 MB/s instance >> S3 bucket. **

FME and S3

Amazon RDS makes it easy to set up, operate, and scale database deployments in the cloud. The following databases are supported:

Oracle

PostgreSQL (PostGIS)

Microsoft SQL Server

MySQL

AWS RDS

On-Premises vs RDS PostGIS

Machine: 16 Cores, 122GB of RAM, Linux, 1TB Storagehttps://awstcocalculator.com/

A compute service that runs your code in response to events. It makes it easy to apply compute to data as it is enters or moves through the cloud.

Run code in response to modifications to objects in Amazon S3 buckets.

Create your own back-end that operates at scale

Extend other AWS services

Lambda (beta)

Example Architecture

Build a Data Download Service in 10 Minutes!

Ken BraggEuropean Services Manager

FME Cloud & Amazon AWS Infrastructure

Easy!

State of Arkansas AGOOnce More Leading the Way

How to Deploy Quickly

1. Download sample from GitHub and make a little change

2. Source data on Amazon S3

3. Web Application on Amazon S3

4. Workspace on FME Cloud

How to Deploy Quickly

1. Download from GitHub

2. Source data on Amazon S3

3. Web Application on Amazon S3

4. Workspace on FME Cloud

DEMO

GitHub Architecture – used at Safe

Thank You!

Questions?

For more information:

Enter name and email address

Enter company name

Enter other resources