© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Axel Larsson, Enterprise Solutions Architect Joyjeet Banerjee, Enterprise Solutions Architect 9 April 2019 Consuming the Data Lake - Reporting, Analytics, Machine Learning
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Axel Larsson, Enterprise Solutions ArchitectJoyjeet Banerjee, Enterprise Solutions Architect
9 April 2019
Consuming the Data Lake -Reporting, Analytics, Machine
Learning
What have we learned so far
Athena?
Anti-Pattern
Everything
Query
Also an Anti-Pattern
Everything
Query
One tool to rule them all
Where do I start?
• Understand your data• Data Structure, Access patterns & characteristics,
Temperature, Cost, Size
• Know your audience• Business Users, Data Scientists, Developers
• Select the right service
Archival
In-memory Warehouse
NoSQL
Hot data Warm data Cold data
Dat
a St
ruct
ure
Low
High
Object
Search
Understand your Data
Latency
Data volumeHighLow
Request rate
Cost / GBHigh Low
Amazon ElastiCache
Amazon ES
AmazonDynamoDB Amazon S3 Amazon Glacier
Hot data Warm data Cold data
Dat
a St
ruct
ure
Low
High
Understand your Data
Latency
Data volumeHighLow
Request rate
Cost / GBHigh Low
NoSQLObject
Archival
Search
In-Memory Warehouse
Amazon Redshift
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Who is your audience?
PRIORITIES NEEDS
Creating engaging visual and narrative journeys for analytical solutionsData Visualizer
Manages data as a product. Ensures freshness and consistency of data; understands lineage and compliance needs; treats DS as customers
Data Product Manager
Monitoring for reliability, quickly diagnose deployment or availability issues
DevOps Engineer
ROLE
VisualizationDashboardsReporting
Reports – data quality, errors
Ad hoc queryingDashboards
Makes sense of data, generates and communicates insights to improve or create business processes, creates predictive ML models to support them
Data Scientist Ad hoc querying Robust ML tools
Builds scalable pipelines, transforms and loads data into structures complete with metadata that can be readily consumed by DS
Data Engineer
Ad hoc queryingQuick visualization
Vetting the priortization and ROI, funding projects, providing ongoing feedback
Business Sponsor
ReportingDashboards
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Enabling your ConsumersDashboards – Reports – Ad-Hoc Analysis – Machine Learning
DashboardsVisual Representation of key metrics that change over time• Data structure - Low• Usage - Near real-time visualization• Data temperature - Hot
Available Services:
LambdaDynamoDB
+ Streams
ElasticsearchAmazon Kinesis Firehose
Dashboards – Near Real-time
Amazon EMR
AWS Glue
OR
ETL
Data Lake
AmazonS3
Raw Bucket Transformed Data Bucket
DynamoDBUsers
EC2
Containers
Serverless
OR
OR
Web serving layer
Dashboards + Search
Amazon EMR
AWS Glue
OR
ETL
Data Lake
AmazonS3
Raw Bucket Transformed Data Bucket
DynamoDB
Amazon Kinesis Firehose
AWSLambda
Dynamo Streams
AmazonElasticsearch Users
ReportsStatic representations of data rendered at a point in time• Usage - Point in time data extraction• Data structure - High• Data temperature - Cold
Available Services:
Amazon Redshift Athena
Ad Hoc AnalysisInformation sought on an as-needed basis• Usage - Dynamic Data Querying• Data structure - Case based• Data temperature - Medium - cold
Available Services:
Amazon Redshift Athena Amazon EMR
Amazon ElasticSearch
Reports and Ad-Hoc Analysis
Amazon QuickSight
OR
Amazon Redshift
Amazon EMR
AWS Glue
OR
ETL
Data Lake
AmazonS3
Raw Bucket Transformed Data Bucket
Athena
Machine LearningData labeled with outcomes to train predication models• Usage - Machine learning data preparation• Data structure - Case based• Data temperature - Medium - cold
Available Services:
Amazon EMR
Reports and Ad-Hoc Analysis
Amazon EMR
AWS Glue
OR
ETL
Data Lake
AmazonS3
Raw Bucket Transformed Data Bucket
Amazon EMR
Users
What else?
Athena?
Processing & Analytics
Transactional & RDBMS
DynamoDB
NoSQL DB Relational DatabaseAurora
BI & Data Visualization
Kinesis Streams & Firehose
Batch
EMRHadoop, Spark,
Presto
RedshiftData Warehouse
AthenaQuery Service
AWS Batch
Predictive
Real-time
AWS LambdaApache Storm
on EMR
Apache Flinkon EMR
Spark Streaming on EMR
ElasticsearchService
Kinesis Analytics, Kinesis Streams
ElastiCache DAX
Thank you!
Related Documents
