Cloudlytics - Analyze S3 & CloudFront Logs

1

Wh

y L

og

Pro

ce

ssin

g

1. Hidden Information in Logs

2. Traditional Way

3. With Cloudlytics

Use

Ca

se

s

1. ISVs

2. e-Learning

3. Media

4. Other Content Distribution Re

po

rts

1. Geographical Distribution

2. Popular Downloads

3. SPAM Attacks

4. Streaming Pattern

Co

min

g S

oo

n

1. Customizable Reports

2. SPAM Attack Alerts

3 .External Plugins

Information Hidden in S3 & CloudFront Logs

3

-

Object Details Download Status

Download /Streaming Time

Number of

Bytes

Transferred

Details about Edge Locations

IP Address of the Requester

Referrer Link Time Taken to Download Object(S3)

Details about Play, Pause, Stop (Streaming Content on CloudFront)

Uncover the Hidden Information

• Generating logs for Amazon S3 &

CloudFront logs is optional

• Log files are stored in S3 buckets

• CloudFront Log files are compressed

and stored in .gz format

• A log file is generated every hour, but we

have seen varied patterns with multiple files

generated every hour

• No ready solution from AWS to process

these log files

4

S3 logs CloudFront

logs

Logs Stored

in S3

Logs Analyzed by

CLOUDLYTICS

Image Courtesy: www.fao.org

Traditional Log Processing

• Extract data from the source using an ETL tool

• Transform data and load in a data-warehouse

• Takes days to process a few GBs of

log files using traditional hardware

• Alternately use a Hadoop distribution

to process logs

• But maintaining a Hadoop cluster is a huge overhead

5

Log Processing with Cloudlytics

• Cloudlytics - Analyze your Amazon S3 &

CloudFront Logs

• Detailed analysis of your S3 & CloudFront

access patterns

• Dynamic Graphs to get a 360 degree

perspective

• Scalable & Reliable service built using

Amazon EMR & RedShift

• Pay as you go

6

Log Processing – Big Data Approach

• Cloudlytics extracts log files stored

in S3 buckets

• Processes the log files to

transform information

• Stores the processed data in

a data-warehouse

• Graphical and tabular reports generated from data-warehouse

7

8

Cloudlytics Use Cases

Independent Software Vendors (ISVs)

• ISVs distribute downloadable software to

end users across the globe

• ISVs need to ensure that downloads are

fast, helps improve user experience

• ISVs need to track each download for

success and failure

• Identify broken links on the website, helps improve user

experience

• Identify the most popular downloads, focus on popular products

• Identify spam attacks, help reduce bandwidth costs

9

E-Learning Companies

• E-Learning companies distribute educational

content in multiple formats

(ebooks, audio, video etc.)

• Figure out the most popular content

• Figure out end user engagement by querying

number of events per request (Play, Pause, Stop)

• Get a breakup of requests by Operating System and Devices,

develop content creation for specific platforms

10

Image Courtesy: http://www.elifescience.in/images

Media Organizations

• Large number of media assets

available online

• Content does not go any changes

during it’s life cycle

• Some content is extremely popular

while others do not get any views

• Identify the most popular content,

set caching mechanism

• Figure out end user engagement by querying number of events per

request (Play, Pause, Stop)

• Identify the edge locations from which the content is downloaded

the most, optimize billing using CloudFront pricing Class

11

Cloudlytics REPORTS

12

Which Reports Resonate

with Your Business Needs?

Geographic Reports

13

Browser & OS Statistics

14

Detail IP Monitoring

15

Timeline Charts

16

The TOP 10

17

Edge Location Traffic

18

Get Started in 3 Easy Steps

19

Configure your Log

buckets

Register for Free

Analyze & Generate

Reports

Pricing

• No upfront costs

• Register for free and analyze upto 25MB logs/month FREE

• Pay only for the amount of logs you subscribe

20

To sum it Up

21

Identifying Popular

Downloads & Streams

Get Geographical Distribution

of Downloads & Streams

Improve User Experience by

Calculating & Optimizing Latencies

Identify Edge Location

Traffic and optimize AWS Billing

Identify Spam

Attacks

Get Streaming Pattern

for Video Content

Our Global User Reach

22

80+ Users and

Counting!

Coming Soon!

23

Image Courtesy: http://www.trophies.com/coming-soon/

Visit :Cloudytics.com

Contact: [email protected]

Call: +1 212 960 8532

Follow Us On :

Our Blog : http://blog.blazeclan.com/

Thank You