Top Banner
elas%csearch meetup #5 welcome, beer is in the fridge outside :-)
23

elas%csearch meetup #5files.meetup.com/7646592/20150916-human-bot-statistics-with-elk.pdf · human/bot sta%s%cs with ELK get more out of your access logs. what is elk anyway? the

May 10, 2018

Download

Documents

ngodiep
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: elas%csearch meetup #5files.meetup.com/7646592/20150916-human-bot-statistics-with-elk.pdf · human/bot sta%s%cs with ELK get more out of your access logs. what is elk anyway? the

elas%csearchmeetup#5welcome,beerisinthefridgeoutside:-)

Page 2: elas%csearch meetup #5files.meetup.com/7646592/20150916-human-bot-statistics-with-elk.pdf · human/bot sta%s%cs with ELK get more out of your access logs. what is elk anyway? the

human/botsta%s%cswithELKgetmoreoutofyouraccesslogs

Page 3: elas%csearch meetup #5files.meetup.com/7646592/20150916-human-bot-statistics-with-elk.pdf · human/bot sta%s%cs with ELK get more out of your access logs. what is elk anyway? the

whatiselkanyway?theelkstack

3

Page 4: elas%csearch meetup #5files.meetup.com/7646592/20150916-human-bot-statistics-with-elk.pdf · human/bot sta%s%cs with ELK get more out of your access logs. what is elk anyway? the

•elas%csearch(searching)• logstash(log“management")

• kibana(query,visualisa%on)

Whatiselkanyway?

4

Page 5: elas%csearch meetup #5files.meetup.com/7646592/20150916-human-bot-statistics-with-elk.pdf · human/bot sta%s%cs with ELK get more out of your access logs. what is elk anyway? the

Whatiselkanyway?

5

log

machine 1

log

loglogstash

log

log

machine 2

log

elasticsearch

Page 6: elas%csearch meetup #5files.meetup.com/7646592/20150916-human-bot-statistics-with-elk.pdf · human/bot sta%s%cs with ELK get more out of your access logs. what is elk anyway? the

whatweuseatJobCloudtheelkstack

6

Page 7: elas%csearch meetup #5files.meetup.com/7646592/20150916-human-bot-statistics-with-elk.pdf · human/bot sta%s%cs with ELK get more out of your access logs. what is elk anyway? the

• running13websites(www.jobs.ch,www.jobup.ch,…)•migra%ngservicesfromalegacyenvironmenttonewinfrastructure

Ourcurrentsetup

7

Page 8: elas%csearch meetup #5files.meetup.com/7646592/20150916-human-bot-statistics-with-elk.pdf · human/bot sta%s%cs with ELK get more out of your access logs. what is elk anyway? the

•newenvironmentbasedonlxccontainers

•42produc%oncontainers(app/search/cache/db/…)• similarsetupforstagingenvironment

• ahugeloadofdistributedlogfileseverywhere

Ourcurrentsetup

8

Page 9: elas%csearch meetup #5files.meetup.com/7646592/20150916-human-bot-statistics-with-elk.pdf · human/bot sta%s%cs with ELK get more out of your access logs. what is elk anyway? the

Ourcurrentsetup

9

app01 app02 app03

logstash logstash logstash

redis01

logstash

elasticsearch

• severallogfilesources• logstashforwarderoneachnode•oneredisclusterforcatchinglogs•onelogstashtopushlogstoelas%csearch

Page 10: elas%csearch meetup #5files.meetup.com/7646592/20150916-human-bot-statistics-with-elk.pdf · human/bot sta%s%cs with ELK get more out of your access logs. what is elk anyway? the

•5GBlogfilesperday• about15m“important”documentsperday

• “real%me”analysis

Ourcurrentsetup

10

Page 11: elas%csearch meetup #5files.meetup.com/7646592/20150916-human-bot-statistics-with-elk.pdf · human/bot sta%s%cs with ELK get more out of your access logs. what is elk anyway? the

howwegatherdataatJobClouddata!data!data!

11

Page 12: elas%csearch meetup #5files.meetup.com/7646592/20150916-human-bot-statistics-with-elk.pdf · human/bot sta%s%cs with ELK get more out of your access logs. what is elk anyway? the

data!data!data!

12

gathering information enhancing information store information

Page 13: elas%csearch meetup #5files.meetup.com/7646592/20150916-human-bot-statistics-with-elk.pdf · human/bot sta%s%cs with ELK get more out of your access logs. what is elk anyway? the

data!data!data!

13

• Jobcloud\TrackerBundle*writestoX-Custom-DataHTTPHeader

•HTTPHeaderiswriYentoaccesslog,alongwithsomeothers

•X-Custom-Datagetsremovedbyreverseproxy

*notopensourcedyet

Page 14: elas%csearch meetup #5files.meetup.com/7646592/20150916-human-bot-statistics-with-elk.pdf · human/bot sta%s%cs with ELK get more out of your access logs. what is elk anyway? the

data!data!data!

14

Page 15: elas%csearch meetup #5files.meetup.com/7646592/20150916-human-bot-statistics-with-elk.pdf · human/bot sta%s%cs with ELK get more out of your access logs. what is elk anyway? the

data!data!data!

15

Page 16: elas%csearch meetup #5files.meetup.com/7646592/20150916-human-bot-statistics-with-elk.pdf · human/bot sta%s%cs with ELK get more out of your access logs. what is elk anyway? the

data!data!data!

16

•uselogstashfiltertoenhancethelogfileinforma%on

• logstash-filter-tordetect*:gathersinforma%onabouttornetworks

• logstash-filter-hitclassifica%on*:addsinforma%onabouttheuser(human,bot,headless)

*notopensourcedyet

Page 17: elas%csearch meetup #5files.meetup.com/7646592/20150916-human-bot-statistics-with-elk.pdf · human/bot sta%s%cs with ELK get more out of your access logs. what is elk anyway? the

data!data!data!

17

• store4weeksofdatainproduc%on•moveeverythingelsetoAmazonS3

• runain-housekibanasetuptohandlelongtermrequests

• “measurestuff,evenifyoudon’tneeditrightnow”—itsmoreexpensivetonothavethisdata

*notopensourcedyet

Page 18: elas%csearch meetup #5files.meetup.com/7646592/20150916-human-bot-statistics-with-elk.pdf · human/bot sta%s%cs with ELK get more out of your access logs. what is elk anyway? the

whatwedowithdataatJobClouddata!nowwhat?

18

Page 19: elas%csearch meetup #5files.meetup.com/7646592/20150916-human-bot-statistics-with-elk.pdf · human/bot sta%s%cs with ELK get more out of your access logs. what is elk anyway? the

• centralisedlogviewusingkibana•deeperunderstandingforopera%ons• insightsfordevelopment

•dashboardsforpmormarke%ng

data!nowwhat?

19

Page 20: elas%csearch meetup #5files.meetup.com/7646592/20150916-human-bot-statistics-with-elk.pdf · human/bot sta%s%cs with ELK get more out of your access logs. what is elk anyway? the

•Whichkeywordwasusedtofindaspecificad?

•HowmanyusersviewedtheNovar%sjobads,reques%ngfromBasel?

•Whichbotsarethemostpainfulonsearch?

•Howodenwasanaddisplayed,andonwhichavg.posi%on?

data!nowwhat?

20

Page 21: elas%csearch meetup #5files.meetup.com/7646592/20150916-human-bot-statistics-with-elk.pdf · human/bot sta%s%cs with ELK get more out of your access logs. what is elk anyway? the

•Demo

data!nowwhat?

21

Page 22: elas%csearch meetup #5files.meetup.com/7646592/20150916-human-bot-statistics-with-elk.pdf · human/bot sta%s%cs with ELK get more out of your access logs. what is elk anyway? the

Wearehiring!:-)onemorething!

22

Page 23: elas%csearch meetup #5files.meetup.com/7646592/20150916-human-bot-statistics-with-elk.pdf · human/bot sta%s%cs with ELK get more out of your access logs. what is elk anyway? the

Thankyou!

23