elas%csearch meetup #5files.meetup.com/7646592/20150916-human-bot-statistics-with-elk.pdf · human/bot sta%s%cs with ELK get more out of your access logs. what is elk anyway? the

elas%csearchmeetup#5welcome,beerisinthefridgeoutside:-)

human/botsta%s%cswithELKgetmoreoutofyouraccesslogs

whatiselkanyway?theelkstack

3

•elas%csearch(searching)• logstash(log“management")

• kibana(query,visualisa%on)

Whatiselkanyway?

4

Whatiselkanyway?

5

log

machine 1

log

loglogstash

log

log

machine 2

log

elasticsearch

whatweuseatJobCloudtheelkstack

6

• running13websites(www.jobs.ch,www.jobup.ch,…)•migra%ngservicesfromalegacyenvironmenttonewinfrastructure

Ourcurrentsetup

7

•newenvironmentbasedonlxccontainers

•42produc%oncontainers(app/search/cache/db/…)• similarsetupforstagingenvironment

• ahugeloadofdistributedlogfileseverywhere

Ourcurrentsetup

8

Ourcurrentsetup

9

app01 app02 app03

logstash logstash logstash

redis01

logstash

elasticsearch

• severallogfilesources• logstashforwarderoneachnode•oneredisclusterforcatchinglogs•onelogstashtopushlogstoelas%csearch

•5GBlogfilesperday• about15m“important”documentsperday

• “real%me”analysis

Ourcurrentsetup

10

howwegatherdataatJobClouddata!data!data!

11

data!data!data!

12

gathering information enhancing information store information

data!data!data!

13

• Jobcloud\TrackerBundle*writestoX-Custom-DataHTTPHeader

•HTTPHeaderiswriYentoaccesslog,alongwithsomeothers

•X-Custom-Datagetsremovedbyreverseproxy

*notopensourcedyet

data!data!data!

14

data!data!data!

15

data!data!data!

16

•uselogstashfiltertoenhancethelogfileinforma%on

• logstash-filter-tordetect*:gathersinforma%onabouttornetworks

• logstash-filter-hitclassifica%on*:addsinforma%onabouttheuser(human,bot,headless)

*notopensourcedyet

data!data!data!

17

• store4weeksofdatainproduc%on•moveeverythingelsetoAmazonS3

• runain-housekibanasetuptohandlelongtermrequests

• “measurestuff,evenifyoudon’tneeditrightnow”—itsmoreexpensivetonothavethisdata

*notopensourcedyet

whatwedowithdataatJobClouddata!nowwhat?

18

• centralisedlogviewusingkibana•deeperunderstandingforopera%ons• insightsfordevelopment

•dashboardsforpmormarke%ng

data!nowwhat?

19

•Whichkeywordwasusedtofindaspecificad?

•HowmanyusersviewedtheNovar%sjobads,reques%ngfromBasel?

•Whichbotsarethemostpainfulonsearch?

•Howodenwasanaddisplayed,andonwhichavg.posi%on?

data!nowwhat?

20

•Demo

data!nowwhat?

21

Wearehiring!:-)onemorething!

22

Thankyou!

23