elas%csearchmeetup#5welcome,beerisinthefridgeoutside:-)
human/botsta%s%cswithELKgetmoreoutofyouraccesslogs
whatiselkanyway?theelkstack
3
•elas%csearch(searching)• logstash(log“management")
• kibana(query,visualisa%on)
Whatiselkanyway?
4
Whatiselkanyway?
5
log
machine 1
log
loglogstash
log
log
machine 2
log
elasticsearch
whatweuseatJobCloudtheelkstack
6
• running13websites(www.jobs.ch,www.jobup.ch,…)•migra%ngservicesfromalegacyenvironmenttonewinfrastructure
Ourcurrentsetup
7
•newenvironmentbasedonlxccontainers
•42produc%oncontainers(app/search/cache/db/…)• similarsetupforstagingenvironment
• ahugeloadofdistributedlogfileseverywhere
Ourcurrentsetup
8
Ourcurrentsetup
9
app01 app02 app03
logstash logstash logstash
redis01
logstash
elasticsearch
• severallogfilesources• logstashforwarderoneachnode•oneredisclusterforcatchinglogs•onelogstashtopushlogstoelas%csearch
•5GBlogfilesperday• about15m“important”documentsperday
• “real%me”analysis
Ourcurrentsetup
10
howwegatherdataatJobClouddata!data!data!
11
data!data!data!
12
gathering information enhancing information store information
data!data!data!
13
• Jobcloud\TrackerBundle*writestoX-Custom-DataHTTPHeader
•HTTPHeaderiswriYentoaccesslog,alongwithsomeothers
•X-Custom-Datagetsremovedbyreverseproxy
*notopensourcedyet
data!data!data!
14
data!data!data!
15
data!data!data!
16
•uselogstashfiltertoenhancethelogfileinforma%on
• logstash-filter-tordetect*:gathersinforma%onabouttornetworks
• logstash-filter-hitclassifica%on*:addsinforma%onabouttheuser(human,bot,headless)
*notopensourcedyet
data!data!data!
17
• store4weeksofdatainproduc%on•moveeverythingelsetoAmazonS3
• runain-housekibanasetuptohandlelongtermrequests
• “measurestuff,evenifyoudon’tneeditrightnow”—itsmoreexpensivetonothavethisdata
*notopensourcedyet
whatwedowithdataatJobClouddata!nowwhat?
18
• centralisedlogviewusingkibana•deeperunderstandingforopera%ons• insightsfordevelopment
•dashboardsforpmormarke%ng
data!nowwhat?
19
•Whichkeywordwasusedtofindaspecificad?
•HowmanyusersviewedtheNovar%sjobads,reques%ngfromBasel?
•Whichbotsarethemostpainfulonsearch?
•Howodenwasanaddisplayed,andonwhichavg.posi%on?
data!nowwhat?
20
•Demo
data!nowwhat?
21
Wearehiring!:-)onemorething!
22
Thankyou!
23
