San Antonio DevOps: Fluentd

Hart Hoover | @hhoover Josh O’Brien | @OBrienCommaJosh

@SADevOps

fluentd

Log all the thingsto all the things!

…in JSON!

input&{&&&file&{&&&&&path&=>&"/var/log/http.log"&&&}&}&filter&{&&&grok&{&&&&&match&=>&[&"message",&"%{IP:client}&%{WORD:method}&%{URIPATHPARAM:request}&%{NUMBER:bytes}&%{NUMBER:duration}"&]&&&}&}

Fluentd Events

Time Tag

Record

passed from source or

adding a parsed time

message routing in fluentd

JSON

#&receive&events&via&HTTP&&<source>&&type&http&&port&8888&&</source>

#&read&logs&from&a&file&&<source>&&type&tailpath&/var/log/httpd.log&format&apachetag&apache.access&&</source>

#&DOCKER&OMG&<source>&&&type&tail&&&format&json&&&path&/var/lib/docker/containers/ID/IDYjson.log&&&pos_file&/var/lib/docker/containers/ID/IDYjson.log.pos&&&tag&docker.container&&&rotate_wait&5&&&read_from_head&true&</source>

#&save&alerts&to&a&file&&<match&alert.**>&&type&file&&path&/var/log/fluent/alerts&&</match>

#&save&access&logs&to&MongoDB&&<match&apache.access>&&type&mongo&database&apache&collection&log&&</match>&

#&Post&to&IRC&<match&**>&&&type&irc&&&host&localhost&&&port&6667&&&channel&fluentd&&&nick&fluentd&&&user&fluentd&&&real&fluentd&&&message&notice:&%s&[%s]&%s&&&out_keys&tag,time,message&&&time_key&time&&&time_format&%Y/%m/%d&%H:%M:%S&&&tag_key&tag&</match>

#&forward&other&logs&to&servers&&<match&**>&&type&forward&&&<server>&&&&&&host&192.168.0.11&&&&&&weight&20&&&&</server>&&&<server>&&&&&&host&192.168.0.12&&&&&&weight&60&&&&</server></match>

#&Send&logs&to&ElasticSearch&<match&**>&&&type&elasticsearch&&&logstash_format&true&&&host&localhost&&&port&9200&&&index_name&fluentd&</match>

Client LibrariesRuby

Python Java PHP

Node.JS Scala

https://github.com/treasure-data/chef-td-agent

Install with Chef

Vagrantfile & kitchen.yml included!

https://github.com/dmytro/fluentd-cookbook

Possibly Better

https://gist.github.com/hhoover/4fceb09148a73f45136a

@SADevOpshttp://meetup.com/SanAntonioDevOps