Altitude SF 2017: Logging at the edge

Logging at the Edge

Chris Jackel Solutions Architect | Fastly

Mehran Nazir | Google

• How Fastly’s logs work What to log

• Set up a service and log endpoint

• What to Log

• JSON logging

• Fun with logs

• Serverless Applicationless Edge Logging

Agenda

• Log endpoint configured

• Customized Fastly log string

• JSON formatted

• BigQuery data import

Expectations at the end

Under the Hood

159847 TxHeader c Connection: keep-alive

159847 TxHeader c X-Served-By: cache-jfk8125-JFK

159847 TxHeader c X-Cache: HIT

159847 TxHeader c X-Cache-Hits: 2

159847 TxHeader c X-Timer: S1489171719.109755,VS0,VE0

159847 VCL_call c log deliver

159847 VCL_call c log

159847 VCL_Log c syslog 4rqBj4oy1YChTPgdapiWS4 gcs-test :: {"client.ip":"207.38.219.230","req.request":"GET","req.http.host":"altitude-fastly.s3-website-us-east-1.amazonaws.com","req.request":"GET","req.url":"\/","req.bytes_read":94,"resp.status":200,"resp.bytes_written":545,"resp.http.X-Cache":"HIT","fastly_info.state":"HIT","time.start.usec":1489171719109755,"time.start.iso8601":"2017-03-10 18:48:39","time.end.usec":1489171719109876,"time.elapsed.usec":121}

159847 VCL_return c deliver

UI/API Control plane

Varnish Logging Aggregator

2 million LPS

Accessing Workshop Accounts

www.loggingworkshop.com

Demo: Create a Service

Demo: Logentries

What to Log

https://docs.fastly.com/guides/streaming-logs/custom-log-formats#version-2-log-format

resp.http.Set-Cookie

server.datacenter

server.identity

server.region

if(req.http.Fastly-FF,"1","0")

resp.http.X-Cache

fastly_info.state

Useful things to log

ERRORHITHIT-CLUSTERHIT-REFRESHHIT-STALEHIT-STALE-CLUSTERHIT-STALE-WAIT-CLUSTERHIT-SYNTHHIT-WAITHIT-WAIT-CLUSTERHITPASS

fastly_info.state

MISSMISS-CLUSTERMISS-WAITMISS-WAIT-CLUSTERPASSPASS-WAIT

regsub(fastly_info.state,"-.*","")

obj.hits

geoip.continent_code

geoip.country_code

geoip.region(codeforregionwithincountry)

req.restarts

Useful things to log

JSON Logging

Somewhat limited support for Apache log format variables

VCL variables can just dropped in

No (good) support for string literals, let alone things like JSON

Log format v1

Full support for Apache log format variables

Some will never return a value

VCL variables (or code) through Fastly specific extension: %{...}V

Supports string literals, and thus JSON is much easier to do

Default for all newly created endpoints

Legacy endpoints can be upgraded to v2

Log format v2

Log format v2 in UI

• <134>2017-03-19T12:47:40Zcache-jfk8127foobarlogs2[39694]:

{"client_ip":"207.237.138.218","req_url":"/"}

Results

Originally hardcoded syslog prefix

Now 4 options:

syslog classic syslog prefix, RFC 3164 (default)

loggly modern syslog structured prefix, RFC 5424

logplex Heroku style prefix

blank no prefix, good for JSON and CSV

Currently available through API — and through the Portal TODAY

See https://docs.fastly.com/guides/streaming-logs/changing-log-line-formats

Prefixes

Log to GCS

<134>2017-03-19T13:13:33Zcache-jfk8147foobarlogs2[453796]:{"client_ip":"207.237.138.218","req_url":"/"}

<134>12017-03-19T13:13:33Zcache-jfk8147-453796foobarlogs2-{"client_ip":"207.237.138.218","req_url":"/"}

114<134>12017-03-19T13:13:33+00:00cache-jfk8147-453796foobarlogs2-{"client_ip":"207.237.138.218","req_url":"/"}

{"client_ip":"207.237.138.218","req_url":"/"}

Prefixes

Back to JSON logging

log{"syslog60idOs66l4AbzuqvgBypS2

foobarlogs::"}{"{"client_ip":""}

client.ip{"","req_url":""}req.url

{"""}"}";

Generated VCL

log"syslog60idOs66l4AbzuqvgBypS2foobarlogs::

{"

{""client_ip":""}client.ip{"","}

{""req_url":""}req.url{"""}

"}";

Custom VCL

Big Query

How Not to Log

Only log errors

if(beresp.status==500||beresp.status==503){log{"syslog…etc…"};}

if(randombool(std.atoi(table.lookup(logging,"percentage"))),100){log{"syslog4rqBj4oy1YChTPgdapiWS4gcs-test::"}var.logstr;}

curl-XPATCH-H"Fastly-Key:<api_token>"-d"item_value=5""https://api.fastly.com/service/<service_id>/dictionary/<dict_id>/item/percentage"

Adjustable percentage

Per URL matching

if(table.lookup(panic_mode_logging,req.url)){

logsyslog4rqBj4oy1YChTPgdapiWS4gcs-test::

}

tablepanic_mode_logging{

“/my_broken_url”:“1”

}

declarelocalvar.logstrSTRING;

setvar.logstr=…;

declarelocalvar.endpointSTRING;setvar.endpoint=randomstr(1,"1234");

if(var.endpoint=="1"){log{"syslog4rqBj4oy1YChTPgdapiWS4gcs-test1::"}var.logstr;}elseif(var.endpoint=="2"){log{"syslog4rqBj4oy1YChTPgdapiWS4gcs-test2::"}var.logstr;}elseif(var.endpoint=="3"){log{"syslog4rqBj4oy1YChTPgdapiWS4gcs-test3::"}var.logstr;}else{/*only"4"left*/log{"syslog4rqBj4oy1YChTPgdapiWS4gcs-test4::"}var.logstr;}

Load spreading

Edge Applications

backend default { .host = “127.0.0.1”; .port = “8080”; .connect_timeout = 60s; .first_byte_timeout = 60s; .between_bytes_timeout = 60s; .max_connections = 800;}

acl purge { “127.0.0.1”; “localhost”;}

sub vcl_recv { set req.grace = 2m;

#Set X-Forwarded-For header for logging in nginx remove req.http.X-Forwarded-For; set req.http.X-Forwarded-For = client.ip;

Sample syntax highlighting style guide

Typography - Roboto MonoBlue - actionRed - variables Green - stringOrange - valueGrey - comment

Colors are in keynote