Enhance system transparency and truthfulness with request tracing

REQUEST TRACING

PROMOTING TRANSPARENCY & TRUTHFULNESS

Hello!I AM SAM KEENpolyglot software developer

You can find me at:github.com/samkeenlinkedin.com/in/samkeen@samkeenstrava.com/athletes/560497

REQUEST TRACING

What is it?

ASSOCIATING A GLOBALLY UNIQUE IDENTIFIER TO ALL REQUESTS OF A GIVEN SERVICE

COMMON OCCURRENCE IN THE WILDPOST https://graph.facebook.com/v2.2/…Response headers…x-fb-rev → 1821576

x-fb-trace-id → E1+VUthAHKh

x-fb-debug → NXN2ONVBqC8lLJVzgjknii86r5AJbmLGS5zX227q+jmLI8sm+IIuAo8jETQ3bmH82w9SmjfAQktXTXpZJ7rtuA==

GREAT, LET’S SEE THE CODE!

ACTUALLY, I’D LIKE TO APPROACH THE SUBJECT IN A DIFFERENT WAY

BEFORE, WHAT, THERE IS WHY

Development teams have a limited bandwidthBefore launching an endeavor, estimate the value it will add to the company and use that to prioritize the work, or possibly rule it out

MEASURE VALUE

How will the work help you obtain your goals

OK, GOALS, WHERE DO WE GET THOSE?

COMPLEXITY

Software is one of the most complex of human endeavors

To organize, prioritize goals, We need a plan

STRATEGIC GOALS

These are the highest level, inward facing drivers for a company

ACQUIRE ONLINE SHOE SALES MARKET LEADER

PRINCIPLES

Rules in place to help you remain aligned with Strategic Goals

SYSTEMS ARE BUILT WITH CONSISTENT, DOCUMENTED INTEGRATION POINTS

^ supports goal

PRACTICES

Often technology specific guidance to ensure Principles are being upheld

ALL SERVICES ARE DOCUMENTED WITH API BLUEPRINT

^ supports principle

GOALS <PRINCIPLES <PRACTICES

Strategic Goals

+ Acquire online shoe sales market leader

Principles

+ Systems are built with consistent and documented integration points

Practices

+ All services are documented with API Blueprint

GOALS <PRINCIPLES <PRACTICES

Strategic Goals

+ Acquire online shoe sale market leader

+ Enable rapid innovation

Principles

+ Systems are built with consistent and documented integration points

+ Ensure systems are transparent and discoverable

Practices

+ All services are documented with API Blueprint

+ All application logs leverage trace tokens

PRINCIPLE: SYSTEM TRANSPARENCYwhat do we mean by that

NOT A NEW CONCEPT

The Art of Linux ProgrammingChapter 6 : Transparency

SOFTWARE SYSTEM TRANSPARENCYGOOGLED

- A strength of OS software is that the src code is available to all so the NSA cannot hide back doors in it.

- DNS is a transparent system

A TRANSPARENT SOFTWARE SYSTEM

- has a low learning curve, low barrier to entry

- values simplicity over complexity- has high discoverability- is highly accessible

TRANSPARENCY SUPPORTING PRACTICES

- gathering performance metrics- All devs have access to all repos- Any given system should be capable of

running on a dev's machine (README)- centralized logging- Verbose modes of operation (-v). Have

verbosity off by default but don't make it inaccessible

WTF, I THOUGHT THIS WAS A TALK ON TRACING?WHEN DO WE GET TO THE TRACING?

PRACTICE TO SUPPORT TRANSPARENCY PRINCIPLE

Request Tracing

LOG ALL THE THINGS

disk is cheapcpu is cheap

logs can be valuable for more than stack traces

THE STEPS

- per request create some sort of UUID token

- override your logger to add said token to all log statements

- add said token to all responses (in case of HTTP, use header)

ALTERNATIVES

- Separate handler to its own log file- a TRACE level in the logger, just below or

above DEBUG

DEMO

CLI

aws --profile test logs filter-log-events \

--log-group-name export-demo \

--filter-pattern 559819cf1381f468e

SCALING

The are many know strategies for scaling log aggregationAlso many for log analysis

For Instance:Real-time Processing of Log Data with Subscriptions [http://goo.gl/Oo1m73]

PRINCIPLE: TRUTHFULNESS

How truthful is our documentation

How do you create / maintain truthful documentation

TRUTHFUL DOCUMENTATIONTHE REALITYStatic docs are at best: Truth’ish

As soon as they are pened, they they start a steady migration towards lies and half-truths

Entropy always wins

OFTEN THE CASEDOCS → MISINFORMATIONplanning stage, we document the system we plan to build. valuable exercise, spawns collaboration and feedbackThen we build the system. Then the system ALWAYS changes.No one goes back and updates the docs

REDUCE PURE STATIC DOCS

- Generated docs can help (javadoc, etc)- Real time metrics : better to look at what

the app is actually doing, than rely on a perf test from 3 months agoContinual Non-functional testing anyone?

- All docs should be wiki style, including diagrams!

DOCUMENTATION BEST PRACTICE

Generate as much documentation as possible from the actual operation of the System and/or the source code of the System

PRACTICE TO SUPPORT TRUTHFUL DOCUMENTATION PRINCIPLE

Request Rendering

DEMO

THANKS!Any questions?

You can find me at:@samkeen / github.com/samkeenSlides:

CREDITS

Special thanks to all the people who made and released these awesome resources for free:○ Presentation template by SlidesCarnival○ Photographs by Unsplash