Service discovery and configuration management at TransferWise

SERVICE DISCOVERY AND CONFIGURATION MANAGEMENT AT

TRANSFERWISE

FIRST INTRODUCTIONS

INTROS

FAMOUS FOR

ESTONIAN FOUNDERS

EVANGELICAL CUSTOMERS

$1 BILLION IN TRANSFERS PER MONTH

SAVING OUR CUSTOMERS $50MPER MONTH

ENGINEERING CULTURE

WEAK CODE OWNERSHIP

EVERY PART OF THE CODE IS OWNED

ANY TEAM CAN CHANGE ANY PART OF THE CODE

ENGINEERING FOR CUSTOMER IMPACT

ENGINEERS GET CLOSE TO CUSTOMERS AND MAKE DECISIONS

BASED ON DATA

SOME OF OUR TEAMS SERVICES NEED TO BE HIGHLY RESPONSIVE,

SYNCHRONOUS AND SCALE REALLY WELL

OUR PLATFORM TEAM IS 8 PEOPLE STRONG

WITH A HUGE BACKLOG

HOW DID WE HANDLE IT BEFORE?

AN ENGINEER ASKS FOR A PRODUCTION SERVICE

AND WAITS…

WAITS SOME MORE…

ASKS AGAIN…

WAITS SOME MORE…

GETS A SERVICE

NEEDS TO CHANGE PRODUCTION CONFIG

WAITS…

THIS NEEDS TO STOP

TOO MUCH CHATTERTOO LONG WAITS

STRESS FOR EVERYONE

CUSTOMERS SUFFER

WE NEED:AUTOMATIC ALLOCATION

PROVISIONINGLOGGING AND MONITORING

WITHOUT ENGINEERS

SERVICE DISCOVERY CAN HELP

RESEARCH AND TEX

WHICH DISCOVERY APPROACH TO CHOOSE?

CLIENT SIDE DISCOVERY

SELF REGISTRATION

NETFLIX OSS IS OUR WAY TO GO

PROOF OF CONCEPT

MAKE OUR MONOLITH CONSUME 1 SERVICE IN PRODUCTION

UNDERSTANDING THE SCOPE

Problem: making friends between spring cloud and anything else except spring boot can be tricky. There is no good

adoption mechanism

Solution: grails and spring are close. Let’s read the source, find out what hides beyond spring boot’s netflix-specific

annotations, mimic the approach it was designed for

Lets create a shared bean in the discovery space (used by our monolith parts that consumes the config and starts

communication with Eureka)

Step 2: Let’s start a Ribbon LoadBalancer in that bean, connect it with Eureka client and let him start listening for

apps

Step 3: Most of our communication is through restTemplate. How can we make restTemplate awesome

and @LoadBalanced? Interceptors! Let’s build one

Step 4: Let’s add the interceptor to the needed RestTemplate

Step 5: What if discovery completely fails? Fallbacks!

IS THIS A PERFECT SOLUTION? OF COURSE NOT!

NEED TO MAINTAINTIED TO RESTTEMPLATES

NEEDS ENGINEERS TO THINK ABOUT IT

NOT GOOD ENOUGH

BUILD CLIENTS AND MAKE EVERY CASE WORK AS A BLACK BOX

IF YOUR SERVICE WANTS TO BE DISCOVERED - MAKE YOUR CLIENT

PROVIDE THE TOOLSET

PROBLEMS

SOME SERVICES HAVE A EUREKA CLIENT OF THEIR OWN

EUREKA IS NOT USED TO IT (CONFIGURATIONS ARE STORED IN A CONTEXT BEAN)

RESULTING IN OVERRIDES UNLESS CONTROLLED

TESTING IS PROBLEMATICAS REQUIRES A FULL COPY OF PRODUCTION

TESTING THE PIECES IS COMPLETELY ON THE ADOPTING ENGINEER

CHALLENGES

VARIATION IN HTTP COMMS AND IMPLEMENTATIONS

FEIGN AS A CLIENT BUILDERDIRECT FALLBACKS

HYSTRIX CIRCUIT BREAKINGSPLITTING BALANCING AND DISCOVERY

BETWEEN SERVICES

SOLUTION COMPATIBILITYSPRING BOOT

EUREKARIBBON

SUPPORTING TOOLSET

AROUND 75% ADOPTION PROBLEMS

3 MONTHS30 SERVICES IN PRODUCTION

20 MORE IN THE MAKING

E

GREATNOW WE HAVE LOTS OF SERVICE

CONFIG TO TRACK

LET’S MAKE THINGS AS BORING AS POSSIBLE

LET’S PUT EVERYTHING IN (PUPPET | CHEF | SALT | ANSIBLE)

JOB DONE

EXCEPT…

SECRET MANAGEMENT IS TRICKY

WE CAN MAKE IT WORK. BUT IS THERE A LESS TEDIOUS WAY?

SPRING CLOUD CONFIG SERVER

NETFLIX ARCHAIUS

HASHICORP VAULT

NICE DOCUMENTATION: HTTPS://CLOUD.SPRING.IO/SPRING-CLOUD-

CONFIG/SPRING-CLOUD-CONFIG.HTML

NAME/VALUE PAIRS

/ENCRYPT & /DECRYPT ENDPOINTS

EASY TO EMBED IN SPRING BOOT APPLICATIONS

SIMPLE REST API, SO PLAYS WELL WITH NON-SPRING TOO!

SUPPORTS TEMPLATE FILES… BUT WE’VE NOT USED THEM

SECRETS DON’T NEED TO BE STORED IN PLAIN TEXT:

/ENCRYPT/DECRYPT

VAULT BACKEND: REQUIRES A TOKEN FROM CLIENT

FAILFAST VS RETRY

GETTING IT INTO PRODUCTION:

“VOLUNTEERED” A FEW SERVICE OWNERS

EVERYTHING WORKED SMOOTHLY. VERY SUSPICIOUS.

BUGS FOUND!

GITHUB DOWN: CONFIG SERVER GOES MENTAL

CONFIG SERVER CACHES LOCALLY, BUT ALSO WILL ALWAYS CHECK FOR

NEW CONFIG IN THE REPO

VERY INFREQUENTLY… IT JUST DIES

AND WE DON’T KNOW WHY (YET)

(NEVER HAD >1 NODE DIE AT ONCE, SO NOT AWFUL IMPACT, BUT CONCERNING)

RESULT: SERVICES ARE GETTING INTO PRODUCTION FASTER

Let us change money transfer forever. Together.

https://transferwise.com/jobs/