LJC: Microservices in the real world

@chbatey

Microservices in the real world

Christopher Batey Freelancer / The Last Pickle @chbatey

@chbatey

S!*t happens when you put a network between your classes

Christopher Batey Freelancer / The Last Pickle @chbatey

@chbatey

Who am I?

• Freelance consultant: Docker, Kubernetes, Cassandra, JVM

• Work on a OS testing library for Cassandra

• Twitter: @chbatey

Heavily recruiting!!

@chbatey

Background

@chbatey

Background

@chbatey

Background

@chbatey

Background

@chbatey

Background

@chbatey

But why?????

@chbatey

Small horizontal scalable services

• Move to small services independently deployed

- Login service

- Device service

- etc

• Move to a horizontally scalable Database that can run active active in multiple data centres

@chbatey

Half way point

@chbatey

So... what do systems look like now?

@chbatey

Tech used

• All examples are on github

• Technologies used:

- Dropwizard

- Wiremock

- Hystrix

- Saboteur

@chbatey

Java microservice definition?

A microservice in Java == no dependency injection framework

@chbatey

Testing microservices

• You don’t know a service is fault tolerant if you don’t test faults

@chbatey

Isolated service tests

Prime

Movie servicePlay Movie

Test

Real HTTP/TCP

Test

@chbatey

Fault tolerance1.Don’t take forever - Timeouts2.Don’t try if you can’t succeed 3.Don’t whack a dead horse4.Turn broken stuff off

@chbatey

1 - Don’t take forever

• If at first you don’t succeed, don’t take forever to tell someone

• Timeout and fail fast

@chbatey

Which timeouts?

• Socket connection timeout

• Socket read timeout

@chbatey

Your service hung for 30 seconds :(

Customer

You :(

@chbatey

Which timeouts?

• Socket connection timeout

• Socket read timeout

• Resource acquisition

@chbatey

Your service hung for 10 minutes :(

@chbatey

Let’s think about this

@chbatey

Adding an automated test

• Vagrant - launches + provisions local VMs

• Saboteur - uses tc, iptables to simulate network issues

• Wiremock - used to mock HTTP dependencies

@chbatey

I can write an automated test for that?

Wiremock: •User Service •Device Service •Pin Service

S a b o t e u r

Vagrant + Virtual box

Test meAcceptance

prime to drop traffic

reset

@chbatey

Dependency VM

Wiremock Stubbed Cassandra

192.168.2.2

@chbatey

Wiremock

192.168.2.2

App under test

localhost

@chbatey

HITTING THE SERVICE

PRIMING

@chbatey

DEMO TIME

@chbatey

A little more detail

@chbatey

Wiremock

192.168.2.2

App under test

localhost 192.168.2.1

Delay is here

@chbatey

Wiremock

192.168.2.2

App under test

localhost 192.168.2.1

Delay is here

@chbatey

Wiremock

192.168.2.2

App under test

localhost 192.168.2.1

Delay is here

@chbatey

Wiremock

192.168.2.2

App under test

localhost 192.168.2.1

Delay is here

@chbatey

Implementing reliable timeouts

@chbatey

Implementing reliable timeouts

• Protect the container thread!

• Homemade: Worker Queue + Thread pool (executor)

@chbatey

Implementing reliable timeouts

• Protect the container thread!

• Homemade: Worker Queue + Thread pool (executor)

• Hystrix

• Spring cloud Netflix

@chbatey

A simple Spring RestController

@RestControllerpublic class Resource { private static final Logger LOGGER = LoggerFactory.getLogger(Resource.class); @Autowired private ScaryDependency scaryDependency; @RequestMapping("/scary") public String callTheScaryDependency() { LOGGER.info("Resource later: I wonder which thread I am on!"); return scaryDependency.getScaryString(); }}

@chbatey

Scary dependency

@Componentpublic class ScaryDependency { private static final Logger LOGGER = LoggerFactory.getLogger(ScaryDependency.class); public String getScaryString() { LOGGER.info("Scary Dependency: I wonder which thread I am on! Tomcats?”); if (System.currentTimeMillis() % 2 == 0) { return "Scary String"; } else { Thread.sleep(5000) return “Slow Scary String"; } }}

@chbatey

All on the tomcat thread

13:47:20.200 [http-8080-exec-1] INFO info.batey.examples.Resource - Resource later: I wonder which thread I am on!13:47:20.200 [http-8080-exec-1] INFO info.batey.examples.ScaryDependency - Scary Dependency: I wonder which thread I am on! Tomcats?

@chbatey

Scary dependency@Componentpublic class ScaryDependency { private static final Logger LOGGER = LoggerFactory.getLogger(ScaryDependency.class); @HystrixCommand() public String getScaryString() { LOGGER.info("Scary Dependency: I wonder which thread I am on! Tomcats?”); if (System.currentTimeMillis() % 2 == 0) { return "Scary String"; } else { Thread.sleep(5000) return “Slow Scary String"; } }}

@chbatey

What an annotation can do...

13:51:21.513 [http-8080-exec-1] INFO info.batey.examples.Resource - Resource later: I wonder which thread I am on!13:51:21.614 [hystrix-ScaryDependency-1] INFO info.batey.examples.ScaryDependency - Scary Dependency: I wonder which thread I am on! Tomcats? :P

@chbatey

Async libraries are your friend

• DataStax Java Driver

- Guava ListenableFuture

@chbatey

Drive this via requirements

• Reliable timeout

• Throttling

• Monitoring of failures / successes

@chbatey

Timeouts take home

• You can’t use network level timeouts for SLAs

• Test your SLAs - if someone says you can’t, hit them with a stick

• Scary things happen without network issues

@chbatey

Fault tolerance1.Don’t take forever - Timeouts2.Don’t try if you can’t succeed 3.Don’t whack a dead horse4.Turn broken stuff off

@chbatey

2 - Don’t try if you can’t succeed

@chbatey

Complexity

“When an application grows in complexity it will eventually start sending emails”

@chbatey

Complexity

“When an application grows in complexity it will eventually start using queues and thread pools”

@chbatey

Don’t try if you can’t succeed

@chbatey

Don’t try if you can’t succeed

• Executor Unbounded queues :(

- newFixedThreadPool

- newSingleThreadExecutor

- newThreadCachedThreadPool

• Bound your queues and threads

• Fail quickly when the queue / maxPoolSize is met

• Know your drivers

@chbatey

This is a functional requirement

• Set the timeout very high

• Use Wiremock to add a large delay to the requests

• Set queue size and thread pool size to 1

• Send in 2 requests to use the thread and fill the queue

• What happens on the 3rd request?

@chbatey

Fault tolerance1.Don’t take forever - Timeouts2.Don’t try if you can’t succeed 3.Don’t whack a dead horse4.Turn broken stuff off

@chbatey

5 - Don’t whack a dead horse

Movie Player

User Service

Device Service

Pin Service

@chbatey

What to do…

• Yes this will happen…

• Mandatory dependency - fail *really* fast

• Throttling

• Fallbacks

@chbatey

Circuit breaker pattern

(Picture from Akka docs)

@chbatey

Fault tolerance1.Don’t take forever - Timeouts2.Don’t try if you can’t succeed 3.Don’t whack a dead horse4.Turn broken stuff off

@chbatey

6 - Turn off broken stuff

• The kill switch

@chbatey

To recap

1.Don’t take forever - Timeouts2.Don’t try if you can’t succeed 3.Don’t whack a dead horse4.Turn broken stuff off

@chbatey

Links

• Examples:

- https://github.com/chbatey/spring-cloud-example

- https://github.com/chbatey/dropwizard-hystrix

- https://github.com/chbatey/vagrant-wiremock-saboteur

• Tech:

- https://github.com/Netflix/Hystrix

- https://www.vagrantup.com/

- http://wiremock.org/

- https://github.com/tomakehurst/saboteur

@chbatey

Questions?

Thanks for listening!Questions later? @[email protected]://christopher-batey.blogspot.co.uk/