Do's and don'ts when deploying akka in production

Deploying Akka in Production

Do’s and Don’ts when

Mike Nash

Introduction

We’ve deployed dozens of Akka-based services and apps to production in the past 5 years

We’ve scaled up to tens of millions of events per hour, day after day

Our team has learned where the potholes are

Architecting for Performance

Going reactive before the first line of code is written

Code Quality and ArchitectureQuality = lower cost of future changes

Correct architecture = longevity, e.g the system lives on even if parts are replaced in flight

Many small pieces usually work better than a few big ones

Find out about Ports and Adapters/Onion Architecture: Keep domain and infrastructure distinct

Delivery Guarantees

Understand what guarantees Akka provides - and what it doesn’t

The best design is often the one that doesn’t need many guarantees

Most “guarantees” are imaginary in any case

Single Points of ContentionDatabase?

Blocking OperationsLook for blocking operations

ImmutabilityEncapsulate state in Actors (or Agents)

Messages must be immutable

Keep it Simple

Just because you can doesn’t mean you should

Simple, independent pieces are best

Keep your code out of other people’s databases

Distinguish between Commands and Events

Think about back-pressureThink about back pressure: Look for any situations where mailboxes might be allowed to grow without limits

Slow producer is not a big problem

Slow consumer can be a big problem

Akka streams

Think about dispatchers and routers: The defaults are not always what you want.

SerializationThink about for remote actors

Default requires some class path magic

Choices include text, json, protobuf, many others, all with tradeoffs

Benchmark your serializer/deserializer!

Actor? Future? Agent?Actors encapsulate State or Behaviour, they have a mailbox and a name

Futures contain and retrieve the result of a concurrent operation

Agents encapsulate a value (at a single location) - immediate read, async writes

TestingCommon performance blockers under heavy load

Testing

Beyond the Obvious

Unit and Functional are the Foundation

Akka TestKit: Test actors as if they were synchronous

“Negative” tests are essential

JVM Tuning

MemoryYou need more than you think you need

Know your Garbage CollectorsMust run under load to see characteristics

Defaults are probably not what you want

Takes more memory to free up more memory…

-verbosegc is your friend

Production DeploymentsTips for a rock-solid deployment pipeline

Deployment Pipeline

Not just about getting to production

How will you deploy new versions without downtime?

How will you revert if you have to?

It’s 2 am - Do you know where your dependencies are?

Deployment PackagingHow will you get your app to production? Onejar?

In our experience, external containers are not helpful

How will you tell if its the right version and revision?

Rolling Restarts?

Dependencies

Your application or service sits atop a tall stack of dependencies

Is the stack in the same order on production?

Is it the same stack?

Configuration

Akka’s configuration works great - don’t fix it!

Discovery beats hard-wiring

Procrastinate

MonitoringBeyond Testing

MonitMore than monitoring

Statsd, GraphiteCustom timing and counts

KamonSpecifically for Reactive Applications

What happens when….?

Boundary Conditions

LoggingRelated to monitoring, but very different Surprisingly expensive - consider async, macros (like ScalaLogging) to omit when level disabled Line numbers are *very* expensive, check your formats Log rolling strategy is critical Consider Graylog or similar to view logs across multiple nodes, filter, route and alert on log contents Consider ssh server or remote message to change log level on the fly for a running system Logback’s Mapped Diagnostic Context can be handy

Security

Beyond the usual security concerns any application has

How will you prevent “fake” messages from being handled?

There is such a thing as too much of a good thing

Summary