a million bots can't be wrong

a million bots can't be wrong @remeniuk, Viaden Media #ScalaSBP, 18-05-2012

In Viaden our aim is to make the best poker ever

we know that performance tests should be

the first-class citizens

and kill 2 birds with one stone, using bots for testing

 #1 we can emulate 50k players using just one medium EC2 instance #2 bots are interactive, so client teams can use them in development, and QA for testing

everyone knows Akka, huh?

why Scala and Akka is a perfect choice for making bots?

actors are !(interactive)straightforward remotingsimple scalability/clustering~30 minutes to make a DSL and CLI with Scala and SBT

..., but, I have to warn you ...

4 dead simple tips for keeping your sanity

when you do asynch with Akka 2.0

tip #1: live fast, die young

typical approach in Akka 1.x

lobby

desk

botlogin lin

k

tourney

Akka 1.x: actors have a long lifecycle

lobby

desk

bot

botplay game

link

linkun

link

tourney

lobby

desk

tourney

bot

bot

botplay tournament linkun

link

Akka 1.x: actors have a long lifecycle

in Akka 2.0 the world has changed

actors

paths

props

newsupervision

now, you're forced to do "the right thing" (c)

lobby

tournament desk

desk

IdleBot

DeskBot

DeskBot

all actors are supervised

lobby

desk login

easy come

lobby

desk

IdleBotplay game

easy go (when supervisor to be changed)

lobby

desk

IdleBotDeskBot

dies

borns

class Lobby extends Actor {

case Login(username, password) => context.actorOf(Props(new IdlePokerBot(...)))

case JoinAnyDesk(props) => findDesk ! JoinDesk(props) } class Desk extends Actor {

case JoinDesk(botProps)=> context.actorOf(Props(new DeskPokerBot(botProps)))

} class IdlePokerBot(props: BotProps) extends Actor {

case PlayNow => context.parent ! JoinAnyDesk(props); context.stop(self)

}

Props Pattern - "the soul" of an actor

IdleBot

DeskBot

BotProps

BotProps

Props remains alive between actor "reincarnations"

case class BotProperties(id: Long, login: String, script: Seq[BotEvent], aggressiveness: Int, sessionId: Protocol.SessionId) class IdlePokerBot(val botProperties: BotProperties) extends Bot[Poker] class DeskPokerBot(val botProperties: BotProperties) extends Bot[Poker]

tip #2: think beyond

when you know, who's supervising, life's simple

akka://gpdev/user/lobby/player1234 akka://gpdev/user/lobby/desk1/player1234 akka://gpdev/user/lobby/tournament1/desk1/player1234

ActorRegistry, actor UUID

but what should I do, now, when I don't know, where to look for my bot?

were removed from Akka

Bad news

you can make your own registry (using Extensions, backed with a distributed data structure)...

or, use the full power of location transparency

lobby

desk

IdleBot

DeskBot

ProjectionManager

Projectionvar location:

ActorPath

/lobby/desk123/player123/projection/player123

class Projection(var container: ActorPath) extends Actor { def receive = { case newContainer: ActorPath => container = newContainer case msg => context.actorFor(container.child(self.path.name)) ! msg } } class ProjectionManager extends Actor { def receive = { case Add(actor) => context.actorOf(Props(new

Projection(actor.path.parent)), actor.path.name) case Forward(msg, path) => context.actorFor(path) ! msg } } projectionManager ! Add(actorRef)projectionManager ! Forward("ping", "actor1")

class Projection(var container: ActorPath) extends Actor { def receive = { case newContainer: ActorPath => container = newContainer case msg => context.actorFor(container.child(self.path.name)) ! msg } } class ProjectionManager extends Actor { def receive = { case Add(actor) => context.actorOf(Props(new

Projection(actor.path.parent)), actor.path.name) case Forward(msg, path) => context.actorFor(path) ! msg } } projectionManager ! Add(actorRef)system.actorFor(projectionManager.path.child("actor" + i)) ! "ping"

class ProjectionManager extends Actor { def receive = { case Add(actor) => context.actorOf(Props(new

Projection(actor.path.parent)), actor.path.name) case Forward(msg, path) => context.actorSelection("../*/" + path) ! msg } } val projectionManager = system.actorOf(Props[ProjectionManagerRoutee]

.withRouter(RoundRobinRouter(resizer = Some(DefaultResizer(lowerBound = 10, upperBound = 20)))), "projection")

projectionManager ! Add(actorRef)projectionManager ! Forward("ping", "actor1")

case class CustomRouter(n: Int, routerDispatcher: String = DefaultDispatcherId, supervisorStrategy: SupervisorStrategy = defaultStrategy) extends RouterConfig { def createRoute(props: Props, provider: RouteeProvider) = { provider.registerRoutees((1 to n).map(i => provider.context.actorOf(Props[ProjectionManager], i.toString))) def destination(sender: ActorRef, path: String) = List(Destination(sender, provider.routees(abs(path.hashCode) % n))) { case m@(sender, Add(actor)) ⇒ destination(sender, actor.path.name) case m@(sender, Forward(_, name)) ⇒ destination(sender, name) } } }

tip #3: don't do anything stupid

you've tried all the options, system load is fine, only 1/10 of the heap is used, but you still can start not more than 1k bots!?

ulimit -n <proper value>

your actor is lacking of throughput?       wait before adding poolsshare responsibility!one fine-grained actor is enough in 99% of the cases

100-300 threads are serving 300 bots!?

spawn futures, backed with standalone [bounded] pools, for blocking operations

class ThirdPartyWrapper extends Actor { case F(x) => sender ! thirdPartyService.f(x) // call to a function that takes a lot of time to // complete } class ThirdPartyWrapper extends Actor { case F(x) => val _sender = sender Future(thirdPartyService.f(x)).map(_sender ! _) // ugly, but safe, and perfectly right}

use separate dispatchers

lobby

desk

DeskBot

ProjectionManager

Projection 

projection-manager-dispatcherBalancingDispatcher

projection-dispatcherDispatcher

lobby-dispatcherPinnedDispatcher

container-dispatcherDispatcher

desk-bot-dispatcherDispatcher

GOTCHA: Akka successfully bootstraps, even if your dispatcher is not configured, or the config is wrong Always check the logs to make sure that dispatchers are used!

[WARN][gpdev-akka.actor.default-dispatcher-1] [Dispatchers] Dispatcher [bot-system.container-dispatcher] not configured, using default-dispatcher[WARN][gpdev-bot-system.container-dispatcher-1] [PinnedDispatcherConfigurator] PinnedDispatcher [bot-system.lobby-dispatcher] not configured to use ThreadPoolExecutor, falling back to default config. [DEBUG][gpdev-akka.actor.default-dispatcher-24] [akka://gpdev/user/poker/lobby] logged in[DEBUG][gpdev-akka.actor.default-dispatcher-14] [akka://gpdev/user/poker/projeciton/$g/player20013] starting projection...

tip #4: analyze that

how to measure? Metrics - pushes various collected metrics to GraphiteCarbon and Graphite - gather metrics, and expose them via web interface 

object BotMetrics { val loggedInCount = new Counter(Metrics.newCounter(classOf[Lobby[_]], "logged-in-count")) GraphiteReporter.enable(1, TimeUnit.MINUTES, "localhost", 2003) } class Lobby extends Actor { case Login(username, pass) => BotMetrics.loggedInCount += 1 }

1. add logged-in user counter 2. update it3. enable reporting to Graphite4. build a graph in Grtaphite

1.

2.

3.

4.

what to measure? - mailbox size1

- throughput- time, before the message is processed (both in actor and future)2

- time to process a message- count of threads- actor pool size- heap size

1 requires implementation of a custom mailbox that can expose mailbox size2 every message should be stuffed with a timestamp

how to tune dispatchers? VisualVM - thread timeline shows, if thread polls behind dispatchers are used effectively 

don't neglect old good logging  [ERROR][05/06/2012 12:55:43.826] [gpdev-bot-system.desk-bot-dispatcher-7] [akka://gpdev/user/poker/lobby/tournament5382577/desk109129/player20121] unprocessed game event: GameEvent(CHAT,None,None)

thanks for listening

viaden.com/careers/vacancies.htmlwe're hiring!