#devshark welcome to #devshark. #devshark HELLO! I’M Ville Rauma Fingersoft Product Owner email [email protected] Web .

#devshark

#devshark

welcome to #devshark

#devshark

HELLO!I’M Ville Rauma

FingersoftProduct Owner

[email protected]

Webhttp://fingersoft.net

#devshark

Introduction

• Today’s talk• Fingersoft• +50 million customers; The Fingersoft AdSystem Azure

Migration Project• What is AdSystem?• Implementation• Problems & Solutions

#devshark

Fingersoft

• Game development / publishing• Based in Kempele, near Oulu Finland• 14 Employees. Currently looking for C++ / Objective-C

programmers• 2 in-house games: Hill Climb Racing & I Hate Fish (soft

launch early December 2014. Check it out!)• 6 partner games published: Benji Bananas, Benji

Bananas Adventures, Fail Hard, Pick a Pet and Javelin Masters 2• 2013, net revenue 15.5meur

#devshark

What is the Fingersoft AdSystem?

• Serve settings to games

• Serve ad network priorities

• Collect real time statistics• Game startups, device

types, locales, platforms etc.

• Adjust settings/priorities in real time

#devshark

Hill Climb Racing

• Each game startup does 1-3 requests to our AdSystem

#devshark

AdSystem History

• Version 1• PHP web server originally in a home computer and dsl

• Very high latency to long distance locations• Moved to Amazon micro instance, then moved to m1.medium instance

while traffic increased.• No data collection or any dynamic functionality.

• Version 2• Clustered Node.js and Nginx based REST API with MySql

database to hold some statistics and settings• Could server from 6k to 8k requests/s at best.• Still the same latency issues due to single location• Could only scale vertically

#devshark

Finding Solutions

• Selection process• Researched competing

Cloud services such as Amazon and Azure

• Talked to contacts with knowledge in the area

• Introduction to Azure via Bizspark membership

• Proof of concept & Decision

AdSystemV3

#devshark

Implementation in AzureGame Clients

Traffic Manager

Cloud Service Eu West

Load Balancer

AdServer EuWest 1

AdServer EuWest 2

...

Cloud Service Eu North

Load Balancer

AdServer EuNorth 1

AdServer EuNorth 2

...

Selects cloud service based on latency

Redis Cache

Statistics Server

Admin interface

Table Storage

Azure Services

Scales Horizontally

Local cache <-> redis cache Write periodically to DBLinux VMs with Node.JS

Linux VMs with Node.JSAdmin GUI is node express app

Blob Storage

Promotion Image data

Statistics, settings etc.Blob location per Cloud service loc.

Serve images from closest blob

#devshark

Results

• Low latency to most locations in the world due to ability to setup servers in any Azure server location• High availability via Availability sets / load balancing• Horizontal/Vertical scaling capability

• Cuts costs since we need only keep up the type and amount of VM instances that we need at any given time. Can serve all clients.

• Ease of deployment• Microsoft provides npm libraries to control azure services. This

means that we write deployment scripts in the same language(js) that we use to script server logic. Basically we have 1 click solution to deploy VMs and software into Azure cloud.

#devshark

A Few Problems / Solutions

Problem1. Windows server based Node.js was

first attempted due to easy deployment scenarios. It was not feasible because of high cpu usage per served client in comparison to Linux.

2. Initial implementation used direct azure table storage reads/writes for each server VM. This didn’t work due to high number of failed operations which then required complicated reattempt and backoff strategies. These also increased memory and processor usage.

3. How to realistically test massive connection amounts to reflect production servers?

Solution1. Moved to use Linux based VMs

instead. This allowed us to easily customize linux configuration to be able to handle massive concurrent connection counts.

2. Data is now first passed to Redis cache. Then a separate Statistics server is used to read cache periodically and write data to DB. This removed all data issues from AdServers serving the clients. All AdServer resources now go towards serving clients.

3. We used Visual Studio 2013 Ultimate edition which can do cloud based load tests via visualstudio.com to simulate a real world scenario. Highly useful!

#devshark

Contact us!

• If you’re looking for a job: [email protected]• If you’re looking to get your mobile game published:

[email protected]

#devshark

#devshark

PEW PEW!(oh and thank you)