#devshark welcome to #devshark. #devshark HELLO! I’M Ville Rauma Fingersoft Product Owner email ville.rauma@fingersoft.net Web .
Post on 28-Dec-2015
215 Views
Preview:
Transcript
#devshark
#devshark
welcome to #devshark
#devshark
HELLO!I’M Ville Rauma
FingersoftProduct Owner
emailville.rauma@fingersoft.net
Webhttp://fingersoft.net
#devshark
Introduction
• Today’s talk• Fingersoft• +50 million customers; The Fingersoft AdSystem Azure
Migration Project• What is AdSystem?• Implementation• Problems & Solutions
#devshark
Fingersoft
• Game development / publishing• Based in Kempele, near Oulu Finland• 14 Employees. Currently looking for C++ / Objective-C
programmers• 2 in-house games: Hill Climb Racing & I Hate Fish (soft
launch early December 2014. Check it out!)• 6 partner games published: Benji Bananas, Benji
Bananas Adventures, Fail Hard, Pick a Pet and Javelin Masters 2• 2013, net revenue 15.5meur
#devshark
What is the Fingersoft AdSystem?
• Serve settings to games
• Serve ad network priorities
• Collect real time statistics• Game startups, device
types, locales, platforms etc.
• Adjust settings/priorities in real time
#devshark
Hill Climb Racing
• Each game startup does 1-3 requests to our AdSystem
#devshark
AdSystem History
• Version 1• PHP web server originally in a home computer and dsl
• Very high latency to long distance locations• Moved to Amazon micro instance, then moved to m1.medium instance
while traffic increased.• No data collection or any dynamic functionality.
• Version 2• Clustered Node.js and Nginx based REST API with MySql
database to hold some statistics and settings• Could server from 6k to 8k requests/s at best.• Still the same latency issues due to single location• Could only scale vertically
#devshark
Finding Solutions
• Selection process• Researched competing
Cloud services such as Amazon and Azure
• Talked to contacts with knowledge in the area
• Introduction to Azure via Bizspark membership
• Proof of concept & Decision
AdSystemV3
#devshark
Implementation in AzureGame Clients
Traffic Manager
Cloud Service Eu West
Load Balancer
AdServer EuWest 1
AdServer EuWest 2
...
Cloud Service Eu North
Load Balancer
AdServer EuNorth 1
AdServer EuNorth 2
...
Selects cloud service based on latency
Redis Cache
Statistics Server
Admin interface
Table Storage
Azure Services
Scales Horizontally
Local cache <-> redis cache Write periodically to DBLinux VMs with Node.JS
Linux VMs with Node.JSAdmin GUI is node express app
Blob Storage
Promotion Image data
Statistics, settings etc.Blob location per Cloud service loc.
Serve images from closest blob
#devshark
Results
• Low latency to most locations in the world due to ability to setup servers in any Azure server location• High availability via Availability sets / load balancing• Horizontal/Vertical scaling capability
• Cuts costs since we need only keep up the type and amount of VM instances that we need at any given time. Can serve all clients.
• Ease of deployment• Microsoft provides npm libraries to control azure services. This
means that we write deployment scripts in the same language(js) that we use to script server logic. Basically we have 1 click solution to deploy VMs and software into Azure cloud.
#devshark
A Few Problems / Solutions
Problem1. Windows server based Node.js was
first attempted due to easy deployment scenarios. It was not feasible because of high cpu usage per served client in comparison to Linux.
2. Initial implementation used direct azure table storage reads/writes for each server VM. This didn’t work due to high number of failed operations which then required complicated reattempt and backoff strategies. These also increased memory and processor usage.
3. How to realistically test massive connection amounts to reflect production servers?
Solution1. Moved to use Linux based VMs
instead. This allowed us to easily customize linux configuration to be able to handle massive concurrent connection counts.
2. Data is now first passed to Redis cache. Then a separate Statistics server is used to read cache periodically and write data to DB. This removed all data issues from AdServers serving the clients. All AdServer resources now go towards serving clients.
3. We used Visual Studio 2013 Ultimate edition which can do cloud based load tests via visualstudio.com to simulate a real world scenario. Highly useful!
#devshark
Contact us!
• If you’re looking for a job: jobs@fingersoft.net• If you’re looking to get your mobile game published:
publishing@fingersoft.net
#devshark
#devshark
PEW PEW!(oh and thank you)
top related