IkaLog: Data Collector for Splatoon and Machine Learning

ThisisatranslatedversionofIkaLogtechpresenta6on,originallyinJapaneselanguage.

Transla6onworkiss6llinprogress!

2

What is Splatoon?

A unique third-person shooter –  Paint the battlefield with your ink,

and claim turf from the enemy team

–  Switch between an Inkling and a Squid for different approaches

–  Several gamemodes in addition to turf war, like KOTH and payload

Simple rules, but deep – Various strategies and tactics – Unique weapon classes

Shooter, Charger, Roller, Splatling, and Slosher

Strategy is the key to winning

STAGERECON

•  There’smorethanonewayaroundeachmap

•  Knowwheretheenemyiscomingfromandhaveaplan

VARIETYOFWEAPONS

•  CertainweaponsarebeCersuitedtosomemapsandmodesoverothers

You play, IkaLog collects the data

HDMICaptureDevice

IkaLog

COLLECTPLAY

IkaLog: Data Collector for Splatoon

– Play Splatoon as you would normally

– Your gameplay footage is analyzed

– The data can then be sent to log files, and/or other tools, like stat.ink

GameConsole

Logfiles

IkaLog

stat.ink,Speechapplica6on,etc.

Forwardtodesiredtools

PROCESS

Supported Integrations

VideoStreaming/Recording

AmaRecTV

OnlineDatabase

CSV/JSON,Screenshots

SNSandChat

IkaLog

stat.ink

The online database provided by @fetus_hina –  Submit your battle results

and statistics with IkaLog

–  Review your past gameplay easily using the website

Review your gameplay on stat.ink

SCOREBOARD

Reviewthescoreboard

later

Filteringallowsforcarefulanalysisofyourpast

gameplay

TIMELINE

Thegraphshowswhathappenedinthegame

Displayskills,deaths,specialweapons,rankedmode

counts/distancesandevents

GLOBALSTATS

Sta6s6csdatafrom

allstat.inkusersisavailable

Seethetrendsamongusers

Timeline Visualization

TheperiodwhenyougotsplaCedand6mespentinac6ve

Alive/Deadstatusofall8players

Team’sspecialweapons,Kills,Deaths

YourTurfScore

Timeline for Ranked Battles

StatusoftheSplatZone

SplatZonecounttheotherteam

earned

SplatZonecountyourteamearned

Thegame-changingmoment

# of IkaLog + stat.ink users

SourcehCps://stat.ink/en6re/user

Avg:200+users,4500+matchesaday

Peak:370usersadayProcessed15,000matches

Statistical data from all players

Kill/DeathHeatmap KOraWo

https://stat.ink/

.96GalDecousersdecreasedaUerthemomentofgameupdate

LongBlasterCustomisgeWngmorepopularsinceSummer

Sourcefootage MaskImage AddedImage

+==

Addi6on(OR)ofsourceimageandcorrectmaskresultsinwhiteimage

Addi6on(OR)ofsourceimageandwrongmaskresultsinnon-whiteimage

18

20

•  Therearetwofonttypesinthegame.Weshouldcoversinglenumberfontface

– Thefontisknown,soweshouldbeabletoiden6fythem.

•  DecidedtohaveMLbasedimageclassifica6on,ratherthanuseofexis6ngOCRlibrary

• 

• 

• 

• 

TesseractOCRcouldnotfindthecharachers

１）Crop the number from the footage, apply some image filters

２）Generate vertical & horizontal histogram to guess each character’s

position

３）Resize the characters to identical size, and make the image binary

４）Classify the image using KNN

•  Basciallysameideawithrecogni6onofnumbers

•  30+%ofaccuracyforsingleclassifica6on.Earnaccuracybyinves6ga6ngmanyframes

–  IkaLoginves6gatesapprox.10framesperasec

–  ThisexampleshowsIkaLoganalyzed49frames,andfound“96gal_deco”ismostlikely(18frames,36%)->correct.

votes={'supershot':6, 'carbon_deco':1, 'bucketslosher':1, 'octoshooter_replica':1,'splashshield':1, 'sshooter_collabo':5, 'hotblaster':2, 'pablo':1, 'nzap89':6,'sharp_neo':3, 'hotblaster_custom':2, '96gal_deco':18, '52gal':1, 'hokusai':1}

30

59Weapons(Splatoonhas90+weaponstoday)

• 

• 

• 

• 

Overlapsofotherequipment SimilarBG&FGcolor-1 SimilarBG&FGcolor2

Note:ThisscreenshotwastakeninquiteearlystageofIkaLogdevelopment.

Input LaplacianFilter

Grayscale Classifyusingk-NearestNeighbor

Thanks@itooon

sschooter_collabo

Featuremap

35

Pros

–  IgnoresColor

–  Ituses“theshape”formatching

Cons

– Doesn’tworkifinputhasdifferentresolu6on

– Cannotclassifysimilarshapes

39

Originalclass(longblaster)

Variantversion(longblaster_custom)

Onemorevariant!（Apr2016）

(longblaster_necro)

Conjunc6ononfeaturesbetweenthevariants.Neededtoupdate(orfind)thefeatureextrac6on

longblaster_necro

longblaster_custom

longblaster

IkaLoginternalfeaturesvaluesPloCedin2DusingPCA

Conjunc6ononfeaturesbetweenthevariants.Neededtoupdate(orfind)thefeatureextrac6on

Anothergameresultsceneswith“GearAbilityicons”Theapproachresultsinlowaccuracy…・Sameicon,differentsizeinsingleframe・Usersusesdifferentcaptureresolu6on→Morerobustimageclassifierneeded

•  Amountofthedataisalwaysjus6ceformachinelearning

•  Usethecrowdtocollectthedata50games/dayperperson

5000gamescanbepossibleby100people

–  Field-generateddatahasthenoisesandoutliers•  ModelofHDMIcaptureunits,HDMIre-transmiCers,andsoUwareconfigura6on

•  It’seasiertocollec6ngtheoutliersfromfield,thentrain,andtestclassifiers

Objectstorageonthecloud

WorkVMonIaaS

4TB+Data(collectfor12months+)

IkaLogusers stat.ink

•  AccuracyofMachinelearning/DeepLearningisaffectedbypre-processing

•  Templatematching-basedalgorithmautoma6callyfixesbroken

inputimages

Wrongaspectra6o offsetedreference

Imagetoclassify

Thanks@itoooon

Pros

–  BeCerclassifica6onaccuracy

Cons

–  Toolargedatasize（20MB->100MB~400MB）

Hardtodistributetheweights

–  BreaksWindowsversionofIkaLog

•  90%+ofusersarerunningIkaLogonWindowsPlazorm

•  MostofdeeplearningframeworkbreaksPy2EXE(PythonscripttoWindowsexecutableconverter)

– Morecompu6ngpowerneeded

–  Input： RGBorHSVcolorvalue(47*45*3=6,345units)

–  Output：possibilityofeachclass（91units,applysoUmax）

–  Connec6on：alwaysusefullyconnec6on

–  Letcomputersthefeatureautoma6cally

•  Inthisusecase,deeplearningwillfindproperweightsautoma6cally

•  Itwillignorebackgroundcolorsautoma6cally

–  TargetPerformance

•  Calc.6me:lessthan350msforeachmul6-classclassifica6on(91classes)<3secondsperaframe

•  99.99+%accuracyagainststat.inkposted(Field)data

0

1

2

3

..

..

n

0

1

2

3

…

89

90

InputLayer OutputLayerHiddenLayer

52gal

52gal_deco

96gal

96gal_deco

…

sschooter_wasabi

wakaba

51

•  TestedonAzureML,beforeimplemen6ngthecode

–  Trainedclassifiersusingsmalldataset

–  Input:Pixelvalues（w/PCA)

–  HiddenLayer:1layer、150units

–  Output:Probably91(AzureMLmanages)

•  Gotgoodresultsanddecidedtomoveforward

–  98+%accuracy(withsmalldataset)

–  Enoughrobustnessforanyimages(e.g.los-res)

54

•  UsedChainer(DeepLearningFramework)totrainthemodel

–  Supervisedtrainingusingstat.inkdata(0.9Mimages)

–  CanbenefitGPUperformance

•  TrainedmodelsareconvertedtoFloat16

– 32bitfloat->16bitfloat

– 50%smallerfilesize

CPU GPU #ofjobs

Throughput(images/s)

GPUTime/epoch(s)

RelaWveperf.

(virtual)

Corei7-4790STyp.3.2GHz

NA 1 3,928 234.5s 1X

Corei7-4790STyp.3.2GHz

GeForceGTX760

1 10,371 39.2s 5.9X

Corei7-4790STyp.3.2GHz

GeForceGTX760

4 (Cl)15,988 25.8s(103/4)

9.0XCorei7-4790STyp.3.2GHz

GeForceGTX1080

4 (Cl)30,320 6.25s(25.0s/4) 37.5X

57

CPU GPU #ofjobs

Throughput(images/s)

GPUTime/epoch(s)

RelaWveperf.

(virtual)

Corei7-4790STyp.3.2GHz

NA 1 3,928 234.5s 1X

Corei7-4790STyp.3.2GHz

GeForceGTX760

1 10,371 39.2s 5.9X

Corei7-4790STyp.3.2GHz

GeForceGTX760

4 (Cl)15,988 25.8s(103/4)

9.0XCorei7-4790STyp.3.2GHz

GeForceGTX1080

4 (Cl)30,320 6.25s(25.0s/4) 37.5X

2XE5-2630Lv3Typ.1.80GHz

TeslaP100 4 (Cl)65,811 2.5s(10.3/4)

93.8X

SpecialThanksto

TypicalLinuxBox

Pre-training(Parametertuning)

–  Testedvariouslayerconfigura6on(#ofunits),balancedmodelfilesizeanditsaccuracy

–  4concurrentjobs

–  Started03:08am〜6:28pm （15h20m）

Training

–  550epochs/24hrs IncludesdiskIO,crossvalida6ontest6me

–  Adoptedthe617thepochforthefirstNN-basedclassifier.

TookApprox.48GPUHoursusing21GBdataset

SpecialThanksto

KNN(Original) ComplexNN ThenewNN

Accuracy Low,oncertainusers 99.99+%

99.99+%

Datasize 20MB 400MB(AlexNet)100MB(GoogleNet)

14MB(Float32)7MB(Float16)

Classifica6on6me@IvyBridge2GHz

(veryfast) ~300ms ~20ms

•  Improvedaccuracyofweaponclassifica6on

•  Fasterthancomplexneuralnetworkmodels

•  Smallerdatasizemakesdistribu6oneasy

SpecialThanksto

•  IkaLogincludesownneuralnetworkimplementa6on

– DeepLearningFrameworksbreakWindowsversion

–  Re-implementedpropaga6onfunc6on

•  Tomakethecodesimple,themodeliskeptsimple

•  Compa6blewithLinearFunc6on,andReLUinChainerhCps://github.com/hasegaw/IkaLog/commit/3238b67749334a3c4254aa6f25c005f83e210895

–  [email protected]@PYNQ-Z1FPGAboard(Cortex-A9650MHz)

•  Applyneturalnetwork-basedapproachtootherclassifiers

– Gearabilityimageclassifier

–  Rainmakertouchdowndetec6on

– UseRNNfor“TideofthebaCle”evalua6onfunc6on

•  Makethedatasetsmaller

–  Thedataset(21GB)isobviouslyredundant

– UsePCA-basedanomalydetec6ontogeneratemoresmallerdata?

•  Severalmarchine-learningapproachappliedinIkaLog

–  KNearestNeighborforreal-6meclassifica6on

–  Neuralnetworkformoreaccurateclassifica6on

•  Autofeatureextrac6on,trainingfrom0.9Msamples

•  IkaLogandDeepLearning

– MostofDeepLearningframeworkbreaksPy2EXE

–  Easytore-implementofpropaga6onifthenetworkissimple

–  Canu6lizeexis6ngDNNframeworksandGPUsfortrainingprocess

LapisLazuli(2000-2014)

©07strikers