Perforce Helix Never Dies: DevOps at Bandai Namco Studios

Perforce Helix Never Dies: DevOps at BANDAI NAMCO StudiosHayato IkedaLead ProgrammerBANDAI NAMCO Studios Inc.

2

Summary

In any case, development and operation of game, Perforce Helix is the most excellent VCS w/o any doubt.Through the introduction of the use example of a good point, I hope to help the new discovery of the audience.I came all the way from Japan.My English is not so good. Sorry.

3

Agenda

Introduction1. History of the Perforce Helix2. Perforce Helix advantages3. DevOps at BANDAI NAMCO Studios4. Best practices: Development and LiveOps5. SummaryAppendix

Introduction

5

Self-Introduction

Hayato IKEDALead ProgrammerAssistant ManagerBANDAI NAMCO Studios Inc.

Engineering: Mobile game (Server, Client)Platforms: iOS, Android, Web browser based

6

About BANDAI NAMCO Studios

See website www.bandainamcostudios.com

Trivia : ‘765’ sounds ‘NAMCO’ in Japanese.

7

A long time ago in Japan far, far away….

There was no concept of version control system...

(C)BANDAI NAMCO Entertainment Inc.

8

We came out from the cave!

It is vital for our dev team to utilize version control system.

(C)Toshiyuki Kubooka (C)BANDAI NAMCO Entertainment Inc.TEKKENTM 7 &(C)BANDAI NAMCO Entertainment Inc.

9

History of Version Control System

CVS(Concurrent Versions System)Microsoft Visual SourceSafeTeam Foundation ServerSubversionAlienbrainGitMercurial

10

Perforce Helix

‘04:NAMCO BANDAI Games America started using Perforce.‘10:NAMCO BANDAI Games started using Perforce.‘16:BANDAI NAMCO Studios own more than 800 users licenses.

11

Products using Perforce Helix

©2016 Pokémon.

©1995-2016 Nintendo/Creatures Inc./GAME FREAK inc.(C)BANDAI NAMCO Entertainment Inc.

TEKKENTM 7 &(C)BANDAI NAMCO Entertainment Inc.

12

THE IDOLM@STER

(C)BANDAI NAMCO Entertainment Inc.

1. History of thePerforce Helix

14

History of VCS at BANDAI NAMCO (1/4)

Corporate Culture : Up to each project to pick a solution

Before VCS (-’00)Keeping versions in different folder named with the date.Merged by Each persons on local PCs.Discontent : Inefficient, Not stable.

15


CVS (’01-)Revolutionary system!Discontent : Poor property list, Difficult to use for none-engineers.CVS + Alienbrain (‘03-)Alienbrain introduced as we wanted to manage a lot of art assets.Discontent : 2 VCSs for source codes and art assets.

16


Subversion+Alienbrain (‘06-)Revision numbers to manage versions.Discontent: Still 2 VCS to manage for source codes and art assets. That been said, this combination is not bad.Alienbrain(Only) (‘09-)Challenging!VC of source code requires considerable ingenuity.

17


Perforce (‘10-)We want to manage source code and art assetsin a single VCS.Compare some of the VCS.We decided to Perforce process of elimination.First adoption was the project of ‘SOULCALIBUR V’.

18

Hardships to select Perforce

Not Free!We have to convince the Product Manager.RejectionAnyone not using.(Aright!)Learning curves for non-engineersGetting them from the time of project start-up.

19

Good point : Noticed after introduction

Synchronization is fast, high-capacity assets can also be handled with ease, there is no stress.When a problem occurs, we can quickly revert to a specific revision.It has all the useful tool for programmers.The first was also operation mistake, but familiar in a few months, became proficient in non-engineers.

20

Perforce became standard

Currently, many of the projects have adopted the game engines such as Unity and Unreal Engine.Especially in the Unreal Engine, because that is a system based on the use of Perforce Helix for VC, hence many game developers are using the Helix.We have been to use the Helix also in-house engine and prototyping.

21

Why did we chose Perforce Helix?

Fast and Stable.De-facto Standard.Varieties of feature with ease of use.Reduce overall costs.

2. Perforce Helixadvantages

23

Perforce Helix advantages (1/7)

1. Helix enables you to manage all source codes and assets.

Asset(binary) and source code(text) are handled in the same VCS, they are managed in a single revisioning.When a problem occurs, we can quickly revert to a specific revision.Dev

R765Rel1

R1024Rel2

R1185

24


2. Helix provides perfect environment for iterative and incremental development.

Development and operation (DevOps) is all about iterative and incremental development.Thus, Helix is best environment to work on.

25


3. Fast!Synchronization is so fast, you never feel stress.Fast and easy to create the branch.Branching in subversion takes a long time. It’s a coffee break (can be a lunch time!).P4Merge is reasonably clever.

26


4. Helix has linked to a variety of tools. Easy integration.

Microsoft Visual StudioJenkinsUnityUnreal Engine

Etc…

27


5. P4V is versatileMost of the basic operation can be accomplished in P4V.Easy to use for non-engineers.Game Designers, Graphics Designers, Sound Creator

Graphical User Interface :-)

28


6. Fantastic customer care and support

Japanese agency Toyo is firmly supported.

29


7. Perfect match for Agile development.Scrum/DevOpsBranch/MargeWorkspace MappingStreamShelveIntegration

30

And more…

Up to 20 workspaces for Free!Workspace may look cumbersome,but it’s powerful. Depot/Workspace/LocalDivide the authority in the user groupReconcile Offline WorkGit Fusion

3. DevOps atBANDAI NAMCOStudios

32

DevOps at BANDAI NAMCO Studios

We didn’t have idea of DevOps.However, we’ve realized that how/what we are doing is something close to the so-called DevOps. Let us see one of the case.

33

Game development and LiveOps

BANDAI NAMCO Studio’s LiveOpsMainly for social mobile game development.Operation while development.Operation corresponds 24/7. It’s truly living operation!Development and operation is on the same team. (Dev and Ops are not divided)

DevOps ≒ LiveOps

34

Good points / Bad points

GoodThe same team is responsible for both development and operation. No confusion.Can design in anticipation of the operation from the start of development.Main engineers become much be involved in the product.

BadNo document! Product isn’t taken over.

35

KAIZEN everyday!

CI(Continuous Integration)Automatic(everything!)TestingCode ReviewIterationAgile/ScrumDevelopment Tools

36

The answer is not visible

We’re doing a lot of ideas and improvements, not out is still clearly the best answer.It’s difficult to summarize in game development. Because there are a number of factors.(For example, the size of the project , such as the contents of the deal game.)I can talk about cases and failures. Please listen as an example.Let's discuss together later.

4. Best practices:Developmentand LiveOps

38

Best practices: Dev and LiveOps (1/5)

1. How to make releases branch, and trunk merge.

We prefer not to branch during the initial development.Starts to deploy branching at DevOps phase followed by trunk work.Automation is highly recommended for Merge to the trunk process.If the trunk fails during scheduled build, it send you alert via e-mail.

39


2. Trick for avoiding the collisions.Text maybe able to merge, but caution with binary data.If the team sit close together in the office, raise a hand and call out. You can also use tool such as Slack.There is also to put the Exclusivecheckout of P4V.(Don’t forget to submit after the edit!)

40


3. Continuous delivery - Jenkins the way of automated build using.

41


4. The Swarm was introduced, try the code review.

42


5. Shelve and debugging technique that uses, operation check method.

5. Summary

44

Summary (again)

In any case, DevOps and LiveOps of game, Perforce Helix is the most excellent VCS w/o any doubt.Through the introduction of the use example of a good point, I hope to help the new discovery of the listener.I came all the way from Japan, please be my friend!I’m waiting for e-mail or message.

May the Perforce be with you.Hayato IkedaTwitter : ikehayaFacebook : hayato.ikeda.372LinkedIn : hayato-ikeda-690b3245

Appendix

47

It is an error to submitted just before returning home. Punishment of workspace delete! The artists don’t write a description. Even 'Get Latest Revision' by deleting the files from the

folder, and rush without revival files. Correct answer : Right Click Menu ->Get Revision ->Check ‘Force Operation’ ->Push ‘Get Revision’

Appendix: Murphy’s law

48

Japanese character code problem UTF-8 and S-JIS I want to use the command even P4V

Appendix: Wishlist

Perforce Helix Never Dies: DevOps at Bandai Namco Studios

Software