Confrontation Pipeline and SCons

Data Massage Therapy:

Pipelines, SCons and 'SOCOM: Confrontation‘

November 18th 2008

Introduction

Who Are We? Who are you?

Introduction

Who Are We?

Dean Giberson:

• Started out in 1999 as tool developer for 4 years working on several StarWars titles

• Currently Sr. Tools & Pipeline Engineer at Slant Six Games

• Specialist areas; data build management and pipelines, …

• After a short period teaching and contracting he returned to professional game development working on Sports titles for 3 years

• Credits:– NBA Live 08 (2007) Xbox 360/PS3

– NBA Live 07 (2006) Xbox 360/PS22/XBox/PC

– NBA Live 06 (2005) PS2/XBox/Gamecube/PC

– Midway Arcade Treasures 2 (2004) Gamecube

– Rogue Squadron 3: Rebel Strike (2003) Gamecube

– Rogue Squadron 2: Rogue Leader (2001) Gamecube

– Battle for Naboo (2000) N64

Paul Martin:

• Started out in the industry 1996 graphics programmer for PlayStation

• Currently a Technical Director and one of the principals of Slant Six games

• Specialist areas are in graphics rendering, special effects and data build pipelines.

• Credits:– SOCOM: Tactical Strike (2007) PSP, Slant Six Games

– Syphon Filter: Dark Mirror (2006) PSP, Sony Computer Entertainment America, Inc.

– Syphon Filter: The Omega Strain (2004) PS2, SCEA

– Sled Storm (2002) PS2, Electronic Arts, Inc.

– SSX (2000) PS2, Electronic Arts, Inc.

– Need for Speed III: Hot Pursuit (1998) PS1, Electronic Arts, Inc.

– [bunch of unreleased stuff prior to this…]

Introduction

Who Are We?

Slant Six Games – Intro:

• Slant Six Games is one of the largest video game development studios within the Vancouver game development hub. We were established in February 2005 and specialize in developing games for the PlayStation 3 and PlayStation Portable.

• We currently have 110 full time staff and occupy 20,000 square feet in our studio located at the mouth of Granville Island in Vancouver, British Columbia.

Introduction

Who Are We?

Slant Six Games – Our Games:

• First project was the creation of the graphics rendering engine for Sony Computer Entertainment America’s (SCEA) Syphon Filter: Dark Mirror (developed by Sony Bend, Oregon).

• Inaugural title, SOCOM: Tactical Strike, published by SCEA, in October 2007. Was subsequently named the Best PSP Multiplayer Experience at E3 2007 and in February 2008 it was awarded the Elan Award for the Best Handheld Game of the Year. To date it has sold over 400,000 copies worldwide. Also in February 2008 we were awarded the prestigious Elan Award for Best New Video Game Company.

• Just completed development of our largest and most significant project to date, SOCOM: Confrontation, for the PS3, in late summer of 2008. SOCOM: Confrontation has been in production for just under two years and was one of the more highly anticipated titles of this fall.

• Currently working on• Shipping SOCOM: Confrontation in other worldwide regions

• Unannounced PSP title for SCEA which is scheduled for completion in June of 2009

• Expansion packs and other new downloadable content for SOCOM: Confrontation on PS3

• Another unannounced new concept game for PS3

Introduction

Who Are You?

• Programmers/Technical Artists/Bunch of other people,

hopefully interested in how to build an efficient data build

pipeline for current/next generation game content

• In the wrong room?

Data Build System Requirements

• Fast!

• Correct

• Extensible

• Debuggable

• Asset management integration

Our Data Builder System

Features:

• Dependency Tracking

• Shared Build Cache

• Tight integration with Perforce

• Partial Builds (build only: Textures, Collision, World Descriptors, etc)

Built on top of Scons…

SCons

What is it?

• From SCons.org:SCons is an Open Source software construction tool—that is, a next-generation build tool. Think of SCons as an improved, cross-platform substitute for the classic Make utility with integrated functionality similar to autoconf/automake and compiler caches such as ccache. In short, SCons is an easier, more reliable and faster way to build software.

Where did it come from?

• Also from SCons.org:SCons began life as the ScCons build tool design which won the Software Carpentry SC Build competition in August 2000. That design was in turn based on the Cons software construction utility. This project has been renamed SCons to reflect that it is no longer directly connected with Software Carpentry (well, that, and to make it slightly easier to type...).

SCons

Why are we using it?

…back to Data Build System Requirements

• Fast!

• Correct

• Extensible

• Debuggable

• Asset management integration

SCons

Pros:

• Robust

• Reliable

• Rich Scripting Interface (Python!)

• Shared Data Caching

• Multi-processor support

SCons

Cons:

• Phase 1 dependency & job building can be

slow

• Core built on non-trivial Python code

• Can make non-standard extensions to the

system tricky to implement (e.g. dependency

insertion, dependency pruning)

SConstructWhat is it?

• Equivalent of a Makefile

• The input file that SCons uses to control the build

• A Python script. With a one main difference• Definition order of SCons builder functions, does not

dictate execution order

• Execution order of non-builder functions is as expected

• Two-step processing• Regular Python (non-SCons Builder) functions first

• Then SCons builders invoked as deemed necessary by the dependency engine

SConstruct

Some usage examples:

• Texture processing

• SSG (wad/pak/zip/rx2/etc) processing

• Tool Target Scanner (examples)

Pretty It Up – Use a GUI!Nature abhors a vacuum

Pretty It Up – Use a GUI!Artists and Designers abhor a command-line interface

Pretty It Up – Use a GUI!Undesirable Desirable

Pretty It Up – Use a GUI!

Not all roses!

• Staged building (the graph problem)

• Maya

• Large trees have large dependency

graphs

Not all roses! Staged building

• You can’t know what Maya files to build

until you have scanned your level files.

• You can’t know what textures to process

until you have scanned your Maya files.

• You don’t know what assets to place into

you ZIP until you have everything else

first.

Solution. Staged building

• We manually scanned the level files using

Python XML

• Texture build can still be a problem

• ZIPs get rebuilt if there is any change

• We have alpha level code for staged

building for automatic LOD generation, but

it was not exercised enough before we

changed LOD methods

Not all roses! Maya

• We use MBs as our source file format for

3D art.

• You can only have one instance of Maya

running (and keep it reliable).

• Maya can take 30 seconds to launch.

• Bottleneck!

Solution. Maya

• Going to use gzip MAs, and write a text

scanner for texture links, or write a simple

MB scanner for textures

• SN-DBS MayaBatch

• Stripping Maya’s default launch scripts can

get ~3 second launches

• Still bottleneck!

Not all roses! Big tree

• Building the dependency tree for the whole

game could take 1.5 minutes for all assets

• MD5 sum of many files can be slow

• Artists that only want to change one

texture (a 2 second problem) should not

have to wait on this

• But it works great for build servers

• Balance?

Solution. Big tree

• Internally trim the tree for some assets based on type

• ResourceBuilder tells SCons the sub portion of the tree the artist is interested in and we only build that graph

• New versions of SCons have the `--interactive’ flags to scan the tree once and let you run multiple commands on it

• Balance

Not all roses! Other platforms

• Make \ XNA \ Ant \ Cmake \ YABS

• All have similar problems

• SCons’ MD5 checking gives best results

so far

• …and Python script, not <gripe> XML

</gripe>

The rebuild story

SOCOM: Confrontation

SOCOM: Confrontation

Some Stats:

• Source data total size: 200 Gb

• Target (PS3) data total size: 3 Gb

• World models for 1 world:49

• Unique prop models for 1 world: 167

• Prop instances for 1 world: 1901

• Unique textures for 1 world: 1209

• Lightmaps for 1 world: 1581 (x2 – day+night)

What Next?

• Debugger

• Staged building

• Link to SN-DBS

• Code?

References

• SCons: http://www.scons.org/

• Slant Six Games: http://slantsixgames.com

• SOCOM: Confrontation: http://www.us.playstation.com/PS3/Games/SOCOM_U_S_Navy_SEALs_Confrontation

Talk to Us!

Paul Martin: [email protected]

Dean Giberson: [email protected]

Slant Six Games

3rd Floor,

1523 West 3rd Avenue,

Vancouver, BC V6J 1J8

http://www.slantsixgames.com

Bye For Now!