Mathias Westerdahl - System Architect, Tools The Asset pipeline for Just Cause 2: Lessons Learned
Mathias Westerdahl - System Architect, ToolsThe Asset pipeline for Just Cause 2: Lessons Learned
2
The Asset pipeline for Just Cause 2: Lessons Learned
Overview• Background
• Just Cause 1
• Just Cause 2
• Current pipeline
• Build system and compilers
• Central code
• Take aways
• Q & A
3
The Asset pipeline for Just Cause 2: Lessons Learned
What is Just Cause?• Large open world
• Sand box game play
• Dynamic game play
• No Levels, continuous streaming
• Multiplatform
• Very flexible editing and testing of content
• Needs a fast and flexible pipeline
4
The Asset pipeline for Just Cause 2: Lessons Learned
JustEdit• Our main editor is called JustEdit
• Edits the terrain, missions, locations, roads, etc.
• Each mission/location contains entities
• An entity contains game objects or other entities
• This highly flexible system led to very many assets to export (~1500 locations and ~4000 entities)
• Runs the game
• No turnaround times for testing
5
The Asset pipeline for Just Cause 2: Lessons Learned
JustEditJustEdit MayaMaya
PerforcePerforce Damage Editor
Damage Editor
TerrainLocationsMissionsParticles
ModelsPhysicsAnimationsSkeletons
Physics
Workflow Overview for Just Cause 1
GameGame
6
The Asset pipeline for Just Cause 2: Lessons Learned
JustEditJustEdit MayaMaya
TerrainLocationsMissionsParticles
ModelsPhysicsAnimationsSkeletons
Physics
Workflow Overview for Just Cause 1
Many manual stepsGame ready data in version control
Game ready data in version control
Editing of game ready data!!!
PerforcePerforce Damage Editor
Damage Editor
GameGame
7
The Asset pipeline for Just Cause 2: Lessons Learned
Lessons learned from Just Cause 1• Remove as many manual steps as possible
• Put export steps in separate tools (i.e. compilers)
• Decrease code complexity
• Increase robustness
• Easier to read and debug
• Introduce intermediate data formats
• Tools code broke easily
8
The Asset pipeline for Just Cause 2: Lessons Learned
Actions Taken after Just Cause 1• We decided on creating our own content build system
• Compilers in C++, derived from previous export steps
• Took existing code and put into use directly
• The Project Compiler was the heart of the system
• Triggered all other compilers
• Dependency checking was implemented separately by each compiler (time stamps)
• Users can right click on any file and compile that dependency tree
• The editor and other tools use this functionality
9
The Asset pipeline for Just Cause 2: Lessons Learned
Actions Taken after Just Cause 1• Auto Builders were setup
• Code
• All compilers, exporters, plug-ins, editor
• Content
• Full nightly builds for all platforms
• Builds ISO’s
• Cruise Control + NAnt was used to create the auto builders
• CCTray is a system tray icon tool
• Alerts users when a build fails
10
The Asset pipeline for Just Cause 2: Lessons Learned
Just Cause 2 – New Tools• A lot of new tools were ordered by the content creators
• Detailed design docs were written
• Each tool was assigned a “customer”
• The programmers and the customer kept close communication
• The customer was responsible to test the tool and approve it
• Tools as plug-ins to JustEdit
• WTL was used for GUI in editors/plug-ins
11
The Asset pipeline for Just Cause 2: Lessons Learned
JustEditJustEdit MayaMaya
PerforcePerforce
CompilersCompilers
TerrainLocationsMissionsParticles
ModelsPhysicsSkeletons
CompileGameGame
Workflow Overview for Just Cause 2
MotionBuilderMotionBuilder
AnimationsCutscenesConstraints
Plug-insPlug-ins
VehiclesCharactersCutscenes
Auto BuilderAuto Builder
Sync
12
The Asset pipeline for Just Cause 2: Lessons Learned
Lessons Learned from Just Cause 2 – New Tools• 10 new tools at start of project!!
• Code not up in the 90%-100% complete where it should be
• Flaws in workflows
• Not all tools were used in the extent that was intended
• Content that was easy to edit was not very suitable in game with respect to performance and memory
• Design docs were accepted “as-is”
• They were essentially a wish list from content creators
13
The Asset pipeline for Just Cause 2: Lessons Learned
Lessons Learned from Just Cause 2 – New Tools• Customers
• They were not part of the dev-team, communication suffered
• Often planned full time on other tasks
• Unclear responsibilities
• Plug-ins
• Unstable C++ interfaces
• Should have gone with network communications
• Too complex
• GUI: WTL and C++
• WTL is too bare bone and C++ slows down iterations
14
The Asset pipeline for Just Cause 2: Lessons Learned
Lessons Learned from Just Cause 2 – Compilers• Compilers in C++
• Using C++ for scripting isn’t always very convenient
• Long build times, making it awkward to tweak
• Some compilers included all game code
• Very platform specific
• Win32 heavy with DirectX amongst other things
• Couldn’t run on auto builders with no desktop
15
The Asset pipeline for Just Cause 2: Lessons Learned
Lessons Learned from Just Cause 2 – Compilers• Programmers implemented things slightly differently
• Causing anomalies in overall behavior
• Breaking dependency checking
• Hard coded parameters in the compilers
• File formats weren’t stable enough
• If something went wrong, compiler probably crashed
• Or worse, the broken data made it into the game
16
The Asset pipeline for Just Cause 2: Lessons Learned
• Reluctance to work with compilers
• The pipeline was considered somewhat magic
• Lack of documentation
• The overall flow wasn’t easily overviewed
• Debugging broken data often done in game code
• Programmers already had all code & data set up
• This sometimes led to runtime fixes being done rather than fixing the bugs in the compilers
Lessons Learned from Just Cause 2 – Compilers
17
The Asset pipeline for Just Cause 2: Lessons Learned
• No central code
• Not using central code meant problems when integrating
• Fixes went missing, you had to find them again
• Built compilers were version controlled in Perforce
• Not automatically rebuilt and used when needed
• You could still use old buggy code
Lessons Learned from Just Cause 2 – Compilers
18
The Asset pipeline for Just Cause 2: Lessons Learned
• Data formats
• A lot of different file types (~30)
• Many of the formats were Xml based
• Mapping “property name” to “value”
• Formats were more optimal for the editor than for the game
• Read/write errors weren’t detected
• No version numbers
• No proper error messages could be logged
• Lot of things to improve!
Lessons Learned from Just Cause 2 – Compilers
19
The Asset pipeline for Just Cause 2: Lessons Learned
• Content build system
• A bit naïve solution
• Slow dependency checking
• Incomplete dependency tree
• Time stamps are not good enough
• Due to bugs & anomalies in the behavior, users didn’t trust it to 100%
• “It went wrong! Did I do it in the wrong order?”
• “I’d better resort to my old ways without asking anyone…”
• No trust meant full clean and full rebuild
Lessons Learned from Just Cause 2 – Compilers
20
The Asset pipeline for Just Cause 2: Lessons Learned
Statistics from Just Cause 2• 3 platforms: Win32, PS3, Xenon
• ~110000 source assets in version control
• ~80000 of those files pass through our compiler pipeline
• Source data amount to ~13GB
• Compiled data amount to
• ~7GB uncompressed
• ~4GB compressed
• A full rebuild took about 4h45m for all 3 platforms
• Building all tools code took ~1h
21
The Asset pipeline for Just Cause 2: Lessons Learned
Current pipeline• What was done to address the retrospective from JC2?
• Fresh goals
• Rebuild the trust in the pipeline
• Incremental builds, no magic workflows, easy to configure
• 100% accuracy using the dependency checking
• Simplify the development
• Move common code to a central repository
• Decrease turn around times when creating tools
22
The Asset pipeline for Just Cause 2: Lessons Learned
Current pipeline• Big decisions
• We needed a build/deployment system for central code
• The content build system also needed a major push
• Needed a simple to use and yet powerful generalized data format
• Development had to be done alongside JC2
• Needed to commit to the task to actually get anywhere soon
23
The Asset pipeline for Just Cause 2: Lessons Learned
Current pipeline - Needy• Dependency resolver and deployment system
• Written in house using python, very light weight
• Handles “packages”
• Built libraries (.h + .lib/.so)
• Built executables
• Bundles of python scripts (or whatever you want)
• 3rd party libraries / executables
• Deploys packages to a central repository
• Fetches packages to local system
Dependencies.txt
Python-2.5.150.1013.1;buildtool
Needy-4.4.0.204986;buildtool
Buildsystem-6.1.*;buildtool
Platform-7.*;buildtool
Debug-8.*;buildtool
Adf-8.*
havok-55x.277059
Product.txt
PRODUCT=animcompiler
VERSION=1.0.0
PLATFORMS="Win32 Linux"
CONFIGURATIONS="Debug Release"
CATEGORY=compilers
24
The Asset pipeline for Just Cause 2: Lessons Learned
Current pipeline - Needy• Big change in the coding process
• Code base evolves towards smaller libraries
• Really fast compile times
• Helps keeping track of version conflicts
• Allows for very rapid development and testing
25
The Asset pipeline for Just Cause 2: Lessons Learned
Current pipeline – WAF• Light weight build system for code and data
• Open source (BSD), source is ~80kb
• Dependency checking
• Multi core support
• Easily to maintain and extend
• Replaces our in house Project Compiler system
• Python
A small snippet from a task definition using waf:
cls = Task.simple_task_type(‘.anim’, ‘${PYTHONC} ${ANIMC} –i ${SRC} –o ${TGT} –p ${PLATFORM}’)cls.ext_in = (‘.anim’, )cls.ext_out = (‘.ban’, )Task.TaskBase.classes[‘.anim’].scan = FindDependenciesFromAnimation
26
The Asset pipeline for Just Cause 2: Lessons Learned
Current pipeline - ADF• Avalanche Data Format
• Serialization framework
• C/C++, Python support
• Xml <-> Binary support
• Hash based versioning for binary files
• Error handling
• The ability to access data both from Python and C/C++ enables seamless cross language interchange
• Cross platform support
• E.g. pass data from tools to/from consoles
27
The Asset pipeline for Just Cause 2: Lessons Learned
Current pipeline - ADF
TM2VertexData = DefStruct( "TM2VertexData", ( "PositionIndex", Array( uint32 ) ), ( "TangentSpaceIndex", Array( uint32 ) ), ( "ColorIndex", Array( uint32 ) ), ( "UVIndices", Array( uint32 ) ) ) TM2Model = DefStruct( "TM2Model", ( "Name", String ), ( "Triangles", Array( uint32 ) ), ( "Vertices", TM2VertexData ), ( "MorphTargets", Array( TM2MorphTarget ) ), ( "Positions", TM2PositionData ), ( "TangentSpaces", Array( float ) ) )
Type definitions:
28
The Asset pipeline for Just Cause 2: Lessons Learned
Current pipeline - ADF
from Adf import Contextctx = Context()ctx.ReadTypeLibrary('mytype.adf')objs = ctx.ReadInstancesFromXml(‘infile.xml')objs[0].Name = 'newvalue'ctx.WriteInstancesToFile( ‘outfile.bin', objs, BIG_ENDIAN )
Read / Write
mytype.pymytype.py AdfPack.exeAdfPack.exe mytype.adfmytype.adf
mytype.hmytype.h
compiler.pycompiler.py
Infile.xmlInfile.xml
Outfile.binOutfile.bin
29
The Asset pipeline for Just Cause 2: Lessons Learned
Current pipeline - Compilers• C++
• Shared libraries, loaded from Python
• Removed dependencies on old game code
• Replaced in-house code with standard code
• fopen() rather than CFile()
• Decrease complexity, increase portability
• Scripting and logic in python
• Data processing in Python or C++
• E.g. read/write/process Havok files using C++
30
The Asset pipeline for Just Cause 2: Lessons Learned
Current pipeline - Compilers• Python
• Extreme turn around times
• Vast collection of built in modules
• We use these a lot:
• optparse, ctypes, md5, numpy, cStringIO…
• Many of the modules have the backend implemented in C++
• Most compilers were completely rewritten
• Cut a lot of dependencies
• Code size down to a 1/10th
31
The Asset pipeline for Just Cause 2: Lessons Learned
Current pipeline – Dependency Checking• Waf handles the mechanics
• Scanners find the dependencies
• Waf caches the result
• MD5 checksums
• Command lines arguments
• Source/Dependency file contents
• A new compiler path will trigger a compilation
• Very handy when updating packages with Needy
32
The Asset pipeline for Just Cause 2: Lessons Learned
Current pipeline – Dependency Checking• A small example of scanning a file for other file references
# Reading a text file containing file referencesdef FindDependencies(path): f = open(path, "rb" ) lines = f.readlines() f.close() return [ x.strip() for x in lines ]
33
The Asset pipeline for Just Cause 2: Lessons Learned
Current pipeline• Platform independence
• Possible to run on win32, linux, ps3, xenon etc.
• Helps keeping the code clean
• Running on linux
• We wanted to run on linux for the file system performance
• Run win32 exe’s through WINE
• Wine is a Win32 implementation in linux. It’s not an emulator.
• Intermediate solution before all code was rewritten
34
The Asset pipeline for Just Cause 2: Lessons Learned
Current pipeline• Incremental auto builder
• Syncs from version control
• Only builds the things necessary
• Full auto builder
• Compares incremental build with full build
• Finds dependency problems
• Find bugs in data formats
35
The Asset pipeline for Just Cause 2: Lessons Learned
Current pipeline• Change of mindset
• Keep it simple!
• Each component should do one thing only
• …and do it well!
36
The Asset pipeline for Just Cause 2: Lessons Learned
Current pipeline• To sum things up so far
• We rewrote ~20 compilers (out of ~30) in 6 man weeks
• Using old code as blue print
• Remaining compilers took longer due to more complex work flows
• Code size decreased to 1/10th
• Performance increased in all cases
• 10% to 1000%, 100% was quite common
• Possible to use compilers in the previous pipeline with little effort
• Early benefits of the new pipeline
37
The Asset pipeline for Just Cause 2: Lessons Learned
Future improvements• Reloading of assets in run time (in progress)
• Virtual File System
• Can load data from hard drive, archives or web server
• Distributed builds
• Looking into using AFS, a distributed file system
• ssh builder01 modelcompiler.exe -i mymesh1.tm …
• ssh builder02 modelcompiler.exe -i mymesh2.tm …
38
The Asset pipeline for Just Cause 2: Lessons Learned
Take Aways• A few thing we’ve learned so far
• Some are quite trivial but also easy to overlook
39
The Asset pipeline for Just Cause 2: Lessons Learned
Take Aways• Hardware
• Defrag the hard drives
• Try using other file systems, ext3, ext4, afs.
• Our pipeline is a lot faster on linux than on win32
• Two separate discs that are designated source/target
• Ram discs/SSD are getting cheaper
40
The Asset pipeline for Just Cause 2: Lessons Learned
Take Aways• How to implement the compilers
• No assumptions on working directory or any other variables
• Keep the compiler data driven, it’s quite easy
• Project specific options could be read from a file or command line
• Create a simple plugin system
• Extremely easy in python
41
The Asset pipeline for Just Cause 2: Lessons Learned
• Example of loading python files. I.e. creating a plug-in system:
for root, dirs, files in os.walk(‘c:/myapp/plugins’): if not root in sys.path: sys.path.append(root) for filename in files: if not filename.endswith('.py'): continue # We do an import() instead of an execfile() so that # the loaded code can act as a regular python file module = __import__(filename.replace('.py', '') )
Take Aways
42
The Asset pipeline for Just Cause 2: Lessons Learned
Take Aways• How to implement the compilers
• Write portable code
• E.g. remove DirectX, Win32 API
• ‘-e’ for endianess rather than ‘-p’ for platform
• Beware of case sensitiveness from the start
• Also, beware of ‘\\’ versus ‘/’
43
The Asset pipeline for Just Cause 2: Lessons Learned
Take Aways• How to organize the data
• Keep source data separate from compiled data
• When creating new file formats
• Avoid doing it “by hand”. Create a generalized file type.
• Include a version number
• Store file dependencies easily accessible
• Write the game ready format with efficiency in mind
• Memory, performance
44
The Asset pipeline for Just Cause 2: Lessons Learned
Take Aways• How to optimize the compilers
• Write compilers with parallelism in mind
• No “global variables” such as reused temp files on disc
• Disc cache thrashing
• It comes down to the access patterns
• Do as few reads/writes as possible
• Use tools like Process Explorer or vmstat
45
The Asset pipeline for Just Cause 2: Lessons Learned
Take Aways• How to optimize the compilers
• Do reads and writes in memory
• Lowers the disc usage
• E.g allocate large buffer and write into it
• Avoid temp files altogether
import cStringIOmemfile = cStringIO.StringIO()MultipleWritesToAFile( mydata, memfile)f = open(options.output, "wb" )f.write( memfile.getvalue() )f.close()memfile.close()
46
The Asset pipeline for Just Cause 2: Lessons Learned
Take Aways• Dare to do stuff!
• If the compiler code is unreadable/inefficient
• Will it be faster/more productive to rewrite it?
• Rewriting will probably introduce new bugs
• But you will also…
• Probably throw away many old bugs as well
• Increase your understanding of the work flow
• Get a chance to improve design/performance more directly
47
The Asset pipeline for Just Cause 2: Lessons Learned
Take Aways• Dare to do stuff!
• Rewriting doesn’t mean you throw the code away
• You will use the code as a blue print of what, or what not, to do
• It’s easy to rewrite something that does the same thing, only better!
48
The Asset pipeline for Just Cause 2: Lessons Learned
Take Aways• Separate code
• Use one code base to read/write data in compilers
• Use another code base for loading the data in game
• Keeps down code dependencies
• Keeps the compilers small and simple
• Less code to maintain in the compiler
• Decreased code size down to 1/10th
• Less code to compile, faster turnaround times
• Suitable for file formats that rarely change
49
The Asset pipeline for Just Cause 2: Lessons Learned
Take Aways• Keep the build working!
• Only replace old code with new working code
• Down time is both costly and stressful
• Keep it simple!
50
The Asset pipeline for Just Cause 2: Lessons Learned
Links• Python – http://www.python.org/
• waf – http://code.google.com/p/waf
• wine – http://winehq.org
• jqplot – http://www.jqplot.com/
• protobuf – http://code.google.com/p/protobuf/
• Download slides and source code on our site!
51
The Asset pipeline for Just Cause 2: Lessons Learned
Q & A
www.avalanchestudios.se/presentations
Download slides and source code on our site!
52
The Asset pipeline for Just Cause 2: Lessons Learned