Data Management and Streaming Strategies in Drakensang Online

Data Management and Streaming Strategies

in Drakensang Online

Andre “Floh” WeissflogHead of Development BerlinBigpoint GmbH

Browser Games Forum 2011

2

• Hack & Slay F2P MMO.• Embedded in browser.• No client download or installation.• Impressive 3D graphics.• 8 months from concept to Closed Beta.

3

4

Team & Technology Background

5

Drakensang Online is developed by the same team that also worked on the previous single-player Drakensang “boxed titles”.

Very skilled and experienced team, very solid and proven production pipeline and processes (and very little online experience).

The technology behind our client and server is the Nebula3 engine (~250k lines of C/C++, 12 years of continuous development).

Epic stories will be written about how we went from building offline boxed-titles to creating an MMO running in a browser in 8 months.

…but not Today…

Today we’ll talk about data managementand streaming in Drakensang Online…

Data Management Core Demands

6

• Browser Game Experience Click & play, no installation.

• Fast Startup Time Start in seconds, not minutes

• On-Demand Streaming: Only download what’s needed

• Client-Side Caching: CDN traffic cost, user experience

• Robustness: The web is slow and unreliable.

• Minimize Data Size: But keep graphics quality high!

Drakensang Online: The Assets

7

• 50 interconnected maps• 240 monsters• 120 NPCs• 1400 items• 300 animated effects

317 MB uncompressed

139 MB compressed

27035 filesAsset Types

TexturesAudioMeshesNavigationLocaleMapsAnimsOther

Two Main Achievements

8

Aggressive Data Size ReductionWe’re working with 3% of the data size we’ve been used to.

This is mainly an organizational achievement.

All departments have to agree on data size budgets……even if it’s hard (e.g. for the graphics guys)

Seamless Data Streaming During GameplayThat’s mainly a programming achievement.

Client engine must support asynchronous I/O from the ground up.

Maps as “static web pages”

9

This was really the initial idea behind our data management.

Our maps reference many small asset files “somewhere on the web”, just like a static web page.

All asset files are loaded asynchronously. Render loop can deal with assets which are not yet loaded.

Really clever local file caching prevents unnecessary HTTP requests (down to no HTTP traffic at all if the cache is completely up-to-date).

Asset Size Reduction: Extreme Granularity

10

Our maps are built from really small and simple 3D objects:

Extremely granular set of reusable objects. Each with only a few dozen to a few hundred triangles.

One map may have 100k of those micro 3D objects, but only a few dozen DIFFERENT objects.

That’s very good for data size. But…

OMG TEH

DRAW CALLS !!

Nebula3 uses hardware instancing and runtime-baking to reduce draw calls from about 5000 down to 250 per frame.

Asset Size Reduction: Textures

11

Typical texture dimensions:

256 x 256128 x 12864 x 64

All textures are DXT compressed (and then – like all other files – ZIP compressed)

Bump textures use DXT5NM compression, aka the Carmack-Trick(that’s why they’re grey, not purple).

Texture types:colorbumpemissivespecular

Typical download sizes for textures are somewhere between 5 kByte and 40 kByte.

Textures are aggressively shared and reused.

Asset Size: Lights, Shadows& Decals

12

All lighting and shadowing is completely dynamic. No light-maps needed.

We use deferred Pre-Pass-Lighting to enable “infinite point-lights” with moderate fill-rate requirements.

Volumetric Decals are used to hide the tile-nature of the ground.

Post-Effects add bloom, fog, color saturation and color balancing.

Two Streaming Strategies:

13

1) ON DEMAND:Client requests data right when it’s needed.IO requests are handled asynchronously.May take several seconds until data is ready.Until then:

Render a placeholder or…Render nothing.

2) BACKGROUND STREAMING:Low-priority background thread just for cache warm-up.One streaming list per map created during build process.Download and update items in local file cache.Very speculative, can’t really predict what’s needed next.

The web as a (really unreliable) hard-disc

14

Nebula3 always had a powerful I/O system:

• async I/O is standard, not the exception• “massively multi-threaded”, currently 10 I/O-threads• pluggable filesystems• all file paths are URLs

We built an “HTTP filesystem” on top of this:

• only uses HTTP GET requests• transparently replaces local-disc file I/O• MD5 checksums for all files• ZIP compression for all files• hierarchical caching system• directory walking and file-exists checks• CDN support• reasonably fail-safe (high latencies,

dropped connections, corrupt downloads, tampering…)

“Massively Multi-Threaded IO”

15

IO Dispatcher Thread

Main Thread

Render Thread

Audio Thread

… Thread

Incoming IO Requests

IO Threads

Cache (Local HD)

Cache Hit?Yes

Web Server

No

Update CacheServed IO Requests

• up to 10 HTTP-GET requests concurrently in flight (tweakable)• low-prio IO requests don’t block high-prio requests• no difference in high-level code between file I/O and http I/O

Hierarchical File Cache

16

ROOT MD5 KEYe4680c8372bc6c527043a97631b404d7

Directory “Table Of Content” File

Leaf TOCs Leaf TOCs Leaf TOCs

1. Daily-Build-Process: compress files and create per-file MD5 hashes.2. Each directory has a “Table Of Content” file with all MD5 hashes.3. TOCs are also compressed and MD5’d.4. ...with one single “Directory TOC” at the top.5. …and a single Root MD5 Key for the Directory TOC.

• Client receives Root MD5 Key at startup.

• Downloads TOC files if not in cache…• TOC files provide MD5 keys for files.• Zero HTTP GETs if cache is up-to-date!

The Cache Chain

17

We basically have a 3-level cache for assets:

RAM DISC CDNPROXY

CDNORIGIN

Level 1 Level 2 Level 3

RAM Cache Hit?Resource objects are shared in RAM using their ResourceIDs.

DISC Cache Hit?MD5-named file must exist in local file cache.MD5 of actual file-content must match build-time MD5 hash.

CDN Proxy Cache Hit?MD5-hash appended to HTTP GET URL, forces a cache miss if proxy only hasout-of-date file of the same name.

CDN Origin Server has the definite data.

404’s Considered Harmful

18

Remember this from the previous page:

RAM DISC CDNPROXY

CDNORIGIN

1st 2nd 3rd 4th

“Sh*t happens”.

Trying to load non-existing assets will happen with such a complex data set.

HTTP requests for non-existing files will cut through all the way to the origin server.

CDN’s usually cache 404’s for a few seconds, but it’s better to not bother the web servers with 404’s at all.

Our Table-Of-Content-Files let us detect non-existing files on the client without a server roundtrip.

Conclusion

19

• Data management is the job of the whole team.• Careful, early asset size planning.• Team must commit on size budgets.• “Programming Magic” alone doesn’t solve problems.

• Programming, Level Design and Art Department must work hand in hand.• Oh, and: Check your CDN cost plan (if you pay per HTTP request, our solution is sub-optimal)

20

Thank You Questions?

Find us on21

Bigpoint GmbH

Alexanderstraße 510178 BerlinGermany

Bigpoint Inc.

500 Howard StreetSuite 300San Francisco, CA 94105

Bigpoint Distribuição de Entretenimento Online Ltda.

Av. Brig. Faria Lima3729 cj. 52804538-905 São PauloBrazil

Bigpoint GmbHAndre WeissflogHead of Development Berlin

Drehbahn 47-4820354 Hamburg Germany

Tel +49 40.88 14 13 - 0Fax +49 40.88 14 13 - 11

[email protected]

Contact us

Bigpoint International Services Limited

1 Villa ZimmermannTa’Xbiex TerraceXBX 1035 Ta’XbiexMalta