Testing Blockbuster Games: Lessons for All Testers

GAJAH ANNUAL REPORT 2015 | ‹#›

Testing Blockbuster GamesTulay McNally,Alex Lucas

Challenge #1Game platforms like the Microsoft Xbox and Sony Playstation are inherently difficult to develop for.

Challenge #2In-house engines, custom renderers, and 1st party platforms mean that we can’t use off-the-shelf tools. We must build our own (where possible).

Challenge #3Nebulous requirements and frequent system overhauls. Achieving ‘fun’ is an iterative process.

Challenge #4Games are massive. Build times are long, binaries are huge, incredible amounts of graphics, data, and sound working (and needing testing) in concert.

Challenge #5Game quality is a function of the interoperability of many complex systems. The individual parts may ‘work’ while the whole does not.

Technical challenges in game quality

assurance.Games feature a level of scope and interconnectedness that

is not typically seen in other kinds of software. Our customers do not merely use our software; they project

themselves into our software as participants in a simulacrum of a real world.

In order to convey a convincing and fully realized world, our developers create a multitude of systems and features that

incorporate truly massive amounts of supporting data. The end result is a piece of software that is nearly impossible to verify

completely; a challenge further complicated by the need to also assure its experiential quality.

Many of the technical hurdles of supporting QA on a title like Dragon Age, while challenging, have been overcome with an ambitious investment in QA technology.

Recognizing that a key contributor to overall game quality is iteration opportunity, we target our QA technology to support increasing developer velocity and unblocking testers so they can provide timely and meaningful feedback.

BioWare QA Technology has developed sophisticated tools and telemetry platforms that are in use all across Electronic Arts. We’ve applied automation technology to solve traditionally non-QA problems, improved the content and development pipelines, and partnered with higher education to apply scientific research in addressing our QA needs.

Game Qualitycomes from iteration & experimentation.

QA TECHNOLOGY @BIOWARE

Pipeline ImprovementsIncrease developer velocity by enhancing build and content pipelines. Discover defects as close to creation as possible, as well as repairing them as they occur.

Data Visualization & TelemetryIncrease knowledge about the product at all stages of development. Things we historically used automation for are often better achieved via development telemetry.

Enhanced ToolsA broad range of bespoke tools, scripts, state editors and supporting technologies increase tester capacity and accuracy. Turn average testers into ‘rockstars’.

Machine LearningLeverage the capabilities of our engineers and partnerships to deliver context aware technology. Notable achievements on DA:I include duplicate bug detection and callstack analysis.

GAJAH ANNUAL REPORT 2015 | ‹#›

Dragon Age: Inquisition

QA TECHNOLOGY

EXAMPLES

Using FBScript to Support TestingON DRAGON AGE INQUISITION

FBScript is an integrated Python-based scripting language that allows the QA Technology team to rapidly locate and enumerate assets within the game.

Allows QA to identify ALL the assets in the game, including new and deprecated assets, where they exist, and any dependencies that may inject risk into the product.

For 200 hours of script development time, 4800 staff hours of verification were saved/discovered.

81% of the scripts provided verification of systems that where not scheduled for testing due to feasibility concerns.

Data-Driven TESTING

Significant ROIGames have massive amounts of data. Asset counts on Dragon Age run into the millions. Models, shaders, scripts, textures, animation, etc. are all referenced frequently across the code base, sometimes with 48-level-deep-object references, and this creates a unique level of challenge for QA. The number of unique permutations is astronomical and so, much of the ‘game’ cannot feasibly be verified. Additionally, this content is largely undocumented (with cause) and so much of this risk is also unknown. For example, how many unique ‘rock’ models are there in the game? How many are being used? Where are they being used? Are they being used correctly?

Using data-driven testing via FBScript allowed us to answer these questions. QA was able to form a picture of the game ‘as it was’ and not ‘as it was intended to be.’

QA, in addition to being able to verify whole swathes of previously untestable content, was able to pinpoint ares of particular risk and adjust testing strategy accordingly.

1

10

100

1000

Net Gain/Loss (hours)Total Dev Time (hours)

Pre-Flight Build SystemDRAGON AGE SERIES

Builds are typically an expensive process in games, and with the large number of developers on a title, along with many product SKUs to support, builds can break frequently.

CIS catches broken builds quickly, but developers are still blocked while waiting on a fix, and QA is not testing latest code.

By submitting to pre-flight instead of source-control, bad code is bounced, good code is checked-in, and CIS produces ‘always working’ and ‘latest code’ builds.

NO BROKEN BUILDS

Case Study: Pre-Flight Build System on Dragon Age Origins• Automated testing of change-lists across multiple SKUs and configurations.

• Does not replace CIS. Works with it by preventing bad code from entering source control in the first place.

• Number of monthly broken builds decreased from ~300 at peak to less than 30. Remaining build breaks where due to developers

circumventing the pre-flight system.

• Supported significant increase in developer velocity by ensuring developers and QA always had a working, up to the minute build.

Remember, game quality comes, in part, from iteration opportunity.

0

100

200

300

400

MAR '06 AUG '07

Pre-Flight Solution

Deployed

E3Demo

HEAT MAPS• Aggregate telemetry to pinpoint trends and adjust the

design of the game accordingly• Identify performance problems; frame-rate, memory

usage, latency• Support QA in making coverage evaluations

SESSION PLAYBACK• All gameplay session events are recorded via

integrated telemetry system• Events, bugs, crashes are all visualized both

spatially and temporally• Supports GPS-style playback to aid design

balancing and crash reproduction• QA is better able to target testing and communicate

concerns

DASHBOARDS• Effective delivery of key data for the project:

• Integrated and presented alongside our game telemetry

• Surfaced throughout the studio on TVs in common areas

• Branded to our projects• High visual fidelity and ‘coolness’ factor increased use

REPORTS• Customized implementation of Kibana & Elastic Search

• Kibana supports rapid report creation and the flat database gives us fast access

• Multiple customers:• Design use for game balance• Programming use for performance• Production & QA use for project health

DATA VISUALIZATION AND TELEMETRY

ADVANCED BUG FILING

CUSTOM

BUG TOOLS• Custom interface that sits on top of

existing industry standard tools like JIRA and DevTrack:

• Unique interface depending on project role: QA vs. Dev

• Details are filled out procedurally• Hot-links to content in editor for rapid

fixes• Advanced duplicate issue detection via

machine learning partnership with local university.

INTEGRATED WITH TELEMETRY

VISUALIZATION• All bugs are aligned with session telemetry• Active bugs can be visualized via a web-

based 2D map (featured) or in-game• Aids in identifying problem areas, untested

areas and rapid regression of ‘claimed fixed’

• Crashes are filed automatically, aggregated with ‘like’ and, due to captured session data, quickly resolved by programmers.

The games BioWare is developing for the future will be larger and more complex than anything we’ve ever made before. Procedural content generation is becoming common place in the industry, and if games were hard to test before, it’s only going to get more difficult.

To meet this challenge BioWare will continue to invest in QA Technology as a means to help deliver quality experiences for our players. As the scope of our games increase, we must invent new ways to assure quality in both our processes and the complex systems that define our games, while at the same time becoming better at informing development on the ‘fun’ of our product.

Player ModellingFormalize models of our players to better understand their preferred play-styles

Run player models against ‘beat reports’ that track the rhythm of game activities, to ensure high-levels of continuous engagement

MVB: Minimum Valuable BugBuilding an automated bug pipeline by adding procedurally generated repro-steps, indexed video aligned with telemetry, and additional machine learning

Create effective, resolvable bugs for development AND reduce tester burden when filing them

Heuristic AutomationProcedural content generation is driving need for procedural quality evaluation

Automatically apply value models for ‘fun’ or ‘interest’ against content

Reduce the amount of content that needs to have eyes on it during formative development stages

LOOKING FWD