SICSA student induction day, 2009Slide 1 Multithreading RePast Models Alex Voss 1, Jing-Ya You 2, Eric Yen 2, Simon Lin 2, Ji-Ping Lin 3, Andy Turner 4.

SICSA student induction day, 2009 Slide 1

Multithreading RePast Models

Alex Voss1, Jing-Ya You2, Eric Yen2, Simon Lin2,

Ji-Ping Lin3, Andy Turner4

1School of Computer Science, University of St Andrews2Academia Sinica Grid Computing, Academia Sinica,

Taiwan3Center for Survey Research, Academia Sinica, Taiwan

4School of Geography, University of Leeds

Workshop on Future Directions in Agent Based Modelling

Leeds, UK, June 2010

SICSA student induction day, 2009 Slide 2

Overview

• Something about the model we want to build on migration

• Quite a bit about how we tweaked the model to make use of multiple CPUs/cores

• A bit about what we will do next and questions we want to explore

• My interest in this…

SICSA student induction day, 2009 Slide 3

Migration in Taiwan

• Migration has been an important factor in Taiwanese social development and influenced by outside factors since the 1600s

• Aim is to test existing theories of migration constructively and to investigate recent developments such as increased outward migration to China.

• Timely as Taiwan is running another census in 2010.

• Based on work conducted by Ji-Ping Lin of the Academia Sinica Center for Survey Research on Migration using the 1990 and 2000 Taiwan Population and Housing Census

SICSA student induction day, 2009 Slide 4

SimTaiwan: Migration in Taiwan

• Based on Taiwan 2000 Population and Housing Census

• Dataset is individual-level but with restricted variables

• Held at Academia Sinica

• Need to identify additional datasets to complement census

• Issues with data protection

• Need to scale up to ca. 22 million individuals

• They are heterogeneous agents with quite a large number of attributes and history.

SICSA student induction day, 2009 Slide 5

SimTaiwan Tests

• Four different model implementations:1. Naïve single-threaded model2. Improved single-threaded model3. Initial multi-threaded model4. Improved multi-threaded model

• Test runs with each of these models to measure:1. Wallclock and CPU time2. Memory usage3. Code hotspots4. Worker thread activity (where applicable)

SICSA student induction day, 2009 Slide 6

Test Code and Parameters

• Simplified model with only fertility and mortality, same for all measured models

• 250k male and 250k female random initial population, running for 365 ticks (=days)

• Measurements taken using JProfiler 6.0.4– CPU sampling (5 sec intervals)

– Memory allocations recording

• JVM Parameters: -Xmx8192M –Xss128M

• Hardware: Dell PowerEdge R610 with 2xXeon E5504 @ 4x2GHz (8 cores total) and 16GB RAM

SICSA student induction day, 2009 Slide 7

Naïve Serial Version

• More time spent in RePast scheduling code than in model code because events are scheduled for each individual agent every step.

SICSA student induction day, 2009 Slide 8

Improved Serial Version

• Event scheduled on DemographicsContext, code iterating through individual agents

• Wallclock time down from 5:32 to 2:23

• Opens up opportunities for parallelising code as well…

SICSA student induction day, 2009 Slide 9

Initial Parallel Version

• Need to partition data to allow multiple worker threads to exploit multiple CPUs & cores

• PartitionedContext keeping agents in separate HashSets that can return independent Iterators for use by multiple threads.

• ThreadPoolExecutor with configurable number of worker threads (here 8)

• Initial version brings only modest / no improvement, wallclock time in some runs > improved serial code

• Max. CPU utilisation ~ 200% (top)

SICSA student induction day, 2009 Slide 10

Initial Parallel Version (II)

• Worker threads blocking a lot on monitors placed around RePast constructs.

• Main issue seems to be that use of RandomHelper is not thread-safe

• Simulation schedule relatively minor issue

• Some contention around simulation objects

SICSA student induction day, 2009 Slide 11

Improved Parallel Version

• Overloading some of RePast’s code to make it thread safe.

• Reducing scope of monitor objects used and pulling code parts that are safe out of synchronized sections

• Introducing thread-local variable containing a per thread random number generator:

protected static ThreadLocal<Uniform> uniform = new ThreadLocal<Uniform>() { @Override protected Uniform initialValue() { RandomEngine generator = new MersenneTwister((int)System.currentTimeMillis()); return new Uniform(generator); }};

SICSA student induction day, 2009 Slide 12

Improved Parallel Version (II)

• Monitor contention is eased signficantly

• Wallclock running time down to 1:03 and max. CPU utilisation up to ~ 600%

• Time spent in serial code for analysis and production of charts is now significant

SICSA student induction day, 2009 Slide 13

What have we learned/developed?

• Advice on structuring RePast code– Parallelise using PartitionedContext

– Iteration instead of scheduling events

• RePast does put some barriers in the way but should be possible to overcome

• Speed-up initially not as much as hoped for but was overcome by introducing thread-local random number generators

• Can we factor this work into development of RePast?

• Or present as tutorial?

SICSA student induction day, 2009 Slide 14

Next Step: Debugging/Profiling

on the Grid• Tests to establish optimum number of partitions

and threads vs no. of agents

• Verification of the code and sensitivity analysis

• Repeated runs to uncover rare events & need to repeat runs to obtain comparable average figures

• Availability of high-memory machines will become an issue once we scale up to full 22 million agents;

– 48GB server available at ASGC

– Upgrade / purchase of server at St Andrews planned

SICSA student induction day, 2009 Slide 15

Questions

• What will happen when we make the model more complex?

• What decisions about the model (will) affect the degree of parallelism and running times?

• How many CPU cores can we effectively utilise?– Need machine with more cores as well as more

memory

– This is now becoming affordable thanks to AMD

• Commodity computing is what we are interested in – less skills involved (?) and availabilty for social scientists

SICSA student induction day, 2009 Slide 16

My Interests…

• Not about building the most sophisticated model

• or the highest performance one but…

• about making ABM framework(s) (RePast) usable for social scientists interested in population-level phenomena,

• addressing the practical issues of developing and using agent-based models in anger

• cf. challenges outlined by Peter McBurney today