ROSS: Parallel Discrete-Event Simulations on Near Petascale Supercomputers Christopher D. Carothers Department of Computer Science Rensselaer Polytechnic Institute [email protected]
Mar 21, 2016
ROSS: Parallel Discrete-Event Simulations on Near Petascale Supercomputers
Christopher D. Carothers Department of Computer ScienceRensselaer Polytechnic Institute
2
Outline Motivation for PDES Overview of HPC Platforms ROSS ImplementationPerformance ResultsSummary
MotivationWhy Parallel Discrete-Event Simulation
(DES)?– Large-scale systems are difficult to understand– Analytical models are often constrained
Parallel DES simulation offers:– Dramatically shrinks model’s execution-time– Prediction of future “what-if” systems
performance– Potential for real-time decision support
• Minutes instead of days• Analysis can be done right away
– Example models: national air space (NAS), ISP backbone(s), distributed content caches, next generation supercomputer systems.
Model a 10 PF Supercomputer• Suppose we want to model a
10 PF supercomputer at the MPI message level
• How long excute DES model?– 10% flop rate 1 PF sustained– @ .2 bytes/sec per flop @ 1%
usage 2 TB/sec– @ 1K size MPI msgs 2 billion
msgs per simulated second– @ 8 hops per msg 16 billion
“events” per simulated second– @ 1000 simulated seconds
16 trillion events for DES model– No I/O included !!– Nominal seq. DES simulator
100K events/sec
• 16 trillion events @ 100K ev/sec
5+ years!!!Need massively parallel
simulation to make tractable
Blue Gene /L Layout
CCNI “fen”• 32K cores/ 16 racks• 12 TB / 8 TB usable RAM• ~1 PB of disk over GPFS• Custom OS kernel
Blue Gene /P Layout
ALCF/ANL “Intrepid”•163K cores/ 40 racks• ~80TB RAM• ~8 PB of disk over GPFS• Custom OS kernel
Blue Gene: L vs. P
How to Synchronize Parallel Simulations?parallel time-stepped simulation:
lock-step execution
PE 1 PE 2 PE 3
barrier
VirtualTime
parallel discrete-event simulation:must allow for sparse, irregular
event computations
PE 1 PE 2 PE 3
VirtualTime
Problem: events arrivingin the past
Solution: Time Warp
processed event
“straggler” event
Massively Parallel Discrete-Event Simulation Via Time Warp
Local Control Mechanism:error detection and rollback
LP 1 LP 2 LP 3
Virtual
Ti
me
undostate ’s
(2) cancel“sent” events
Global Control Mechanism:compute Global Virtual Time (GVT)
LP 1 LP 2 LP 3
Virtual
Ti
me
GVT
collect versionsof state / events& perform I/O
operationsthat are < GVT
processed event
“straggler” event
unprocessed event
“committed” event
Our Solution: Reverse Computation...
• Use Reverse Computation (RC)– automatically generate reverse code from model source– undo by executing reverse code
• Delivers better performance– negligible overhead for forward computation– significantly lower memory utilization
if( qlen < B )qlen++
delays[qlen]++else
lost++
NB
on packet arrival...
Original
if( b1 == 1 )delays[qlen]--qlen--
elselost--
Reverseif( qlen < B )
b1 = 1qlen++
delays[qlen]++else
b1 = 0lost++
Forward
Ex: Simple Network Switch
Beneficial Application Properties
1. Majority of operations are constructive– e.g., ++, --, etc.
2. Size of control state < size of data state– e.g., size of b1 < size of qlen, sent, lost,
etc.3. Perfectly reversible high-level
operationsgleaned from irreversible smaller operations– e.g., random number generation
• Destructive assignment (DA):– examples: x = y;
x %= y;– requires all modified bytes to be saved
• Caveat:– reversing technique for DA’s can degenerate to
traditional incremental state saving
• Good news:– certain collections of DA’s are perfectly reversible!– queueing network models contain collections of
easily/perfectly reversible DA’s• queue handling (swap, shift, tree insert/delete, … )• statistics collection (increment, decrement, …)• random number generation (reversible RNGs)
Destructive Assignment...
RC Applications • PDES applications include:
– Wireless telephone networks– Distributed content caches– Large-scale Internet models –
• TCP over AT&T backbone • Leverges RC “swaps”
– Hodgkin-Huxley neuron models– Plasma physics models using PIC– Pose -- UIUC
• Non-DES include:– Debugging– PISA – Reversible instruction set
architecture for low power computing
– Quantum computing
if( qlen < B ) qlen++ delays[qlen]++else lost++
B
packet arrival...
Original
if( b1 == 1 ) delays[qlen]-- qlen--else lost--
Reverseif( qlen < B )
b1 = 1 qlen++ delays[qlen]++else b1 = 0 lost++
Forward
Local Control Implementation
Local Control Mechanism:error detection and rollback
LP 1 LP 2 LP 3
Virtual
Ti
me
undostate ’s
(2) cancel“sent” events
• MPI_ISend/MPI_Irecv used to send/recv off core events
• Event & Network memory is managed directly.– Pool is allocated @ startup
• Event list keep sorted using a Splay Tree (logN)
• LP-2-Core mapping tables are computed and not stored to avoid the need for large global LP maps.
Global Control ImplementationGVT (kicks off when memory is
low):1. Each core counts #sent, #recv2. Recv all pending MPI msgs.3. MPI_Allreduce Sum on (#sent -
#recv)4. If #sent - #recv != 0 goto 25. Compute local core’s lower
bound time-stamp (LVT).6. GVT = MPI_Allreduce Min on
LVTsAlgorithms needs efficient MPI
collectiveLC/GC can be very sensitive to OS
jitter
Global Control Mechanism:compute Global Virtual Time (GVT)
LP 1 LP 2 LP 3
Virtual
Ti
me
GVT
collect versionsof state / events& perform I/O
operationsthat are < GVT
So, how does this translate into Time Warp performance on BG/L & BG/P?
Performance Results: Setup• PHOLD
– Synthetic benchmark model– 1024x1024 grid of LPs– Each LP has 10 initial events– Event routed randomly among all LPs based on a configurable “percent
remote” parameter– Time stamps are exponentially distributed with a mean of 1.0 (i.e., lookahead is
0).• TLM – Tranmission Line Matrix
– Discrete electromagnetic propagation wave model– Used model the physical layer of MANETs– As accurate as previous “ray tracing” models, but dramatically faster…– Considers wave attenuation effects– Event populations grows cubically outward from the single “radio” source.
• ROSS parameters– GVT_Interval number of times thru “scheduler” loop before computing GVT.– Batch number of local events to process before “check” network for new
events.• Batch X GVT_Interval events processed per GVT epoch
– KPs kernel processes that hold the aggregated processed event lists for LPs to lower search overheads for fossil collection of “old” events.
– Send/Recv Buffers – number of network events for “sending” or “recv’ing”. Used as a flow control mechanism.
7.5 billion ev/sec for 10% remote on 32,768 cores!!
2.7 billion ev/sec for 100% remote on 32,768 cores!!
Stable performance across processor configurations attributed to near noiseless OS…
Performance falls off after just 100 processors on a PS3 cluster w/ Gigabit Eithernet
12.27 billion ev/sec for 10% remote on 65,536 cores!!
4 billion ev/sec for 100% remote on 65,536 cores!!
Rollback Efficiency = 1 - Erb /Enet
Model a 10 PF Supercomputer (revisited)
• Suppose we want to model a 10 PF supercomputer at the MPI message level
• How long excute parallel DES model?
• 16 trillion events @ 10 billion ev/sec
~27 mins
Observations…• ROSS on Blue Gene indicates billion-events per
second model are feasible today!– Yields significant TIME COMPRESSION of current models..
• LP to PE mapping less of a concern…– Past systems where very sensitive to this
• ~90 TF systems can yield “Giga-scale” event rates.• Tera-event models require teraflop systems.
– Assumes most of event processing time is spent in event-list management (splay tree enqueue/dequeue).
• Potential: 10 PF supercomputers will be able to model near peta-event systems
– 100 trillion to 1 quadrillion events in less than 1.4 to 14 hours– Current “testbed” emulators don’t come close to this for
Network Modeling and Simulation..
Future Models Enabled by X-Scale Computing
• Discrete “transistor” level models for whole multi-core architectures…
– Potential for more rapid improvements in processor technology…
• Model nearly whole U.S. Internet at packet level…
– Potential to radically improve overall QoS for all • Model all C4I network/systems for a whole
theatre of war faster than real-time many time over..
– Enables the real-time“active” network control..
Future Models Enabled by X-Scale Computing
• Realistic discrete model the human brain– 100 billion neurons w/ 100 trillion synapes
(e.g. connections – huge fan-out) – Potential for several exa-events per run
• Detailed “discrete” agent-based model for every human on the earth for..
– Global economic modeling– pandemic flu/disease modeling– food / water / energy usage modeling…
But to get there investments must be made in code that are COMPLETELY parallel from start to
finish!!
Thank you!!• Additional Acknowledgments
– David Bauer – HPTi– David Jefferson – LLNL for helping us get
discretionary access to “Intrepid” @ ALCF– Sysadmins: Ray Loy (ANL), Tisha Stacey
(ANL) and Adam Todorski (CCNI)• ROSS Sponsers
– NSF PetaApps, NeTS & CAREER programs– ALFC/ANL