High Performance Computing Use of SPEEDES for BMDSsim Bill Grenard Metron High Performance Computing (858) 792-8904 [email protected] Dr. Ron Van Iwaarden.

High Performance Computing

Use of SPEEDES for BMDSsim

Bill GrenardMetron High Performance Computing(858) [email protected]

Dr. Ron Van IwaardenMetron High Performance Computing(719) [email protected]

26 July 2004

2


Overview

Metron

SPEEDES

Approach to BMDSsim: Clustering

Summary

3


Metron, Inc.

Highest Clearance HeldAcademic ConcentrationHighest Degree Attained

Statistics18%

Applied Math14%

Engineering36%

Computer Science

32%Ph.D.35%

M.S.25%

B.S.40%

SI/TK18%

Top Secret38%

Secret41%

Uncleared3%

•Founded in 1984•Products include Naval Simulation System, SPEEDES simulation engine•Technical staff of 63 . . . 30 based in Reston VA and 33 in San Diego CA.•Onsite simulation experts at Pentagon (OPNAV), JNIC, COMPACFLT and JWARS

•Missile Defense projects include - SPEEDES simulation engine for MDWAR wargames -- MDA- Bayesian algorithm development for target designation -- MDA- Theater missile defense simulation -- Navy

4


SPEEDESSynchronous Parallel Environment for Emulation and Discrete-Event Simulation

Powerful optimistic-processing parallel processing engine Developed, maintained, and distributed by Metron since 1996

– Open source code downloadable to qualified users– On-line documentation– On-line change request system

Primary users: MDA projects– MDWAR– IMDSE (conservative-processing variant)

• Windows NT• SPEEDES just works and lets them model

– C2BMC– ABL

Other SPEEDES-related efforts:– Air Force Research Lab (Rome Labs)

• Rome Labs funded iterator improvement, incorporated in Version 2• Distributed Information Enterprise Modeling and Simulation (DIEMS)• Parallel multiple-course-of-action SPEEDES enhancement

– NASA KSC– Independent IV&V of the JSIMS Simulation Engine (the CCSE)

5


Metron’s SPEEDES TeamRon Van IwaardenPhD, Applied Mathematics, University of ColoradoMS, Applied Mathematics, University of ColoradoBS, Mathematics, University of Colorado

SPEEDES developer for 7 yearsLead, SPEEDES development and documentationMDWAR wargame support at the JNIC

Scott ShupeBS, Computer Sci., Rensselaer Polytechnic Inst.

SPEEDES Developer for 5 yearsSPEEDES enhancements for MDWARSPEEDES load balancing studies for AFRLSPEEDES FAAsim GUIHLA RTI developer

At MITRE, developed RTI verification test set

Gary BlankMS, Comp. Science, University of VirginiaBS, Applied Mathematics, Brown University

SPEEDES developer for 7 yearsLead, SPEEDES Multi-COA Enhancement projectSPEEDES enhancements for MDWARSPEEDES support of AFRL (DIEMS, GIEsim)SPEEDES FAAsim prototypeHLA RTI developerHLA Federations

Jacob BurckhardtBS, Computer Science, UC Berkeley

SPEEDES developer for 7 yearsLead, JSIMS sim engine IV&VSPEEDES enhancements for MDWARSPEEDES testing

SPEEDES Configuration Management

Steve HeistandBS, Aerospace Engineering, Iowa State Univ

SPEEDES developer for 6 monthsExtensive previous work in tuning, porting and developing of parallel algorithms

Aircraft flight dynamics modelsJet turbine engine modelsGlobal weather modelsASCI codes.

6


SPEEDES Early Development and Modern Versions

Chosen as framework for MDWAR in late 1996– Early beta versions concentrated on functionality rather than reliability– Frequently buggy, undocumented, poor performing

Version 1.0 (November 2000)– Completed the Unified API– Added the SPEEDES User’s Guide*– Added the API Reference Manual*– Much of the obsolete code was removed

Version 2.0 (September 2001)– Added object proxy attribute subscription– Added automatic lazy re-evaluation– General code optimization (size and speed)

*On-line at www.speedes.com

7


Approach to BMDSsim: Clustering

Clustering can lead to high performance federations– Retains ability for easy debugging modes

– Can link up through shared memory or TCP/IP

– Design allows for MDWAR Standard Gateway (MSG) connections

– Elements could hook together in variety of fashions• Optimistic: Full optimistic time management with rollbacks

– Includes the option of connecting through shared memory on the same machine or TCP/IP for those that are remote.

• Conservative: Linked through MSGs

• Playback: A element could be replaced by an MSG Playback for standalone testing/debugging

• Any combination of the above

8


How clustering works

MDWAR

ABLGhost

C2BMCGhost

Hi-FiThreatGhost

ABLMDWARGhost

High FidelityThreatMDWAR

Ghost

C2BMCMDWARGhost

MSG

MSG

MSG MDWAR

ABLGhost

C2BMCGhost

Hi-FiThreatGhost

SPEEDESServer

Massive HPC

Cluster 1High Fidelity

ThreatMDWARGhost

Cluster 2

C2BMCMDWARGhost

Cluster 3

ABLMDWARGhost

SPEEDES can connectsimulations using conservative time management . . .

. . . or as one large simulationusing optimistic time managementand high-speed communications

Cluster 4

ABL = Airborne Laser simulationC2BMC = Command and Control, Battle Management, and Communications systemMDWAR = Missile Defense Wargaming and Analysis ResourceMSG = MDWAR Standard Gateway

9


Summary

Main SPEEDES focus is and will be on stability, and reliability– Performance has already been proven– Continuing use in wargames provides rigorous test environment– Mature set of tools help optimize performance, minimize overhead

• SPEEDES instrumentation• MDWAR simulation instrumentation and analysis tools• Rules of thumb

Continuing improvement– Changes for usability– Reduction in memory and CPU footprint– AFRL funded parallel course-of-action engine (due March 2006)

MDA can have confidence in high performance, low risk for BMDSsim


Back-ups

11


Lessons learned

SPEEDES has been extraordinarily resilient– Almost all performance problems have been due to improper

modeling

– Framework bugs are now rare• Significantly impacted development in the early (< v 0.8) years

– Proxy mechanism is solid but tightly couples models• Use of proxy updates has decreased significantly• Proxies use often indicates incorrect modeling

– Not communicating through message sets

– Unnecessary or excessive notifications

• Attribute subscription carries a small penalty– Often used to simply unsubscribe totally to proxy updates

12


Lessons learned (cont)

Performance tuning requires analysis tools– Real time performance does not come for free

• Built up suite of analysis tools– SPEEDES instrumentation has been extensive and varied– MDWAR has many tools to analyze the instrumentation files

• Rules of thumb learned about modeling• New APIs added to improve parallelism• SPEEDES overhead is minimal (usually 10s of micro-seconds/event)

– Recent tests using MDWAR 5.0 (SPEEDES 2.1) on 1 node (optimistic) shows a ~15% framework overhead.

– BTW is within 20% of sequential on SPEEDES 2.1, should be 10-15% with 2.2

I/O is a killer. Data collection has a minor impact

– Biggest problem is making sure we collect enough

13


SPEEDES, IMDSE Release HistoryVersion Type Title Date of release

beta6 major SPEEDES 12/1/1995beta6.1 minor SPEEDES 1/1/1996

0.1 major SPEEDES 4/4/19970.11 minor SPEEDES 5/9/19970.2 major SPEEDES 5/27/1997

0.21 minor SPEEDES 8/26/19970.22prerelease minor SPEEDES 9/12/1997

0.22 minor SPEEDES 9/15/19970.3 major SPEEDES 11/3/1997

0.4prereleaseForHuntsville major SPEEDES 1/9/19980.4 major SPEEDES 1/13/1998

0.41 minor SPEEDES 3/4/19980.42 minor SPEEDES 3/17/19980.43 minor SPEEDES 4/14/19980.44 minor SPEEDES 4/24/19980.45 minor SPEEDES 5/6/19980.5 major SPEEDES 6/22/1998

0.51 minor SPEEDES 7/13/19980.6 major SPEEDES 8/17/1998

0.61 minor SPEEDES 11/15/19980.7 major SPEEDES 1/13/19990.7 major SPEEDES 1/13/1999

0.71 minor SPEEDES 3/18/19990.72 minor SPEEDES 5/7/1999

0.721 minor SPEEDES 5/28/19990.73 minor SPEEDES 6/30/19990.8 major SPEEDES 9/10/1999

0.81 minor SPEEDES 12/20/19990.81.1 patch SPEEDES 2/1/20000.81.2 patch SPEEDES 2/1/2000

0.9 major SPEEDES 5/22/20000.9.1 patch SPEEDES 9/29/2000

1.0 major SPEEDES 11/6/20001.1e engineering SPEEDES 5/21/20012.0 major SPEEDES 9/17/2001

2.0.1 patch SPEEDES 12/14/20012.0.1e engineering SPEEDES 4/17/2002

0.5 minor IMDSE 2/28/20031.0 major IMDSE 4/30/20032.1 minor SPEEDES 9/1/2003

2.1.1 patch SPEEDES 10/1/2003

High Performance Computing Use of SPEEDES for BMDSsim Bill Grenard Metron High Performance Computing (858) 792-8904 [email protected] Dr. Ron Van Iwaarden.

Documents