SRB 1

Safety and reliability are paramount con-cerns in rocket motor design because ofthe enormous cost of typical payloadsand, in the case of the Space Shuttle and

other manned vehicles, for the crew’s safety. Inthe spring of 1999, for example, a series of threeconsecutive launch failures collectively cost morethan US$3.5 billion. The most notorious launchfailure, of course, was the tragic loss of the SpaceShuttle Challenger and its seven crew members.Thus, there is ample motivation for improvingour understanding of solid rocket motors(SRMs) and the materials and processes onwhich they are based, as well as the methodol-ogy for designing and manufacturing them.

The use of detailed computational simulationin the virtual prototyping of products and de-vices has heavily influenced some industries—for example, in automobile and aircraft design—but to date, it hasn’t made significant inroads inrocket motor design. Reasons for this include themarket’s relatively small size and the lack of suf-ficient computational capacity. Traditional de-sign practices in the rocket industry primarily

use top-down, often one-dimensional modelingof components and systems based on gross ther-momechanical and chemical properties, com-bined with engineering judgement based onmany years of experience, rather than detailed,bottom-up modeling from first principles.Moreover, there has been a tendency to study in-dividual components in isolation, with relativelylittle emphasis on the often intimate couplingbetween the various components. For example,SPP1—an industry-standard code for analyzingsolid propulsion systems—includes a fairly de-tailed model of the propellant thermochemistry,but no structural analysis and no detailed modelof internal flow.

One of our primary goals at the Center forSimulation of Advanced Rockets (CSAR) is todevelop a virtual prototyping tool for SRMsbased on detailed modeling and simulation oftheir principal components and the dynamic in-teractions among them. Given a design specifi-cation—geometry, materials, and so on—wehope to be able to predict the entire system’s re-sulting collective behavior with sufficient fidelityto determine both nominal performance char-acteristics and potential weaknesses or failures.Such a “response tool” could explore the spaceof design parameters much more quickly,cheaply, and safely than traditional build-and-test methods. Of course, we must validate such a

21 COMPUTING IN SCIENCE & ENGINEERING

VIRTUAL PROTOTYPING OF SOLIDPROPELLANT ROCKETS

Researchers seek a detailed, whole-system simulation of solid propellant rockets undernormal and abnormal operating conditions. A virtual prototyping tool for solid propellantrocket motors based on first principles models of rocket components and their dynamicinteractions meets this goal.

A S C I

MICHAEL T. HEATH AND WILLIAM A. DICK

Center for Simulation of Advanced Rockets, UIUC

1521-9615/00/$10.00 © 2000 IEEE

22 COMPUTING IN SCIENCE & ENGINEERING

capability through rigorous and extensive com-parison with data for known situations to haveconfidence in its predictions for unknown situa-tions. Although it is unlikely that simulation willever totally replace empirical methods, it can po-tentially dramatically reduce the cost of thosemethods by identifying the most promising ap-proaches before building actual hardware.

Challenges in rocket simulation

Solid propellant boosters are the “heavylifters” of the space launch industry. Most of theworld’s large, multistage launch vehicles—in-cluding the Ariane, Delta, Titan, and SpaceShuttle—employ two or more SRBs in the ini-tial stage to provide 80% or more of the im-mense thrust needed to lift a payload in excessof 10,000 pounds off the launch pad and propelit the first few tens of miles above Earth. Beyondthis point, subsequent stages—typically liquid-fueled—take over into orbit and beyond.

SRMs are notably simpler than liquid rocketengines.2 The latter have far more moving parts(pumps, valves, and so on) and require storingand handling of liquids that might be cryogenicor potentially hazardous. SRMs, though, havealmost no moving parts (often only a gimballednozzle for thrust vector control), and the com-posite solid propellant (containing both fuel andoxidizer) forms the combustion chamber. Themain disadvantage of SRMs is that once ignited,combustion is essentially uncontrollable: thepropellant burns at maximum rate until ex-hausted. Thus, solid motors are ideal for the ini-tial stages of flight, when raw power is more im-portant than finesse, and then liquid-propellantrockets take over for the portions of flight re-quiring more delicate maneuvering. Despitetheir relative simplicity, SRMs are still fiendishlycomplex in terms of the chemical and thermo-mechanical processes that take place during fir-

ing, as well as the design and manufacturingprocesses required to make them reliable, safe,and effective.

Figure 1 shows a schematic drawing of a typi-cal SRM—the major parts are indicated alongwith the types of mathematical models thatmight be used to represent them. Reality, ofcourse, is considerably more complex than this2D picture, and we note the following majorchallenges in achieving our goals.

• The complex behavior of SRMs requiresfully 3D modeling to capture the essentialphysics adequately. Examples include thecombustion of composite energetic materi-als; the turbulent, reactive, multiphase fluidflows in the core and nozzle; the globalstructural response of the propellant, case,liner, and nozzle; and potential accident sce-narios such as pressurized crack propaga-tion, slag ejection, and propellant detona-tion.

• The coupling between components isstrong and nonlinear. For example, the load-ing due to fluid pressure deforms the solidpropellant, which changes the geometry ofthe fluid flow, which in turn affects pressure,and so on. Similarly, the burn rate increaseswith pressure and vice versa.

• The geometry is complex and changes dy-namically as the rocket consumes propel-lant. The inclusion of slots and fins, whichforms a star-shaped cross-section, enhancesthe amount of burning surface area. What-ever its initial shape, the propellant surfaceregresses at a pressure-dependent rate as thepropellant burns, and discrete representa-tions of the solid and fluid components, aswell as the interface between them, mustadapt accordingly.

• The spatial and temporal scales are ex-tremely diverse. For example, processes

Combustion interface

Combustion-injectionboundary layer model

Compressibleturbulence LES model

Solid propellant Interior cavity

Igniter Insulation

Case Nozzle Exhaustplume

Thermo-mechanicalfoam model

Thermo-visco-elasticmodel

Thermo-elastic model

Elasticity/ablation model

LESmodel

Figure 1. Ide-alized solidpropellantrocket. Majorcomponentsare indicatedin red, our ini-tial simulationmodels ingreen.

MARCH/APRIL 2000 23

such as combustion and crack propagationoccur on micron length scales and mi-crosecond time scales, or less, which are en-tirely infeasible to treat a two-minute burnof a 125-foot-long rocket.

• Manufacturing and transportation con-straints necessitate the use of numerousjoints, including field joints where motorsegments are assembled at the launch site.This significantly complicates the geometryand structural response of the motor and in-troduces potential points of failure.

• Modeling and simulating each componentis challenging both methodologically andcomputationally. Although there is consid-erable experience in the field in modelingthe various rocket motor components, amore fundamental understanding of theconstitutive and energetic properties of ma-terials and of the processes they undergo re-quires much greater detail along with teras-cale computational capacity.

• Modeling and simulating component cou-pling is even more demanding because it re-quires not only still greater computationalcapacity, but it also demands that the corre-sponding software modules interact in amanner that is physically, mathematically,and numerically correct and consistent.When data are transferred between compo-nents, they must honor physical conserva-tion laws, mutually satisfy mathematicalboundary conditions, and preserve numeri-cal accuracy, even though the correspond-ing meshes might differ in structure, reso-lution, and discretization methodology.

• Integrated, whole-system SRM simulationrequires enormous computational capac-ity, currently available only through mas-sively parallel systems that have thousandsof processors. Thus, the software integra-tion framework, mesh generation, numer-ical algorithms, input/output, and visual-ization tools necessary to support suchsimulations must be scalable to thousandsof processors.

In September 1997, CSAR embarked on anambitious plan to tackle these daunting chal-lenges and produce a virtual prototyping tool forSRMs.3 This article is a progress report almosttwo years into our five-year plan.

Our initial plans seemed audacious, but thesubstantial resources our sponsor (the US De-partment of Energy’s Accelerated Strategic

Computing Initiative program) provided let usassemble a team of over 100 researchers, in-cluding roughly 40 faculty, 40 graduate stu-dents, and 20 staff (research scientists, pro-grammers, and postdoctoral associates), thatrepresented 10 departments across our univer-sity. This diverse group provided the broad ex-pertise needed in combustion, fluid dynamics,structural mechanics, and computer science,but it also presented the additional challenge ofcoordinating a large collaborative project thatcuts across traditional departmental boundaries.We organized our effort along basic discipli-nary lines without regard to the academic de-partments of the participants. This has had thesalutary effect of inducing collaboration amongfaculty and students in a given discipline, suchas fluid dynamics, regardless of which depart-ment they might occupy. Cross-cuttingteams—such as System Integration and Valida-tion and Specification—draw members from allfour disciplinary groups and require an addi-tional level of collaboration.

Staged approach

We realized from the outset that a project ofthis complexity would require a staged approach:we would need to learn to walk before we couldrun (much less fly). The primary axes of com-plexity in our problem are physical and geomet-ric (see Figure 2). Physical complexity refers tothe detail and sophistication of physical modelsemployed and the degree of coupling amongthem. Geometric complexity refers to the di-mension of the problem and the degree of detailand fidelity in representing a real SRM. Inessence, we wish to move along the diagonal ofthis diagram over time. In this spirit, we definedthree successive generations of integrated rocketsimulation codes:

Weaklycoupled

Accidents

Joints

Star grain

3D

2D

1D

Fullycoupled

GEN0

GEN1family

GEN2family

Detailed

Physical complexity

Geo

met

rical

com

plex

ity

Figure 2. Ourcode develop-ment followsthis stagedapproach withincreasingcomplexity incomponentmodels andcoupling.

24 COMPUTING IN SCIENCE & ENGINEERING

• GEN0: 2D ideal rocket with steady-stateburning at chamber pressure, power law forpropellant regression, Euler equations forfluid flow, a rigid case, linearly elastic pro-pellant, and one-way coupling from fluid tosolid. We based its physical parameters onthe Space Shuttle reusable solid rocket mo-tor (see sidebar). GEN0 was intended pri-marily as a warm-up exercise.

• GEN1: Fully 3D whole-system simulationcode using relatively simple componentmodels, two-way coupling, and reasonablyrealistic geometry approximating that of theSpace Shuttle RSRM. Star grain of ShuttleRSRM is included, but not joints, inhibitors,or cracks. Solid components include vis-coelastic propellant and linearly elastic case.Fluid component is an unsteady, viscous,compressible flow, with a large-eddy simu-lation turbulence model but with no parti-cles, radiation, or chemical reactions in the

flow. Combustion model assumes homoge-neous surface burning and pressure-depen-dent regression rate. There is full, two-wayaeroelastic coupling between fluid and solidcomponents. Development of GEN1 wasexpected to span the first three years of thefive-year project.

• GEN2: Fully capable rocket simulation toolwith detailed component models, complexcomponent interactions, and support forsubscale simulations of accident scenariossuch as pressurized crack propagation, slagaccumulation and ejection, and potentialpropellant detonation. GEN2 includesmore detailed geometric features, such asjoints and inhibitors, and also includes moredetailed and accurate models for materialsand processes based on separate subscalesimulations. GEN2 was expected to spanthe last three years of the five-year project,overlapping with the final year of GEN1.

We chose the Space Shuttle Reusable Solid Rocket Motor asour primary simulation target for a variety of reasons, includ-ing its national importance, its public visibility, its fairly typicaldesign, and the availability of detailed specifications andextensive test data. We outline here the basic technical factsabout the Space Shuttle RSRM.1,2 Figure A shows a compositedrawing of the RSRM.

Height: 126.11 ftDiameter: 12.16 ftWeight: 149,275 lb empty; 1,255,415 lb full Case: High-strength D6AC steel alloy, 0.479 in. to 0.506 in.

thick Nozzle: Aluminum nose-inlet housing and steel exit cone,

with carbon-cloth phenolic ablative liners and glass-clothphenolic insulators. Nozzle is partially submerged and ismovable for thrust vector control.

Insulation: Asbestos-silica-filled nitrile butadiene rubber Propellant (material percent by weight):

Ammonium perchlorate oxidizer: 70Powdered aluminum fuel: 16Polybutadiene polymer (PBAN) binder: 12Epoxy curative agent: 2Ferric oxide burn rate catalyst: trace

Propellant grain: 11-point star-shaped perforation in headend of forward segment, aft-tapered cylindrical perforationin remaining segments. Liquid and solid ingredients are firstthoroughly mixed into a thick paste, then curative agent isadded before mixture is vacuum-cast into a mold and thencured in a “slow” oven for several days. Consistency of

resulting composite solid is similar to that of a pencil eraser. Igniter: Solid rocket pyrogen igniter mounted in forward end,

47.5-in. long and containing 134 lb of TP-H1178 propellantTotal launch weight: 4.5 million lb (including two SRBs, ex-

ternal tank, orbiter, and payload)Maximum thrust: 3,320,000 lb. force (each SRB)Acceleration: Lift-off 1.6 g (maximum 3.0 g)Launch timeline:Liquid engines fire: –6.0 sec

SRB igniter initiated: 0.0 secLift-off pressure: 564 psia reached at 0.23 secAll exposed propellant ignited: 0.3 secMaximum operating pressure: 914 psia reached at 0.6 secRoll program begins: 10 secStar grain burnout: 21 secLiquid engines throttled down: 30 secMach 1 reached: 40 secSolid propellant burnout: 111 secSRB separation: 126 sec

Velocity at separation: 3,100 mphAltitude at separation: 25 nmi

References1. Design Data Book for Space Shuttle Reusable Solid Rocket Motor, Thiokol

Space Operations, Publication No. 930480, TRW-16881, Revision A,

Brigham City, Utah, 1997.

2. A.J. McDonald, “Return to Flight with the Redesigned Solid Rocket Motor,”

Proc. AIAA/ASME/SAE/ASEE 25th Joint Propulsion Conf., AIAA Paper No. 89-

2404, AIAA Press, Reston, Va., 1989, pp. 1–15.

Space Shuttle Reusable Solid Rocket Motor

MARCH/APRIL 2000 25

Progress to date

We assembled the integrated GEN0 codefrom existing in-house modules for fluids, solids,and combustion components, and we completedit in May 1998. We ran it with modest levels ofparallelism on a shared-memory SGI PowerChallenge. Computed results agreed reasonablywell with predictions of classical 1D theory, butwe didn’t extensively validate it because we neverintended GEN0 as a realistic, high-fidelity sim-ulation; we saw it simply as a start-up system in-tegration exercise to get our team accustomedto working together. Visit www.csar.uiuc.edu toview animations of our results.

We based the subsequent GEN1 code onnewly written or substantially modified in-housecodes for the various modules. We completed asimplified, serial, but fully 3D version of it in Oc-tober 1998. Its principal simplifications includedour use of a strictly cylindrical geometry (no stargrain, which is a slot-and-fin geometry used to

increase the propellant’s initial burning surfacearea), no support for interface regression due toburning (not a significant factor for short burntimes, but obviously necessary for longer runs),the requirement that the solid and fluid meshesmust match at the interface to simplify data trans-fer, and our use of a linearly elastic (rather thanviscoelastic) model for the propellant.

Computed results without the star grain in thepropellant gave a head-end pressure at the on-set of steady burning of roughly half the empir-ically measured value for the Space ShuttleRSRM. This was not surprising, as the wholepoint of the star grain is to increase the exposedsurface area of propellant early in the burn,which increases pressure and thrust accordingly.Thus, implementing the star grain became highpriority. Another high priority was a fully paral-lel implementation, not only because this was animportant general goal, but also for the practi-cal reason that we could not otherwise make

Figure A. The Reusable Solid Rocket Motor (RSRM) is a primary booster for NASA’s Space Transportation System (STS). Section A-A shows11-point slot and fin star grain structure in the RSRM’s forward segment. Propellant in forward-center and aft-center segments form straight-walled cylinders; aft-segment propellant tapers outward to submerged nozzle. Inhibitors between segments are asbestos-filled carboxyl-terminated polybutadiene used to tailor burning surface to meet the motor’s thrust requirements.

26 COMPUTING IN SCIENCE & ENGINEERING

runs of significant duration at a reasonable reso-lution. By May 1999, we had completed a fullyparallel implementation of GEN1.4 With thestar grain geometry, computed head-end pres-sure was now within a few percentage points ofthe Space Shuttle RSRM’s measured value.

Recent efforts have focused on implementinginterface regression in both fluid and solid mod-ules and on allowing for nonmatching meshes atthe interface. Both fluid and solid modules nowsupport interface regression using an ALE (Ar-bitrary Lagrangian-Eulerian) approach in whichthe mesh moves to allow for the dynamicallychanging geometry. We also devised general-mesh-association and conservative data-inter-polation schemes that permit accurate datatransfer between components with nonmatch-ing meshes at the interface.

The main features of the solids codes (Rocsolid)include finite elements, unstructured hexahedralmeshes, linear elastodynamics, ALE treatment ofregression, implicit time integrations, multigridequation solvers, and Fortran 90 with MPI paral-lelism. The main features of the fluids codes(Rocflo) include finite volume; block-structuredmeshes; unsteady, viscous, compressible flow; ALEtreatment of moving boundaries; second-orderupwind total variation diminishing schemes; ex-plicit time integrations; and Fortran 90 with MPIparallelism. Figure 3 shows how the two compare.

Parallel scalability of Rocsolid, Rocflo, and theGEN1 code that integrates them has been excel-lent. We’ve run Rocflo on up to 2,048 processors,and we’ve run Rocsolid and GEN1 on up to 512processors on a variety of platforms, including theSGI Origin at NCSA, the Cray T3E at PittsburghSupercomputer Center, and all three ASCI plat-forms at the US Department of Energy laborato-ries. Figure 4 shows “scaled speedup,” meaningthat the problem size per processor is constant.The largest mesh sizes have about four million el-ements for the solid and seven million zones forthe fluid. Visualizations of some computational re-sults are shown in Figures 5 and 6.

The features we still have to implement to

Rocsolid

• Finite element

• Linear elastodynamics

• Unstructured hexahedral meshes

• ALE treatment of interface regression

• Implicit time integration

• Multigrade equation solver

• F90, MPI parallelism

Rocflo

• Finite volume

• Unsteady, viscous, compressible flow

• Block-structured meshes

• ALE moving boundaries

• Explicit time integration

• 2nd order upwind total variation

• F90, MPI parallelism

Figure 3. Solids and fluids codeshave different approaches tomulticomponent simulation.

(a)1

1,000

100

10

110 100 1,000

Processors

T3E

O2K

SP2

Sca

led

spee

dup

(b)1

1,000

100

10

110 100 1,000

Processors

Sca

led

spee

dup

(c)1

1,000

100

10

110 100 1,000

Processors

Sca

led

spee

dup

T3E

O2K

CLU

T3E

O2K

Figure 4. Graphs show scaled speedup of separate (a) Rocsolid and(b) Rocflo modules and of (c) integrated GEN1 code for Space Shut-tle RSRM. Problem size is fixed at 14,625 fluid grid points and 8,192solid elements per processor as number of processors grows. Com-puters used include Cray T3E, SGI Origin 2000 (02K), IBM SP2, andIntel Pentium cluster (CLU).

MARCH/APRIL 2000 27

complete the full GEN1 code include viscoelas-ticity, large deformations, and thermal effects inthe solid; large-eddy simulation turbulencemodel in the fluid; more detailed model of com-bustion and interface regression; and a flame-spreading model to capture ignition transients.

System integration issues

A number of technical issues arise in buildingan integrated, multicomponent code such asGEN1. First is the overall integration strategy,where the fundamental choice is between mod-ular and monolithic approaches. In building theGEN1 code, we chose a modular or partionedapproach in that we used separately developedcomponent modules and created an interface totie them together. In such an approach, sepa-rately computed component solutions might re-quire subiterations back and forth between com-ponents to attain self-consistency. This approachcontrasts with a more monolithic strategy inwhich all the physical components are incorpo-rated into a single system of equations and all therelevant variables are updated at the same time,thereby obviating the need to iterate to self-con-sistency. Although it has some theoretical ad-vantages, a monolithic approach impedes sepa-rate development and maintenance of individual

component modules by specialists in the respec-tive areas. The modular approach not only ex-pedites separate development and maintenance,it also allows swapping of individual moduleswithout replacing the entire code or even affect-ing the other modules. The modular strategyseemed to offer clear practical advantages in oursomewhat dispersed organizational setting, aswell as potentially letting users include com-mercial modules when appropriate.

However, even less coupled approaches arecommonly used in practice, in which entirely in-

Figure 6. Gas temperature computed by Rocflo in star grain regionof Space Shuttle RSRM near onset of steady burning, visualized byRocketeer. Values range from 3,364 K (magenta) to 3,392 K (red).Temperature is represented as (a) tint on interior surface of propel-lant and as (b) a series of translucent colored isosurfaces in interiorat slightly later time. Rocket is cut in half along lateral axis toimprove visibility.

Figure 5. Fully coupled 3D simulation of star grain in SpaceShuttle RSRM showing stress in propellant and gas pressureisosurfaces in slots and core region. Executed on 256-proces-sor SGI Origin 2000, visualized with Rocketeer, our in-housevisualization tool.

28 COMPUTING IN SCIENCE & ENGINEERING

dependent codes interact only offline (often withhuman intervention), perhaps through exchangeof input and output data files. By contrast, in ourmodular GEN1 code, the component modulesare compiled into a single executable code andthey exchange data throughout a run with sub-routine calls and interprocessor communication.

Another important issue is the physical, math-ematical, and geometric description of the inter-face between components, which in our case in-cludes the jump conditions combustion induces.A careful formulation of the interface boundaryconditions is necessary to satisfy the relevant con-servation laws for mass and linear momentum, aswell as the laws of thermodynamics.

Time-stepping procedures are another signifi-cant issue in component integration. Here, thetime steps for the fluid are significantly smallerthan those for the solid. Thus, we employ a pre-dictor-corrector approach in which the fluid is ex-plicitly stepped forward by several (say, 10) timesteps, based on the current geometry the solid de-termines. The resulting estimate of fluid pressureat the future time is then available for taking animplicit time step for the solid. However, the re-sulting deformation and velocity of the solidchange the fluid’s geometry, so the time-steppingof the fluid repeats and so on, until we attain con-vergence, which usually requires only a few subit-erations. Unless iterated until convergence, thisscheme is only first-order accurate, and it is “ser-ial” in that only one component computes at atime. Parallel time-stepping schemes that are sec-ond-order accurate without subiterations are pos-sible,6 and we plan to investigate these. But be-cause we map both fluid and solid componentsonto each processor, our current scheme does notprevent us from utilizing all processors.

As we mentioned briefly earlier, data transferbetween disparate meshes of different compo-nents is another highly nontrivial issue in com-ponent integration. In our approach, we let themeshes differ at the interface in structure, reso-lution, and discretization methodology, and in-deed this is the case in GEN1, because the fluidmesh is block-structured, relatively fine, andbased on cell-centered finite volumes, whereasthe solid mesh is unstructured, relatively coarse,and based on node-centered finite elements. Al-though in principle the two interface meshesshould abut because they discretize the same sur-face, in practice we can’t assume this because ofdiscretization or rounding errors. Thus, we havedeveloped general mesh association algorithmsthat efficiently determine which fluid points are

associated with each element (facet) of the solidinterface mesh;7 we then use the local coordinatesof the associated element to interpolate relevantfield values in a physically conservative manner.

Yet another thorny issue in component inte-gration is partitioning the component meshesfor parallel implementation in distributed mem-ory. The block-structured fluid mesh is relativelyeasy to partition in a highly regular manner, butthe unstructured solid mesh is partitioned by aheuristic approach, currently using Metis, whichoften yields irregular partitions (visit www-users.cs.umn.edu/~karypis/metis for further in-formation). Moreover, because we partition thetwo component meshes separately, there is noway to maintain locality at the interface—adja-cent partitions across the interface may not beplaced on the same or nearby processors. In ourcurrent approach, this effect complicates thecommunication pattern and might increase com-munication overhead, but it has not been a seri-ous drag on parallel efficiency so far. Neverthe-less, we plan to explore more global, coordinatedpartitioning strategies that will preserve localityand perhaps simplify communication patterns.

Software integration framework

Our overarching goal in CSAR is not only todevelop a virtual prototyping tool for SRMs, butalso to develop a general software framework andinfrastructure to make such integrated, multi-component simulations much easier. Toward thisend, we have initiated research and developmentefforts in several relevant areas of computer sci-ence, including parallel programming environ-ments, performance monitoring and evaluation,parallel input/output, linear solvers, mesh gen-eration and adaptation, and visualization.

Our work in parallel programming environ-ments has focused on creating an adaptive soft-ware framework for component integrationbased on decomposition and encapsulationthrough objects. This environment provides au-tomatic adaptive load balancing in response todynamic change or refinement, as well as high-level control of parallel components. It is pur-posely designed to maintain compatibility withcomponent modules in conventional languagessuch as Fortran 90, and it provides an automatedmigration path for the existing parallel MPI codebase. This work is in part an extension of the pre-viously developed Charm++ system, which hassuccessfully built a large parallel code for molec-ular dynamics, NAMD.8 Although this frame-

MARCH/APRIL 2000 29

work is not yet mature enough to serve as the ba-sis for the current GEN1 code, results for pilotimplementations of the GEN1 component mod-ules using the framework show great promise.

In performance evaluation, we are leveragingongoing development of the Pablo performanceenvironment,9 which provides capabilities for dy-namic, multilevel monitoring and measurement,real-time adaptive resource control, and intelli-gent performance visualization and steering of dis-tributed, possibly geographically dispersed, appli-cations. We used these tools to instrument theGEN1 code, view the resulting performance data,and relate it back to the application code’s callgraph to identify performance bottlenecks. Wealso updated the popular ParaGraph performancevisualization tool10 to display the behavior and per-formance of parallel programs using MPI, and weused it to analyze the GEN1 component modules.

Parallel I/O is often a bottleneck in large-scalesimulations, both in terms of performance and

of programming effort required to manage I/Oexplicitly. For large-scale rocket simulations, weneed good performance for collective I/O acrossmany processors for purposes such as periodi-cally taking snapshots of data arrays for visual-ization or checkpoint/restart. Because we use ge-ographically dispersed machines, we also needefficient and painless data migration betweenplatforms and back to our home site. Panda11

provides all these services—it runs on top ofMPI and uses the standard file systems providedwith the platforms we use. Its library offersserver-driven collective I/O, support for varioustypes of data distributions, self-tuning of perfor-mance, and integrated support for data migra-tion that exploits internal parallelism. We’ve al-ready incorporated Panda into Rocflo, and aredoing the same in Rocsolid. Again, early resultsare promising, and we plan to use Panda to han-dle parallel I/O within our main visualizationtool, Rocketeer (see sidebar).

Rocketeerby Robert A. Fiedler and John Norris

We need a powerful scientific visualization tool to analyzethe large, complex 3D data sets our whole system and sub-scale rocket simulations generate. After an extensive reviewof existing packages, we decided to develop our own tool,which we call Rocketeer. (Visit www.csar.uiuc.edu/F_software/rocketeer to download a user guide andsoftware.) The tool has a number of features that make itideal for visualizing data from multicomponent simulations,including its support for both structured and unstructuredgrids, cell-centered and node-centered data, ghost cells,seamless merging of multiple data files, automated anima-tion, and a smart reader for HDF (hierarchical data format).A particularly useful feature for visualizing field data in theinterior of an SRM is Rocketeer’s ability to depict translucentisosurfaces, which lets us view isosurfaces of temperature orpressure, for example, without blocking the view of otherisosurfaces deeper inside (see Figure B). Although our needto visualize data from SRM simulations specificallymotivated many of these features, Rocketeer is a broadlyuseful, general-purpose visualization tool.

Rocketeer is based on the Visualization Toolkit (VTK),1

which is in turn based on OpenGL to take advantage ofgraphics hardware acceleration. It currently runs onMicrosoft Windows and Unix/Linux. Planned enhancementsinclude implementing a client-server model so that we canperform most compute-intensive operations (in parallel) ona remote supercomputer while interactive control and ren-dering are performed locally on a graphics workstation.

Reference1. W.Schroeder, K. Martin, and W.E. Lorensen, The Visualization Toolkit: An

Object-Oriented Approach to 3D Graphics, Prentice Hall, New York, 1997.

Figure B. Rocketeer visualization of temperature profile computedby Rocflo in the star grain region of RSRM at the onset of steadyburning. Values range from 3,364 K (magenta) to 3,392 K (red).Temperature is represented by tint on the propellant fin’s surface andby a series of translucent colored isosurfaces inside the slot. The imageis cut in half along the rocket’s axis to enhance visibility. The hottestpoint in this imaged flow is in the stagnation region in the middle ofthe rocket’s head end; the coolest point is in the middle of the stargrain region’s open end.

30 COMPUTING IN SCIENCE & ENGINEERING

Validation

Comparison of simulation results with knowntest cases and laboratory data is essential to es-tablish this approach’s validity for science and en-gineering. With our GEN1 integrated coderapidly maturing, we have begun an aggressiveseries of computational experiments to verify andvalidate its efficacy and fidelity. Fluid-solid inter-action problems we use for validation includeflow over an elastic panel, flow over a wing(Agard Wing 445.6), and a model of inhibitor de-formation in an SRM. We hope also to be ableto make comparisons with test data for small lab-oratory-scale rockets through collaboration withvarious government laboratories. A larger-scaletest we are pursuing is to try to predict the pro-pellant “slumping” that led to failure for the Ti-tan IV SRB’s original design. Our ultimate testwill be comparison with the immense amount oftest data for the Space Shuttle RSRM taken dur-ing static firing tests after its redesign. These datainclude literally thousands of readings from straingauges, pressure curves, and so on for a liberallyinstrumented test version of the Shuttle RSRM.

New research directions

Our second-generation rocket simulationcode, GEN2, will require significantly more de-tailed and sophisticated component models thanthose in GEN1. Moreover, GEN2 will also sup-port accident scenarios that will require evengreater detail and finer resolution. To have thesenew models ready by the time we need them inGEN2, we have already begun extensive re-

search into these modeling issues, some of whichwe outline here.

Heterogeneous propellant flamesThe propellant in the Shuttle SRM consists of

a high-density packing of ammonium perchlo-rate (AP) oxidizer particles embedded in a ma-trix of powdered aluminum fuel and polymericbinder. The aluminum particles burn in the gas-phase products of AP-binder combustion. Thepropellant uses an initial bimodal distribution ofAP particle sizes of 20 and 200 microns, and theprimary combustion field is located within a fewtens of microns of the propellant surface. Flowdisturbances originating in the chamber due toacoustic-wave or mean-flow interactions andturbulence can affect this field, leading to what isknown as erosive burning.

Some of the heat generated in the combus-tion field is conducted to the propellant surface.This heat is responsible for the surface regres-sion and the conversion of solid propellant tocombustible gases. The resulting burning sur-face is not flat because the instantaneous re-gression rates of AP and binder differ, and ifcracks form in the propellant, the increase inpropellant surface can lead to a sharp increasein combustion intensity.

To describe the surface regression and to gen-erate boundary conditions for chamber flow, wemust resolve the 3D combustion field and coupleit with physical processes such as heat conduc-tion in a thin surface layer in the propellant andallow for pressure and thermal feedback fromthe chamber flow (see Figure 7).

Figure 7. The propellant surface model employs two sizes of AP particles imbedded in a fuel/binder matrix. Mixture mod-els for this matrix account for smaller AP particles. (a) 3D flames supported by AP and binder decomposition productgases for configuration appear at midline region. (b) Primary flame results from burning binder/AP mixture.

MARCH/APRIL 2000 31

Crack propagation in solid propellantThe initiation and propagation of one or more

cracks in the solid propellant or along the grain-case interface can dramatically affect rocket per-formance. Cracks create additional burning sur-faces in the SP, so the propagation of cracks cangreatly affect the pressure history in the rocketchamber, leading in some cases to the rocket’sdestruction. Modeling crack propagation inburning SP is quite complex, because the prob-lem is characterized by a tight coupling betweenstructural dynamics, combustion, and fluid me-chanics, along with a rapidly evolving geometry.Using a fully coupled explicit aeroelastic finite-element, finite-volume code, we are investigat-ing potential accident scenarios associated withthe presence of pre-existing cracks at various lo-cations in the solid booster, with special empha-sis on propellant-liner interfacial failures. We’reusing a novel cohesive-volumetric finite-elementscheme to capture the spontaneous motion ofthe crack, allowing for the possibility of crack ar-rest or branching. As the crack propagates andthe reacting gas pressurizes newly created crackfaces, the fluid domain undergoes complexchanges that an adaptive unstructured finite-vol-ume scheme can capture. As illustrated in Fig-ure 8, the reactive gas flow emanating from apre-existing radial crack in the SP interacts withthe core flow in the rocket motor. This gener-ates a region of high pressure on the leeward faceof the crack and a region of low pressure in thedownstream vicinity of the crack that leads tosubstantial deformation of the propellant grain.

Aluminum particle combustionThe solid propellants in modern rockets use

aluminum (Al) particles as fuel. As combustionproceeds, these particles in the propellant meltand agglomerate on the combustion interface. Acomplex process follows as Al droplets detachfrom the surface and are injected into the coreflow. The droplets, whose initial size varies from20 to 300 microns, are injected into a strongcross-flow, and they provide a significant sourceof heat as they burn to form Al oxides. Near-wallturbulence plays an important role in the dis-persion of Al droplets, and as a result heat re-lease is volumetrically distributed, althoughdominant mainly in the near-wall region. Al2O3is the primary product of combustion, and it ap-pears either as fine powder of micron size or aslarger residual particles. The deposition of Al2O3in the form of slag in the submerged nozzle canadversely affect motor performance.

The combustion of Al droplets strongly cou-ples the dispersed phase (droplets and Al2O3 par-ticles) with the continuous phase (the surround-ing core flow). We expect the GEN2 version ofRocflo to include a Eulerian implementation ofthe core flow and a Lagrangian implementationof the Al droplets and the larger oxide particles.The simulation will introduce tens of millionsof Al droplets into the flow at the combustioninterface according to a specified probability dis-tribution and local mass injection rate. The po-sition, velocity, temperature, and species con-centration of each droplet will be tracked in thesimulation over time by solving a set of ordinarydifferential equations. The effect of the sur-rounding flow will be parameterized in terms oflift and drag coefficients, heat and mass transfercoefficients, and droplet burn rate. So far wehave developed a detailed time-dependent 3Dsubsystem simulation of the flow, evaporation,and burning of an isolated Al droplet to obtainaccurate state-of-the-art parameterizations (seeFigure 9).

The individual problems just discussedare themselves examples of fluid−solid interactions, albeit on finerscales, for which we will be able to

use the same software integration framework asfor the global simulation of the SRM. In thisway, we hope to leverage much of our currentsoftware development effort in component in-tegration, and to be able to spin off these sub-scale simulations from the larger-scale systemsimulation in a highly compatible manner.

Figure 8. Effect ofpre-existingradial crack onmotor’s coreflow: Coloredcontours denotepressure in coreflow and hydro-static pressure insolid propellant.Note region ofhigh pressure(red) in deformedcrack and of lowpressure (blue)downstreamfrom crack.

32 COMPUTING IN SCIENCE & ENGINEERING

AcknowledgmentsWe thank our many colleagues at CSAR for their researchcontributions to this article. This program is truly acollaborative effort based on the technical strengths ofmany people. We thank Amit Acharya, Prosenjit Bagchi,S. Balachandar, Dinshaw Balsara, John Buckmaster,Philippe Geubelle, Changyu Hwang, Thomas L. Jackson,and Biing-Horng Liou for their contributions to the “Newresearch directions” section. The CSAR research programis supported by the US Department of Energy throughthe University of California under subcontract B341494.

References1. G.R. Nickerson et al., The Solid Propellant Rocket Motor Perfor-

mance Prediction Computer Program (SPP), Version 6.0, Tech. Re-port AFAL-TR-87-078, US Air Force Materials Lab., Edwards AirForce Base, Calif., 1987.

2. G.P. Sutton, Rocket Propulsion Elements, 6th ed., John Wiley &Sons, New York, 1992.

3. M.T. Heath and W.A. Dick, “Virtual Rocketry: Rocket ScienceMeets Computer Science,” IEEE Computational Science & Eng.,Vol. 5, No. 1, Jan.–Mar. 1998, pp. 16–26.

4. I.D. Parsons et al., “Coupled Multi-Physics Simulations of SolidRocket Motors,” Parallel and Distributed Processing Techniquesand Applications Conf., Vol. VI, CSREA Press, 1999.

5. P.V.S. Alavilli, D. Tafti, and F. Najjar, “The Development of an

Advanced Solid-Rocket Flow Simulation Program ROCFLO,” Proc.38th AIAA Aerospace Sciences Meeting and Exhibit, AIAA Press, Re-ston, Va., 2000.

6. C. Farhat and M. Lesoinne, “Two Efficient Staggered Proceduresfor the Serial and Parallel Solution of Three-Dimensional Nonlin-ear Transient Aeroelastic Problems,” Computer Methods in Ap-plied Mechanics and Eng., Vol. 182, Nos. 3 and 4, 2000.

7. X. Jiao, H. Edelsbrunner, and M.T. Heath, “Mesh Association:Formulation and Algorithms,” Proc. Eighth Int’l Meshing Round-table, Tech. Report 99-2288, Sandia Nat’l Labs., Albuquerque,New Mexico, 1999, pp. 75–82.

8. L. Kale et al., “NAMD2: Greater Scalability for Parallel MolecularDynamics,” J. Computational Physics, Vol. 151, No. 1, May 1999,pp. 283–312.

9. L. DeRose et al., “An Approach to Immersive Performance Visu-alization of Parallel and Wide-Area Distributed Applications,”Proc. Eighth IEEE Symp. High-Performance Distributed Computing,IEEE Computer Soc. Press, Los Alamitos, Calif., 1999, pp.247–254.

10. M.T. Heath and J.A. Etheridge, “Visualizing the Performance ofParallel Programs,” IEEE Software, Vol. 8, No. 5, Sept. 1991, pp.29–39.

11. Y. Cho et al., “Parallel I/O for Scientific Applications on Hetero-geneous Clusters: A Resource-Utilization Approach,” Proc. 13thACM Int’l Conf. Supercomputing, ACM Press, New York, 1999,pp. 253–259.

Michael T. Heath is the director of the Center for Sim-ulation of Advanced Rockets at the University of Illinois,Urbana-Champaign. He is also a professor in the De-partment of Computer Science, the director of theComputational Science and Engineering Program, anda senior research scientist at the National Center forSupercomputing Applications at the university. His re-search interests are in numerical analysis—particularlynumerical linear algebra and optimization—and in par-allel computing. He wrote Scientific Computing: An In-troductory Survey (McGraw-Hill, 1997), and has servedas editor of several journals in scientific and high-per-formance computing. He received a BA in mathemat-ics from the University of Kentucky, an MS in mathe-matics from the University of Tennessee, and a PhD incomputer science from Stanford University. Contacthim at CSAR, 2262 Digital Computer Lab., 1304 WestSpringfield Ave., Urbana, IL 61801; [email protected];www.csar.uiuc.edu.

William A. Dick is the managing director of the Centerfor Simulation of Advanced Rockets at the Universityof Illinois, Urbana-Champaign. He received a BS in me-chanical engineering from the University of Delawareand an MBA from the University of Illinois. Prior tocoming to the University of Illinois, he was a deputydirector and composites engineer in the National Sci-ence Foundation’s Engineering Research Center forComposites Manufacturing Science and Engineering.Contact him at CSAR, 2266 Digital Computer Lab.,1304 West Springfield Ave., Urbana, IL 61801; [email protected]; www.csar.uiuc.edu.

Figure 9. Results obtained from a time-dependent 3D simulation offlow and heat transfer from a spherical droplet in cross-flow. Thesimulation employs a resolution of 81 × 96 × 32 points along the ra-dial, circumferential, and azimuthal directions at Reynolds numberRe = U∞D/ν = 350, based on free stream velocity (U∞) and droplet di-ameter (D). At this Reynolds number, flow is unsteady with time-periodic vortex shedding. (a) Azimuthal velocity contours at an in-stant in time, where we can see an imprint of vortex shedding. (b)Temperature contours, where approaching flow is hotter (red) thanthe droplet (blue), which is considered isothermal.