ORNL is managed by UT-Battelle for the US Department of Energy Experiences with Applying Tapenade to a Modern 3D CFD Code Charlotte Kotas † Jacob Barhen † Bob Wilson † Mike Traweek ‡ EuroAD February 2017 † ORNL ‡ Office of Naval Research DISTRIBUTION STATEMENT A. Approved for public release. Distribution is unlimited.
22
Embed
Experiences with Tapenade and CFD - EuroAd Workshop... · 2017-02-17 · ORNL is managed by UT-Battelle for the US Department of Energy Experiences with Applying Tapenade to a Modern
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
ORNL is managed by UT-Battelle
for the US Department of Energy
Experiences with Applying Tapenade to a Modern 3D CFD Code
Charlotte Kotas†
Jacob Barhen†
Bob Wilson†
Mike Traweek‡
EuroADFebruary 2017
† ORNL
‡ Office of Naval Research
DISTRIBUTION STATEMENT A. Approved for public release. Distribution is unlimited.
2 Experiences with Tapenade and CFD
Outline
• Motivation
– Model validation using uncertainty and sensitivity analysis
• Application
– Computational fluid dynamics code, REX
• Applying Tapenade to REX
– Work in progress
3 Experiences with Tapenade and CFD
Motivation: Model Validation
• Experimental data contain uncertainties due to measurement bias, precision, and random factors
• Computational model predictions contain uncertainties due to parameters, simplifications in physical equations, and numerical techniques
Given these uncertainties, we want to combine experimental data with computational model responses to determine a best estimate for the independent variables, model parameters, and experimental responses
And use these best estimates to reduce the uncertainties
For details see:
D. G. Cacuci, Sensitivity and Uncertainty Analysis, Chapman & Hall (2003)
J. Barhen et al, “Consistent uncertainty reduction in modeling nonlinear systems”, SIAM Journal on Scientific Computing, 26 (2), 653-665 (2004).
and references therein.
4 Experiences with Tapenade and CFD
Notation• Let 𝒙 be input variables, 𝜽 be
model parameters, 𝒎(𝒙, 𝜽) be the computational model response,𝒆the be experimentally measured response, 𝒓 be the true system response, 𝜺 be experimental errors, and 𝜹 𝒙, 𝜽 be model errors
– 𝒓 = 𝒆 − 𝜺 = 𝒎 𝒙, 𝜽 + 𝜹 𝒙, 𝜽
• Let 𝒂 be a vector collecting 𝒙 and 𝜽 such that 𝒎 𝒙, 𝜽 = 𝒎(𝒂)
• Let 𝑪𝒛 be a covariance matrix computed from standard deviations in the elements of vector 𝒛
• Let ^ denote the “best estimate” of a quantity
Model
𝒙
𝜽
𝒎(𝒙, 𝜽)
5 Experiences with Tapenade and CFD
Sensitivity Analysis and Uncertainty Quantification (1)
• The sensitivities of the system describe the relative importance of individual input values in overall system responses.
• Mathematically, the sensitivity matrix (𝑺) is the derivative of the desired responses 𝑹 = 𝑹(𝒎, 𝒂). We assume that the computation model computes 𝑹 directly, thus,
– 𝑺𝒏𝒊 =𝒅𝑹𝒏
𝒅𝒂𝒊=
𝝏𝑹𝒏
𝝏𝒂𝒊+
𝝏𝑹𝒏
𝝏𝒎𝝂
𝝏𝒎𝝂
𝝏𝒂𝒊=
𝝏𝒎𝒏
𝝏𝒂𝒊
– To first order, 𝒎 = 𝒎0 + 𝑺 𝒂−𝒂𝟎
• The sensitivity matrix is used to propagate uncertainty from the inputs to the model responses (to the first order):
– 𝑪𝒎 = 𝑺𝑪𝒂𝑺𝑻, where 𝑺 is evaluated at 𝒂
– If independent variables, 𝑪𝒂 is diagonal and this becomes the more
familiar 𝜎𝑚2 =
𝜕𝑚
𝜕𝑥
2𝜎𝑥
2 +𝜕𝑚
𝜕𝑦
2
𝜎𝑦2 + ⋯
Tapenade computes
6 Experiences with Tapenade and CFD
Sensitivity Analysis and Uncertainty Quantification (2)
• We can combine the model and experimental responses to estimate the true response and reduce uncertainty.
𝜺 = 𝒆 − 𝒓 = 𝒆 − 𝒎 𝒙, 𝜽 + 𝜹 𝒙, 𝜽
≅ 𝒆 − 𝒎 𝒂 − 𝑺 ෝ𝒂−𝒂
• We create a generalized loss function 𝑄 based on the experimental error and the model constraints:
𝑸 =ො𝒓 − 𝒆ෝ𝒂−𝒂
⋮
𝑇 𝑪𝒆 𝑪𝒑 …
𝑪𝒑𝑇 𝑪𝒂 …
… … …
−1ො𝒓 − 𝒆ෝ𝒂−𝒂
⋮
• ℒ can be minimized for the best estimate of ො𝒓 and ෝ𝒂, and used to compute new covariance (uncertainty) estimates
ℒ = 𝑸 + 𝜆 𝒎 𝒂 − 𝒆 + 𝑺 ෝ𝒂−𝒂
7 Experiences with Tapenade and CFD
Sensitivity Analysis and Uncertainty Quantification (3)
• Fully turbulent, k-ω, two-equation turbulence model
– Turbulent kinetic energy, k
– Specific turbulent dissipation rate, ω
12 Experiences with Tapenade and CFD
Applying Tapenade to REX
• TAPENADE is an Automatic Differentiation program that operates by source transformation on C and FORTRAN 77/90/95 codes. It was developed at INRIA Sophia-Antipolis, and is currently maintained by Laurent Hascoët and Valerie Pascual.
• As REX is written in (mostly) FORTRAN 95, Tapenade can be applied with minor changes to the source code to bring the code to F95 standards or simplify things for Tapenade.
– “flush(11)” replaced with “call flush(11)”
– Tapenade option –linelength to allow lines longer than F95 standard
– Adding some intermediate variables manually where Tapenade needed help seeing array size. For example, MAXVAL(ABS(A(:))) becomes result1=ABS(A(:)); MAXVAL(result1)
– Replacing DOT_PRODUCT(a(:),b(:)) with SUM(a(:)*b(:))
13 Experiences with Tapenade and CFD
Tapenade “features”
• REX is complex. There are a few places where Tapenade gets confused.
– Wants to initialize at the top level a couple of local array variables
– Leaves in “#” in a couple of places, as in EXTERNAL #FUNCTION_NAME#
– Calls for nonexistent module PACKINGD. (Module PACKING exists, but not PACKINGD.)
– Puts in module variable that is not within local scope in initializing array.
• Original: A(:,:)=0.0, where A is brought into scope through “USE MYMODULE, ONLY: A”
• Adds: Ad(NX,NY)=0.0, but NX and NY are not in scope. “USE MYMODULE, ONLY: NX,NY” added by hand.
14 Experiences with Tapenade and CFD
External library: PETSc
• The PETSc library is designed for parallel, scalable solutions of PDEs (https://www.mcs.anl.gov/petsc/index.html)
– Distributed matrix and vector data structures
– Linear and non-linear solvers
– Written in C with interfaces for C, C++, Fortran, and Python
– Built on the MPI library
• REX uses PETSc to store distributed matrices for use in PETSc’s linear solver routines.
15 Experiences with Tapenade and CFD
Using PETSc (1)
• Tapenade is designed to generate calls to derivative versions of unknown functions if it thinks they are necessary
• For PETSc, setting the value of a vector looks like:
• PETSc’s linear solvers can be handled similarly to the example on Tapenade’s FAQ (https://www-sop.inria.fr/tropics/tapenade/faq.html#linearsolvers) for hand coding the SOLVE_D and SOLVE_B routines
– For linear solvers, hand coding is typically better than letting Tapenade compute the derivatives even when the solver source code is available.
• For forward mode, SOLVE becomes SOLVE_D which solves for 𝑦 and ሶ𝑦– 𝑦 = 𝐴−1𝑏 (as before)
• The message passing interface (MPI) library allows different compute nodes to communicate, sharing partial results and status messages.
• Automatic Differentiation adds data dependencies to the original program that must be accounted for in the communication patterns. Plain MPI libraries are not sufficient to handle these dependencies without additional hand coding.
• The Adjoinable MPI library (http://www.mcs.anl.gov/~utke/AdjoinableMPI/AdjoinableMPIDox/index.html) was written to alleviate pressure on code developers by providing an interface between AD tools and a modified MPI library suitable for computing adjoints.
– Tapenade attempts to add mpif.h and ampi/ampif.h, resulting in duplicate definitions
– AMPI_BARRIER calls become calls to AMPI_BARRIER_D rather than TLS_AMPI_BARRIER
– Tapenade generated calls to TLS_AMPI_ALLGATHER, TLS_AMPI_REDUCE, TLS_AMPI_ALLREDUCE, TLS_AMPI_BCAST, TLS_AMPI_SCATTER have wrong format
– Tapenade wants to create derivatives for MPI_INTEGER data types
20 Experiences with Tapenade and CFD
Using AMPI (3)
• Laurent Hascoet has been working with us to be sure that the AMPI library implements the missing functions
• Laurent Hascoet has been working to improve the Tapenade code as well
– We greatly appreciate his help!
• In addition, AMPI_Waitall is not implemented in the FORTRAN version of the AMPI library, as is noted on the AMPI website.
– As a workaround, we can loop over AMPI_WAIT calls
• Currently, the webpage user manual does not provide an example where both the “PlainC” and “Common” libraries are used. It may be necessary to link both libraries for code that contains undifferentiated routines.
– Both libraries have some common functions like AMPI_COMM_SIZE, so I’m not sure this was the original design
21 Experiences with Tapenade and CFD
Validating code
• Tapenade has some features to help validate its output against finite difference estimates.
– Command line options “debugTGT” and “debugADJ”
– Requires support code from Tapenade’s “ADFirstAidKit”
• ADFirstAidKit has both a C and Fortran version of “debugAD”, which are used differently but both require simultaneous two simultaneous runs of the code
– Fortran requires two executables with small changes between them
> ./test1 | ./test2
– C requires environment variables to be set so that the code can determine which phase is currently executing