Intro Dynamics DFI Closure
Analysis of Uncertain Dynamical Network Models
R.D. Berry1 H.N. Najm1 B. Sonday1
B.J. Debusschere1 H. Adalsteinsson1 Y.M. Marzouk2
1Sandia National Laboratories, Livermore, CA
2Mass. Inst. of Tech., Cambridge, MA
Stochastic Multiscale Methods: Bridging the Gap Between Mathematical Analysisand Scientific and Engineering Applications
BANF International Research Station, BANF, CanadaMarch 27 – April 1, 2011
SNL Najm Uncertain Networks 1 / 37
Intro Dynamics DFI Closure
Acknowledgement
R.G. Ghanem — Univ. Southern California, Los Angeles, CAO.M. Knio — Johns Hopkins Univ., Baltimore, MD
This work was supported by:
The US Department of Energy (DOE), Office of Advanced Scientific ComputingResearch (ASCR), Applied Mathematics program, 2009 American Recovery andReinvesment Act.
The DOE Office of Basic Energy Sciences (BES) Division of Chemical Sciences,Geosciences, and Biosciences.
BS acknowledges DOE Computational Science Graduate Fellowhip support,which is provided under grant number DE-FG02-97ER25308.
Sandia National Laboratories is a multiprogram laboratory operated by Sandia Corporation, a Lockheed MartinCompany, for the United States Department of Energy under contract DE-AC04-94-AL85000.
SNL Najm Uncertain Networks 2 / 37
Intro Dynamics DFI Closure
Outline
1 Introduction
2 Dynamical Analysis for Model Reduction
3 Data Free Inference
4 Closure
SNL Najm Uncertain Networks 3 / 37
Intro Dynamics DFI Closure
Motivation
Many physical systems are governed by network modelsElectric gridsBiochemical/chemical modelsInternet, communication networks, ...
Resulting models are complexLarge number of governing equations (dimension n)Large number of connections/reactionsStrong non-linearity – ODEs/DAEsLarge range of time scales – stiffness
Need for analysis and model reduction methodsKrylov projection methodsMethods based on dynamical analysis
– Automated identification of slow manifolds
SNL Najm Uncertain Networks 4 / 37
Intro Dynamics DFI Closure
Uncertainty in Network Models
Network ODE models typically rely on empirically-basedparameters/inputs
Uncertain parameters/inputsUncertain network structure
Need for dynamical analysis methods thatCan handle uncertaintyProvide model reduction with quantified fidelity
– accounting for uncertainty
Uncertain ODE systems, x(t) ∈ Rn
dxdt
= f (x;λ)
x(0) = x0
SNL Najm Uncertain Networks 5 / 37
Intro Dynamics DFI Closure
UQ Challenges in complex Network models
Bifurcations– Transitions between operating regimes, switching– Instability; Ignition
⇒ MC; Smooth observables; Multi-element local PC methods
Phase error growth and oscillatory dynamics– Uncertain dynamics over long time horizons
⇒ MC; Smooth observables; Time-shifting
High Dimensionality– Large number of uncertain parameters or degrees
of freedom
⇒ MC; Non-intrusive Sparse-Quadrature; Adaptive bases
SNL Najm Uncertain Networks 6 / 37
Intro Dynamics DFI Closure
Deterministic Nonlinear ODE System Analysis
Computational Singular Perturbation (CSP) analysis
Jacobian eigenvalues provide first-order estimates of thetime-scales of system dynamics: τi ∼ 1/λiJacobian right/left eigenvectors provide first-orderestimates of the CSP vectors/covectors that definedecoupled fast/slow subspacesWith chosen thresholds, have M “fast" modes
M algebraic constraints define a slow manifoldFast processes constrain the system to the manifoldSystem evolves with slow processes along the manifold
CSP time-scale-aware Importance indices provide meansfor elimination of “unimportant" network nodes andconnections for a selected observable
SNL Najm Uncertain Networks 7 / 37
Intro Dynamics DFI Closure
Analysis of Uncertain ODE Systems
Handle uncertainties using probability theory
Every random instance of the uncertain inputs provides a“sample" ODE system
– Uncertainties in fast subspace lead to uncertaintyin manifold geometry
– Uncertainties in slow subspace lead to uncertainslow time dynamics
One can analyze/reduce each system realization– Statistics of x(t;λ) trajectories
This can be expensive!
Explore alternate means
SNL Najm Uncertain Networks 8 / 37
Intro Dynamics DFI Closure
Spectral Stochastic Representations
Let (Ω, σ, ρ) be a probability space.Let ξ : Ω → Rm be an L2 RV.Let (Ξ, s, µ) = ξ♯(Ω, σ, ρ).Let {ϕα(ξ) : α = 0, 1, 2, . . .} be an orthonormal basis of L2(Ξ).Let X : A × Ω → R be an L2(Ω) A-process. Its closestrepresentative in L2(Ξ) is
X(a, ω) ≃∑
α
Xα(a)ϕα(ξ(ω))
where
Xα(a) =∫
ΩX(a, ω)ϕα(ξ(ω)) dρ(ω) = 〈ϕα,X〉.
Take m = 1 for simplicity. m > 1 holds by tensor productarguments.
SNL Najm Uncertain Networks 9 / 37
Intro Dynamics DFI Closure
Galerkin Reformulation
Consider an ODE
ẋ = f (ξ, x) x(ξ, 0) = x0(ξ)
with x(t, ω) ∈ Rn. Represent x as
x(ξ, t) =∑
α
xα(t)ϕα(ξ)
wherexα(t) = 〈ϕα(ξ), x(ξ, t)〉
and so these coefficients have dynamics
ẋα =
〈ϕα(ξ),
ddt
x(ξ, t)
〉
= 〈ϕα(ξ), f (ξ, x)〉
SNL Najm Uncertain Networks 10 / 37
Intro Dynamics DFI Closure
Jacobian of Sampled System
The dynamical system can be locally characterized by theeigenstructrure of the Jacobian matrix. The entries of theJacobian matrix J of the sampled system are given by
Jij(ξ, t) =∂f i
∂xj(ξ, x(ξ, t))
At each value of time, J(ξ, t) is a random matrix.
SNL Najm Uncertain Networks 11 / 37
Intro Dynamics DFI Closure
Jacobian Matrix of Reformulated System
The Jacobian matrix of the coefficient system can be thought ofas a block matrix with blocks
Jαβ(t) = Dxβ
∫
Ξf (ξ, x(ξ, t))ϕα(ξ) dµ(ξ)
=
∫
Ξϕα(ξ) J(ξ, t)ϕβ(ξ) dµ(ξ)
= 〈ϕα, Jϕβ〉
Truncate the representation so that α, β = 0, . . . ,P.J is then a n(P + 1)× n(P + 1) matrix.
SNL Najm Uncertain Networks 12 / 37
Intro Dynamics DFI Closure
Dynamical Analysis of the Galerkin PC System
Key questions:
How do the eigenvalues and eigenvectors of the Galerkinsystem relate to those of the sampled original system
What can we learn about the sampled dynamics of theoriginal system from analysis of the Galerkin system
– fast/slow subspaces– slow manifolds
Can CSP analysis of the Galerkin system be used foranalysis and reduction of the original uncertain system
SNL Najm Uncertain Networks 13 / 37
Intro Dynamics DFI Closure
Relevant Prior Work
Ghosh and Ghanem (2002-2005)
Homescu, Petzold, and Serban (Siam Review, 2007)
Tryoen and Le Maître (JCP, JCAM, 2010)
Fisher and Bhattacharya, PC Galerkin system eigenvalues(2008)
– First numerical illustrations that the Galerkinsystem eigenvalues seem to exist in the supportof the eigenvalues of the sampled system
– System with continguous locus of each stochasticeigenvalue
Nevai (1980)
SNL Najm Uncertain Networks 14 / 37
Intro Dynamics DFI Closure
Infinite Jacobian J (P = ∞)
For P = ∞, we can prove that the eigenvalues λi of J(ω, t) arealso eigenvalues of J (t) in the L2-sense.
We can construct vectors wi = {wi,kγ}, k = 1, . . . , n; γ = 0, . . . ,∞,where
Jwi = λiwi, i = 1, . . . , n
with equality in L2, i.e. for each jα,
limP→∞
∥∥∥∥∥∥
n∑
k=1
P∑
γ=0
Jjkαγ(t)wi,kγ(ω, t)− λi(ω, t)wi,jα(ω, t)
∥∥∥∥∥∥L2(Ω)
= 0.
SNL Najm Uncertain Networks 15 / 37
Intro Dynamics DFI Closure
Essential Numerical Range of J
The numerical range of a matrix M is
W(M) = {v∗Mv : v ∈ Cm, v∗v = ‖v‖2 = 1}.
Note thatspect(M) ⊂ W(M).
The essential numerical range of J(ξ) is
W̃(J) =⋃
a.e. ξ
W(J(ξ)).
SNL Najm Uncertain Networks 16 / 37
Intro Dynamics DFI Closure
Eigenpolynomials
Let λiα, viα be an eigenvalue/vector pair of J P:
J viα = λiαviα.
Alternatively,
〈ϕβ(ξ), (J(ξ) − λiα) viα(ξ)〉 = 0 for β = 0 . . . P
where viα(ξ) in an n-vector with components
vkiα(ξ) =
P∑
γ=0
vkγiα ϕγ(ξ).
SNL Najm Uncertain Networks 17 / 37
Intro Dynamics DFI Closure
n-dimensional system – Key Results
1 The spectrum of J P is contained in the convex hull of theessential range of the random matrix J.
spect(J P) ⊂ conv(W̃(J))
2 For any orthonormal basis {ϕα}∞α=0:
As P → ∞, the eigenvalues of J P(t) converge weakly, i.e.in the sense of measures, toward
⋃ω∈Ω spect(J(ω)).
3 The J P eigenvalues and eigenpolynomials can be used toconstruct a polynomial approximation of the PCE for therandom eigenvalues.
– for continuous and separated λi(ξ) in C.
Sonday et al., SISC, in press; Berry et al., in review
SNL Najm Uncertain Networks 18 / 37
Intro Dynamics DFI Closure
1D Example
ẋ(ξ, t) = a(ξ)x(ξ, t); ξ(ω) ∼ U[−1, 1];
J = a(ξ) ≡
{ξ + 1 for ξ ≥ 0,ξ − 1 for ξ < 0.
W̃(J) = [−2,−1] ∪ [1, 2]; conv(W̃(J)) = [−2, 2].
LU PC: eigenvalues of J P shown for P = 10, 15, 20, 25, 45
SNL Najm Uncertain Networks 19 / 37
Intro Dynamics DFI Closure
Eigenpolynomials approximate the PCE of λ(ξ)
SNL Najm Uncertain Networks 20 / 37
Intro Dynamics DFI Closure
Stochastic Vectors composed of Galerkin eigenvectorsapproximate the stochastic eigenvectors well
SNL Najm Uncertain Networks 21 / 37
Intro Dynamics DFI Closure
CO Oxidation Example
The oxidation of CO on a surface can be modeled as(Makeev et al., JCP, 2002)
u̇ = az − cu − 4duv v̇ = 2bz2 − 4duv
ẇ = ez − fw z = 1 − u − v − w
a = 1.6, b = 20.75 + .45ξ, c = 0.04, d = 1.0, e = 0.36, f = 0.016
u(0) = 0.1, v(0) = 0.2,w(0) = 0.7exhibits Hopf bifurcations for b ∈ [20.3, 21.2]
SNL Najm Uncertain Networks 22 / 37
Intro Dynamics DFI Closure
CO Oxidation: PC order 10. Slow eigenvalues.
SNL Najm Uncertain Networks 23 / 37
Intro Dynamics DFI Closure
CO Oxidation: PC order 10. Eigenvectors.
SNL Najm Uncertain Networks 24 / 37
Intro Dynamics DFI Closure
Data Free Inference (DFI)
Input uncertainties are not well characterized in manypractical network modelsMay have nominal parameter values and bounds
No information on correlationsNo joint PDF on parameters
Joint PDF structure can have a drastic effect on resultinguncertainties in predictions
When original raw data is available, Bayesian inferenceprovides the requisite posteriorWhen original data is not available, what can be done?
DFI: discover a consensus joint PDF on the parametersconsistent with given information
(Berry et al., JCP, in review)
Demonstrate on a chemical ignition problem (ODE)
SNL Najm Uncertain Networks 25 / 37
Intro Dynamics DFI Closure
Generate ignition "data" using a detailed model+noise
Ignition using a detailedchemical model formethane-air chemistry
Ignition time versus InitialTemperature
Multiplicative noise errormodel
11 data points:
di = tGRIig,i (1 + σǫi)
ǫ ∼ N(0, 1) 1000 1100 1200 1300Initial Temperature (K)
0.01
0.1
1
Igni
tion
time
(sec
)
GRI
GRI+noise
SNL Najm Uncertain Networks 26 / 37
Intro Dynamics DFI Closure
Fitting with a simple chemical model
Fit a global single-stepirreversible chemicalmodel
CH4 + 2O2 → CO2 + 2H2O
R = [CH4][O2]kfkf = A exp(−E/R
oT)
Infer 3-D parametervector (ln A, ln E, lnσ)
Good mixing withadaptive MCMC whenstart at MLE
28
30
32
34
36
lnA
10.6
10.8
lnE
0 2000 4000 6000 8000 10000Chain Step
-3-2.5
-2-1.5
-1-0.5
lnσ
SNL Najm Uncertain Networks 27 / 37
Intro Dynamics DFI Closure
Bayesian Inference Posterior and Nominal Prediction
30 31 32 33 34 35
10.6
10.65
10.7
10.75
10.8
10.85
1000 1100 1200 1300Initial Temperature (K)
0.01
0.1
1
Igni
tion
time
(sec
)
GRIGRI+noiseFit Model
GRI
GRI+noise
Marginal joint posterior on(ln A, ln E) exhibits strongcorrelation
Nominal fit model is con-sistent with the true model
SNL Najm Uncertain Networks 28 / 37
Intro Dynamics DFI Closure
Marginal Posteriors on ln A and ln E
30 32 34lnA
0
0.2
0.4
0.6
0.8
p(ln
A)
10.6 10.7 10.8 10.9lnE
0
5
10
15
p(ln
E)
ln A = 32.15 ± 3 × 0.61 ln E = 10.73 ± 3 × 0.032
SNL Najm Uncertain Networks 29 / 37
Intro Dynamics DFI Closure
Data Free Inference Challenge
Discarding initial data, reconstruct marginal (ln A, ln E) posteriorusing the following information
Form of fit model
Range of initial temperature
Nominal fit parameter values of ln A and ln E
Marginal 5% and 95% quantiles on ln A and ln E
Further, for now, presume
Multiplicative Gaussian errors
N = 8 data points
SNL Najm Uncertain Networks 30 / 37
Intro Dynamics DFI Closure
DFI Algorithm Structure
Basic idea:
Explore the space of hypothetical data sets
Accept data sets that lead to posteriors that are consistentwith the given information
Evaluate pooled posterior from all acceptable posteriors
Algorithm uses two nested MCMC chains
An outer chain on the data, (2N + 1)–dimensional– N data points (xi, yi) + σ– Likelihood function captures constraints on
parameter nominals+bounds
An inner chain on the model parameters– Likelihood based on fit-model– parameter vector (ln A, ln E, lnσ)
SNL Najm Uncertain Networks 31 / 37
Intro Dynamics DFI Closure
Short sample from outer/data chain
-2.4
-2.2
-2
-1.8
-1.6
lnσ
1000
1100
1200
1300
Initi
al T
emp
(K)
0 200 400 600 800 1000Chain Step
0
0.2
0.4
0.6
0.8
Ign.
tim
e (s
ec)
1000 1100 1200 1300Initial Temperature (K)
0.01
0.1
1
Igni
tion
time
(sec
)
SNL Najm Uncertain Networks 32 / 37
Intro Dynamics DFI Closure
Reference Posterior – based on actual data
31 31.5 32 32.5 33 33.5 10.66
10.68
10.7
10.72
10.74
10.76
10.78
10.8
0 10 20 30 40 50 60 70 80
ln A
ln E
SNL Najm Uncertain Networks 33 / 37
Intro Dynamics DFI Closure
Ref + DFI posterior based on a 1000-long data chain
31 31.5 32 32.5 33 33.5 10.66
10.68
10.7
10.72
10.74
10.76
10.78
10.8
0 10 20 30 40 50 60 70 80
ln A
ln E
SNL Najm Uncertain Networks 34 / 37
Intro Dynamics DFI Closure
Ref + DFI posterior based on a 5000-long data chain
31 31.5 32 32.5 33 33.5 10.66
10.68
10.7
10.72
10.74
10.76
10.78
10.8
0 10 20 30 40 50 60 70 80 90
ln A
ln E
SNL Najm Uncertain Networks 35 / 37
Intro Dynamics DFI Closure
Marginal Pooled DFI Posteriors on ln A and ln E
29 30 31 32 33 34 35lnA
0
0.2
0.4
0.6
0.8
p(ln
A)
ReferencePooled 1kPooled 5k
10.6 10.7 10.8 10.9lnE
0
5
10
15
p(ln
E)
ReferencePooled 1kPooled 5k
SNL Najm Uncertain Networks 36 / 37
Intro Dynamics DFI Closure
Closure
Analysis of uncertain network model dynamics:Outlined relationship between eigen-analysis of a sampledstochastic ODE system and the Galerkin PC system.Galerkin system eigenvalues/eigenvectors can be used toanalyze the dynamics of the stochastic systemWork in progress on
– associated stochastic model reduction strategies– structural uncertainty in network models
Data Free Inference:Developed a DFI procedure for estimation of self-consistentparametric posteriors in the absence of dataDemonstrated effective and convergent estimation ofmissing posterior in a chemical ignition problemIn progress: algorithm optimization and generalization tohandle a range of different constraints
SNL Najm Uncertain Networks 37 / 37
IntroductionDynamical Analysis for Model ReductionData Free InferenceClosure