Visualising the Input Space of a Galaxy Formation Simulation · 2012-04-16 · UQ12 minitutorial - session 6 2 / 68 • In many scientiﬁc disciplines complex computer simulators

Visualising the Input Space of aGalaxy Formation Simulation

UQ12 Minitutorial

Presented by: Tony O’Hagan,

Peter Challenor, Ian Vernon

Overview

UQ12 minitutorial - session 6 2 / 68

• In many scientific disciplines complex computer simulators are employed tohelp understand corresponding real world physical processes (e.g climatemodels are used to analyse climate).

Overview



• These simulators, referred to as Computer Models, share manyattributes and also many problems.

Overview




• Often they take a long time to run, and require the specification of a largenumber of input parameters x.

Overview





• An area of Statistics has arisen to deal with such models, and is mainlycentred around the construction of ‘Emulators’: fast stochasticapproximations to the Computer Model.

Overview





• An area of Statistics has arisen to deal with such models, and is mainlycentred around the construction of ‘Emulators’: fast stochasticapproximations to the Computer Model.

• Often the most pressing question is: are there any inputs that giveacceptable matches between the model output and observed data?

Overview


• Going to History Match a Galaxy formation simulation known asGalform.

Overview



• This involves learning about acceptable inputs x to the Galform model,using observed data z.

Overview




• We use emulators and implausibility measures to cut out input spaceiteratively.

Overview





• We will discuss relevant uncertainties: model discrepancy, observationalerrors, function uncertainty etc.

Overview






• Finally, we will consider various visualisation issues: even when we canidentify the acceptable input space, visualisation is difficult.

Overview






• Finally, we will consider various visualisation issues: even when we canidentify the acceptable input space, visualisation is difficult.

• The approach described is completely general, and can be used for anymodel that is relatively slow to run and requires lots of inputs.

Why History Match?


• History Matching is an efficient technique that seeks to identify the set Xof all acceptable inputs x.

Why History Match?



• Often X only occupies a tiny fraction of the original input space.

Why History Match?




• This set X may be empty: we do not presuppose that any such inputsexist.

Why History Match?





• This is the main difference between History Matching and the relatedtechnique of Probabilistic Calibration.

Why History Match?





• This is the main difference between History Matching and the relatedtechnique of Probabilistic Calibration.

• The later is a useful technique, but assumes a single ‘best input’ and givesits posterior distribution.

Andromeda Galaxy and Hubble Deep Field View


• Andromeda Galaxy: closest large galaxy to our own milky way, contains1 trillion stars.

• Hubble Deep Field: one of the furthest images yet taken. Covers 2millionths of the sky but contains over 3000 galaxies.

The Galform Model


• The Cosmologists at the ICC are interested in modelling galaxy formationin the presence of Dark Matter.

The Galform Model



• First a Dark Matter simulation is performed over a volume of (1.63 billionlight years)3. This takes 3 months on a supercomputer.

The Galform Model




• Galform takes the results of this simulation, includes the more realisticphysics of ‘normal matter’, and models the evolution and attributes ofapproximately 1 million galaxies.

The Galform Model





• Galform requires the specification of 17 unknown inputs in order to run.

The Galform Model






• It takes approximately 1 day to complete 1 run (using a single processor).

The Galform Model






• It takes approximately 1 day to complete 1 run (using a single processor).

• The Galform model produces lots of outputs, some of which can becompared to observed data from the real Universe.

Galform: Which Inputs to Use?


• PROBLEM: We want to identify the set of all inputs X that lead toacceptable matches between model outputs and observed data,given all relevant uncertainties.




• 17-dimensional input space is large! If we did the simplest grid basedsearch (setting each input to max or min), we would require 217 runs.





• This would take approximately 360 years to complete (on one processor)!






• We would really want a higher definition, so would want say 1017 runs...






• We would really want a higher definition, so would want say 1017 runs...This would take far longer than the current age of the Universe.







• SOLUTION: Construct an Emulator, which is a stochastic function thatapproximates the Galform model, and is fast to evaluate.







• SOLUTION: Construct an Emulator, which is a stochastic function thatapproximates the Galform model, and is fast to evaluate.

• Use the Emulator to find the acceptable inputs.

The Dark Matter Simulation


The Galform Model


Galform Outputs: The Luminosity Functions


• Galform provides multiple output data sets.

• Initially we analyse the luminosity functions which give the number ofgalaxies per unit volume, for each luminosity.

• Bj Luminosity: corresponds to density of young (blue) galaxies• K Luminosity: corresponds to density of old (red) galaxies

Input Parameters


• To perform one run, we need to specify numbers for each of the following17 inputs:

vhotdisk: 100 - 550 VCUT: 20 - 50aReheat: 0.2 - 1.2 ZCUT: 6 - 9alphacool: 0.2 - 1.2 alphastar: -3.2 - -0.3vhotburst: 100 - 550 tau0mrg: 0.8 - 2.7epsilonStar: 0.001 - 0.1 fellip: 0.1 - 0.35stabledisk: 0.65 - 0.95 fburst: 0.01 - 0.15alphahot: 2 - 3.7 FSMBH: 0.001 - 0.01yield: 0.02 - 0.05 eSMBH: 0.004 - 0.05tdisk: 0 - 1

• What input values should we choose to get ‘acceptable’ outputs?



• Basic problem is that we pick inputs:• vhotdisk = 290.5, aReheat = 1.15, alphacool = 0.31, ...

• And find that after 1 Day of Runtime:




• And find that after 1 Day of Runtime:• 1st run is rubbish.




• And find that after 2 Days of Runtime:




• And find that after 2 Days of Runtime:• 2nd run is rubbish.




• And find that after 3 Days of Runtime:




• And find that after 3 Days of Runtime:• 3rd run is rubbish.



• Pick 20 inputs and find after 20 Days of Runtime:• All runs are rubbish.







11 Outputs Chosen


• We do 1000 runs using carefully chosen inputs (a space-filling maximinlatin hypercube design).

• (Again all runs are found to be unacceptable.)

11 Outputs Chosen


• We do 1000 runs using carefully chosen inputs (a space-filling maximinlatin hypercube design).

• (Again all runs are found to be unacceptable.)• We choose 11 outputs that are representative of the Luminosity functions

and emulate the functions fi(x).

Linking Model to Reality


• We represent the model (Galform) as a function, which maps the vector of17 inputs x to the vector of 11 outputs f(x).




• We use the “Best Input Approach” to link the model f(x) to the realsystem y (i.e. the real Universe) via:

y = f(x+) + d

where we define d to be the model discrepancy and assume that d isindependent of f and x+.





y = f(x+) + d


• Finally, we relate the true system y to the observational data z by,

z = y + e

where e represent the observational errors.





y = f(x+) + d


• Finally, we relate the true system y to the observational data z by,

z = y + e

where e represent the observational errors.

• We will use the Bayes Linear methodology, which only involvesexpectations, variances and covariances.

Galform: Emulation


• For each of the 11 outputs we pick active variables xA then emulateunivariately (at first) using:

fi(x) =∑

j

βij gij(xA) + ui(x

A) + δi(x)

Galform: Emulation



fi(x) =∑

j

βij gij(xA) + ui(x

A) + δi(x)

• The∑

j βij gij(xA) is a 3rd order polynomial in the active inputs.

Galform: Emulation



fi(x) =∑

j

βij gij(xA) + ui(x

A) + δi(x)

• The∑


• ui(xA) is a Gaussian process.

Galform: Emulation



fi(x) =∑

j

βij gij(xA) + ui(x

A) + δi(x)

• The∑



• The nugget δi(x) models the effects of inactive variables as random noise.

Galform: Emulation



fi(x) =∑

j

βij gij(xA) + ui(x

A) + δi(x)

• The∑




• The ui(xA) have covariance structure given by:

Cov(ui(xA1 ), ui(x

A2 )) = σ2

i exp[−|xA1 − xA

2 |2/θ2

i ]

Galform: Emulation



fi(x) =∑

j

βij gij(xA) + ui(x

A) + δi(x)

• The∑




• The ui(xA) have covariance structure given by:

Cov(ui(xA1 ), ui(x

A2 )) = σ2

i exp[−|xA1 − xA

2 |2/θ2

i ]

• The Emulators give the expectation E[fi(x)] and variance Var[fi(x)] atpoint x for each output given by i = 1, .., 11, and are fast to evaluate.

Emulation: a 1D Example
















Model Discrepancy


Before calculating the implausibility we need to assess the Model Discrepancyand Measurement error:

Model Discrepancy Var[d] = Φ40 + Φ9 + ΦE

Model Discrepancy


Before calculating the implausibility we need to assess the Model Discrepancyand Measurement error:

Model Discrepancy Var[d] = Φ40 + Φ9 + ΦE

• Φ40: Discrepancy term due to choosing first 40 sub-volumes from full 512sub-volumes. Assess this by repeating 100 runs but now choosing 40random regions. More advanced: exchangeable models paper.

• Φ9: As we have initially neglected 9 parameters (due to expert advice) weneed to assess effect of this (by running latin hypercube design across all17 parameters)

• ΦE : Expert assessment of model discrepancy of full model with 17parameters and using 512 sub-volumes

It is straightforward to find the multivariate expressions for Φ40 and Φ9, butΦE requires more careful thought.

Model Discrepancy: Subjective ΦE


• Experts assert that there are clear ways that the model could be defective.

• Model predicts too many (or too few) galaxies. This would lead to ahighly correlated model discrepancy across all outputs.

• Model systematically gets the colours of galaxies wrong: results in too few(too many) blue galaxies and too many (too few) red galaxies. Givesnegatively correlated model discrepancy between outputs from differentcoloured (bj and K) luminosity graphs.

• We therefore assume the model discrepancy term ΦE has the form:

ΦE = a

1 b .. c .. cb 1 .. c . c: : : : : :c .. c 1 b ..c .. c b 1 ..: : : : : :

• Obtain values for a, b and c from expert assessment.

Expert Assessment of ΦE: Elicitation Tool


• We obtain expert assessments of a, b and c using an elicitation tool.

Observational Errors


Observational Errors Var[e] are composed of 4 parts:

• Normalisation Error: correlated vertical error on all luminosity outputpoints

• Luminostiy Zero Point Error: correlated horizontal error on all luminositypoints

• k + e Correction Error: Outputs have to be corrected for the fact thatgalaxies are moving away from us at different speeds (light is red-shifted),and for the fact that galaxies are seen in the past (as light takes millionsof years to reach us)

• Galaxy Production Error: assumed Poisson process to describe galaxyproduction

The multivariate form for each of these quantities is straightforward(!) tocalculate.

Implausibility Measures (Univariate)


We can now calculate the Implausibility I(i)(x) at any input parameter pointx for each of the i = 1, .., 11 outputs. This is given by:

I2(i)(x) =

|E[fi(x)] − zi|2

(Var[fi(x)] + Var[di] + Var[ei])




I2(i)(x) =

|E[fi(x)] − zi|2


• E[fi(x)] and Var[fi(x)] are the emulator expectation and variance.




I2(i)(x) =

|E[fi(x)] − zi|2



• zi are the observed data and Var[di] and Var[ei] are the (univariate)Model Discrepancy and Observational Error variances.




I2(i)(x) =

|E[fi(x)] − zi|2




• Large values of I(i)(x) imply that we are highly unlikely to obtainacceptable matches between model output and observed data atinput x.




I2(i)(x) =

|E[fi(x)] − zi|2




• Large values of I(i)(x) imply that we are highly unlikely to obtainacceptable matches between model output and observed data atinput x.

• Small values of I(i)(x) do not imply that x is good!



• We can combine the univariate implausibilities across the 11 outputs bymaximizing over outputs:

IM (x) = maxi I(i)(x)





• We can then impose a cutoff IM (x) < cM in order to discard regionsof input parameter space that we now deem to be implausible.






• The choice of cutoff cM is often motivated by Pukelsheim’s 3-sigma rule.






• The choice of cutoff cM is often motivated by Pukelsheim’s 3-sigma rule.

• We may simultaneously employ other choices of implausibility measure:e.g. multivariate, second maximum etc.

Multivariate Implausibility Measure


• As we have constructed a multivariate model discrepancy, we can define amultivariate Implausibility measure:

I2(x) = (E[f(x)] − z)T Var[f(x) − z]−1(E[f(x)] − z),

which becomes:

I2(x) = (E[f(x)]− z)T (Var[f(x)] + Var[d] + Var[e])−1(E[f(x)] − z)

• where Var[f(x)], Var[d] and Var[e] are now the multivariate emulatorvariance, multivariate model discrepancy and multivariate observationalerrors respectively (all 11×11 matrices).

• We now have two implausibility measures IM (x) and I(x) that we can useto reduce the input space.

• We impose suitable cutoffs on each measure to define a smaller set ofnon-implausible inputs.

History Matching via Implausibility: a 1D Example












Iterative Refocussing Strategy for Reducing Input Space.


We use an iterative strategy to reduce the input parameter space. Denotingthe current non-implausible volume by Xj , at each stage or wave we:




1. Design a set of runs over the non-implausible input region Xj





2. Construct new emulators for f(x) only over this region Xj






3. Evaluate the new implausibility function IM (x) over Xj







4. Define a new (reduced) non-implausible region Xj+1, by IM (x) < cM ,which should satisfy X ⊂ Xj+1 ⊂ Xj .







4. Define a new (reduced) non-implausible region Xj+1, by IM (x) < cM ,which should satisfy X ⊂ Xj+1 ⊂ Xj .

This algorithm is continued until a) we run out of computational resources, orb) the emulators are found to be of sufficient accuracy compared to the otheruncertainties present (model discrepancy and observational errors).

History Matching via Implausibility: 1D Example - Wave 1




History Matching via Implausibility: 1D Example


History Matching via Implausibility: 1D Example




Galform: Visualizing the non-implausible volumes Xj


• Plotting the non-implausible region Xj in 1 dimension is trivial, but howdo we view a 17-dimensional Xj?




• Even though our emulators are very fast to evaluate, we still cannot coverthe 17-dimensional space with a simple grid design.





• We instead use efficient emulator designs, developed specifically forconstructing lower dimensional projections, e.g.






• 2-Dimensional Projection: for each pair of active variables we evaluatethe emulator on a large (2D grid) x (15D latin hypercube) design.







• Very fast emulator design as we can take several algebraic shortcuts due tosymmetries of emulator update: approximately 5 times faster.







• Very fast emulator design as we can take several algebraic shortcuts due tosymmetries of emulator update: approximately 5 times faster.

• Using this, we can now produce 2D Minimised ImplausibilityProjections and Optical Depth Plots.

2D Minimised Implausibility Projections: Wave 1


• Minimised Implausibility Projections: at each 2D grid point, minimisethe implausibility IM (x) over the 15D hypercube.




• If a point on these plots is implausible (coloured red), then it will beimplausible for any choice of the 15 other inputs.




• If a point on these plots is implausible (coloured red), then it will beimplausible for any choice of the 15 other inputs.

• If a point is green, it may or may not prove to be an acceptable input.

2D Min Implausibility Projections: Wave 1 to 4 (0.12%)
















2D Optical Depth Plots: Wave 2


• Optical Depth Plots: at each 2D grid point plot the proportion of the15D latin hypercube points that survive the cutoff IM (x) < cM .




• These plots show the ‘depth’ of the non-implausible volume Xj for wave j,at each grid point.




• These plots show the ‘depth’ of the non-implausible volume Xj for wave j,at each grid point.

• Shows where the majority of non-implausible points can be found, but notnecessarily where the best matches are.

2D Optical Depth Plots: Wave 1 to Wave 4








2D Implausibility Projections: Stage 4 (0.12%)


Why Does Iterative Refocussing Work?


Why do we reduce space in waves? Why not attempt to do it all at once?



Why do we reduce space in waves? Why not attempt to do it all at once?Because this requires an accurate emulator valid over whole input space.

• In contrast, the iterative approach is far more efficient.





• At each wave the emulators are found to be significantly more accurate (inthat Var[f(x)] becomes smaller). This is expected as:

1. We have ‘zoomed in’ on a smaller part of the function, it will besmoother and most likely easier to fit with low order polynomials.







2. We have a much higher density of runs in the new volume, and hencethe Gaussian process part of the emulator will do more work.








3. We can identify more active variables, leading to more detailedpolynomial and Gaussian process parts of the emulator, as previouslydominant variables are now somewhat suppressed.








3. We can identify more active variables, leading to more detailedpolynomial and Gaussian process parts of the emulator, as previouslydominant variables are now somewhat suppressed.

• This is a major strength of the History Matching approach.

Visualisation: Fast Approximate Emulators


The emulation approach we have taken, that of building substantial structurein the polynomial part of the emulator, allows the construction of fastapproximations to the emulators.

• If we take the regression part of the wave 4 emulator only:

fi(x) =∑

j

βij gij(xA) + wi,

and assume Var[wi] = α2(σ2ui

+ σ2δi

) for some conservative choice ofα > 1, we can then use this fast approximate emulator to screen allcandidate input points.

• Using this method we were able to reduce the input space to 1.2% of itsoriginal volume.

• We then only had to use the full, slower wave 1 to 4 emulators on thesurviving points. Used to generate higher dimensional figures.

3D Minimised Implausibility and Optical Depth Plots


• 3D projections created using the Fast Approximate Emulator approach.

4-Dimensional Implausibility Plots: Anyone?


4-Dimensional Implausibility Plots: Anyone?


2D Implausibility Projections: Stage 4 (0.12%)


Wave 5 runs


bj Luminosity Output of Waves 1,2,3 and 5
















Conclusions and Further Issues


• History Matching using iterative refocussing: very efficient technique forlearning about the set of acceptable inputs X .




• Often more appropriate than a fully probabilistic calibration, or should atthe very least precede calibration.





• We have developed novel methods to visualise X that exploit the structureof the emulators.





• We have developed novel methods to visualise X that exploit the structureof the emulators.

• We now have a large set of acceptable (Wave 5) runs that can be analysedby the Cosmologists, and used to explore other features of Galform.



• Future work: beginning to explore the more advanced Galform 2 model:more galaxies, longer run time, far more outputs to match anduncertainties to assess.




Vernon, I., Goldstein, M., and Bower, R. (2010), “Galaxy Formation: a

Bayesian Uncertainty Analysis”, Bayesian Analysis, 5(4): 619–670. Inviteddiscussion paper. MUCM Technical Report 10/03.






Bower, R., Vernon, I., Goldstein, M., et al. (2010), “The Parameter Space of

Galaxy Formation”, Mon.Not.Roy.Astron.Soc., 407: 2017–2045. MUCMTechnical Report 10/02.






Bower, R., Vernon, I., Goldstein, M., et al. (2010), “The Parameter Space of

Galaxy Formation”, Mon.Not.Roy.Astron.Soc., 407: 2017–2045. MUCMTechnical Report 10/02.

— History Matching now available on the MUCM toolkit —

References


P.S. Craig, M. Goldstein, A.H. Seheult, J.A. Smith (1997). Pressure matchingfor hydocarbon reservoirs: a case study in the use of Bayes linear strategiesfor large computer experiments (with discussion), in Case Studies in BayesianStatistics, vol. III, eds. C. Gastonis et al. 37-93. Springer-Verlag.

M. Goldstein and J.C.Rougier (2008). Reified Bayesian modelling andinference for physical systems (with discussion), JSPI, to appear, .

Kennedy, M.C. and O’Hagan, A. (2001). Bayesian calibration of computermodels (with discussion). Journal of the Royal Statistical Society, B,63,425-464

Santner, T., Williams, B. and Notz, W. (2003). The Design and Analysis ofComputer Experiments. Springer Verlag: New York.

Bower, R.G., Benson, A. J. et.al.(2006). The Broken hierarchy of galaxyformation, Mon.Not.Roy.Astron.Soc. 370, 645-655

Visualising the Input Space of a Galaxy Formation Simulation · 2012-04-16 · UQ12 minitutorial - session 6 2 / 68 • In many scientiﬁc disciplines complex computer simulators

Documents