NC STATE UNIVERSITY Prediction of Optical Scale Turbulence with a Typical NWP Code: Lessons Learned (?) Projected to Petascale Computing D. Scott McRae.

NC STATE UNIVERSITY

Prediction of Optical Scale Turbulence with a Prediction of Optical Scale Turbulence with a Typical NWP Code: Lessons Learned (?) Typical NWP Code: Lessons Learned (?)

Projected to Petascale ComputingProjected to Petascale Computing

D. Scott McRae

Aerospace Engineering

North Carolina State University

NCAR Theme of the Year Workshop

May 6, 2008

NC STATE UNIVERSITYAcknowledgmentsAcknowledgments

• Prior funding: – The US Army Research Laboratory, Battlefield Environments

Division, WSMR and the HELJTO monitored by Dr. David Tofsted, ARLWSMR

• Current funding:– The US Air Force Research Laboratory, Space Vehicles

Directorate, Hanscom AFB, MA through contract

FA8718-04-C-0019; monitored by Dr. George Jumper, AFRL/VSBYA

– NorthWest Research Associates, CORA division, monitored by Dr. Joe Werne

• Contributors – Xudong Xiao, H. A. Hassan, and Yih-Pin Liew, NCSU; Talat Odman, GIT; and Frank Ruggiero,AFRL

NC STATE UNIVERSITYOutlineOutline

• Current goal– Prediction of Optical and Clear Air Turbulence

by modifying existing NWP codes to increase prediction accuracy

• Approach• Examples of current work• Projection to Petascale• Analysis of solution• Numerical Issues• Concluding Remarks

NC STATE UNIVERSITYWhat is optical turbulence?What is optical turbulence?

• The term “Atmospheric Optical Turbulence” refers to fluctuations in the refractive index of air due to turbulence in the atmosphere.– Affects optical propagation by random refraction– Reduces the effective power of optical signals

• Quantitative measure of the intensity of atmospheric optical turbulence: structure parameter of refractive index, Cn

2 ,

• Integration of an accurate prediction of Cn2 along the

beam/viewing path is the primary need for many optical systems

• Accurate prediction requires well resolved dynamics and physics, whether from observation or from atmospheric models. In the latter case, physically accurate turbulence models are required for the scales unresolved by the atmospheric model

NC STATE UNIVERSITYObservation versus Clear 1 ModelObservation versus Clear 1 Model

Jumper, Beland,

2000

NC STATE UNIVERSITYApproachApproach

• Prediction usually required for sub-mesoscale domains– Radiosonde soundings

– Numerical weather prediction codes with parameterizations for

– DNS simulation– Statistical techniques using many sources

• Present approach- modify existing NWP codes to increase accuracy of prediction– LES scale Prediction Using Dynamic 3-D Adaptive Grid– Hybrid LES/RANS Turbulence Model with direct Output-

described by Hassan in a later talk

• Resolution of shear requires adaptation in all three coordinate directions

• Accuracy of prediction can be increased by including more physics of turbulence in the model

2TC

2nC

NC STATE UNIVERSITYModel ModificationsModel Modifications

• NCSU r-refinement Dynamic Solution Adaptive Grid Algorithm (DSAGA) – Resolve selected features/characteristics/properties

dynamically– Criteria selected initially for resolution– Code determines location and resolution automatically– Adapts in all three dimensions

• The NCSU k- hybrid turbulence model

– Four equations based on exact equations derived from the Navier-Stokes and modeled term by term- described by Hassan in a following talk

NC STATE UNIVERSITYResults – 2D caseResults – 2D case

x

z

-50000 0 50000 100000

5000

10000

15000

20000

25000

220 km

2 km

Geometry of the computational domainGeometry of the computational domain

22

2

ax

haz

Mesh size:

221X126

Same setup as in Ref. 14 (by Doyle etc.)

NC STATE UNIVERSITYResults – 2D caseResults – 2D case

u , m/s

z,m

10 20 30 400

5000

10000

15000

20000

25000

Inflow wind speed profile from the Grand Junction, CO, sounding for 1200 UTC 11 January 1972

NC STATE UNIVERSITYResults – 2D case(MILES)Results – 2D case(MILES)

Adaptive mesh at t=3h

x (m)

z(m

)

-20000 0 20000 40000

0

10000

20000

NC STATE UNIVERSITYResults – 2D(MILES)Results – 2D(MILES)

Potential temperature contoursPotential temperature contours

NC STATE UNIVERSITYNumerical Lidar (x= -2km)Numerical Lidar (x= -2km)

time (sec)

z(m

)

5000 10000

4000

6000

8000

10000

12000

14000

16000

18000

20000

22000

24000

w1411852

-1-4-7-10-13-16

NC STATE UNIVERSITYVelocity vectors- 3hoursVelocity vectors- 3hours

x

z

0 20000 40000

0

5000

10000

15000

20000

25000

NC STATE UNIVERSITYDetail- Velocity vectors- 3 hoursDetail- Velocity vectors- 3 hours

x

z

30000 35000 40000 45000 50000

10000

15000

NC STATE UNIVERSITY““Isolated “ vorticesIsolated “ vortices

NC STATE UNIVERSITYVelocity vector animationVelocity vector animation

NC STATE UNIVERSITYCnCn22 contours and balloon trajectory contours and balloon trajectory

0y200000

00

XY

Z

1.30E-165.50E-172.32E-179.82E-184.15E-181.75E-187.41E-193.13E-191.32E-195.60E-202.37E-201.00E-20

NC STATE UNIVERSITYComparison of zonal wind profileComparison of zonal wind profile

u (m/s)

z(m

)

-25 -20 -15 -10 -5 0 5 10 15 20 25

5000

10000

15000

20000

25000ObsStandard MM5 (Ruggiero)Adaptive

NC STATE UNIVERSITYComparison of zonal wind profileComparison of zonal wind profile

u (m/s)

z(m

)

-25 -20 -15 -10 -5 0 5 10 15 20 25

5000

10000

15000

20000

25000ObsUniformAdaptive

NC STATE UNIVERSITYComparison of meridional wind profileComparison of meridional wind profile

v (m/s)

z(m

)

-35 -30 -25 -20 -15 -10 -5 0 5 10

5000

10000

15000

20000

25000ObsStandard MM5 (Ruggiero)Adaptive

NC STATE UNIVERSITYComparison of meridional wind profileComparison of meridional wind profile

v (m/s)

z(m

)

-35 -30 -25 -20 -15 -10 -5 0 5 10

5000

10000

15000

20000

25000ObsUniformAdaptive

NC STATE UNIVERSITYComparison ofComparison of

Cn2 (m-2/3)

z(m

)

10-20 10-19 10-18 10-17 10-16 10-15

5000

10000

15000

20000

25000

ObsUniformAdaptive

2nC

1W V

NC STATE UNIVERSITYComparison of vertical spacingComparison of vertical spacing

z (m)

z(m

)

0 200 400 600 800 1000

5000

10000

15000

20000

25000

UniformAdaptive

NC STATE UNIVERSITYWeight Function Along TrajectoryWeight Function Along Trajectory

Normailzed Weight Function

z(m

)

0 0.2 0.4 0.6 0.8 1

5000

10000

15000

20000

25000

Snapshot of the Adaptive Grid: Tennessee Valley Ozone

SimulationGrid adapting to density of NOx plumes

From Dabberdt W. F. et al., “Meteorological Research Needs for Improved Air Quality Forecasting” Bulletin of the American Meteorological Society, vol. 85, no.4, pp. 563-586, April 2004.

Superior O3 Predictions with Adaptive Grid

Sumner Co., TN

Graves Co., KY

0.0

40.0

80.0

120.0

160.0

1 13 25 37 49 61 73

Time starting from 7/14/1995 (hour)

O3 C

on

cen

trat

ion

(p

pb

)

Observation 4-km Static 8-km Static Adaptive

0.0

40.0

80.0

120.0

160.0

200.0

1 13 25 37 49 61 73

Time Starting from 7/11/1995 (hour)

O3 C

on

cen

trat

ion

(p

pb

)

Observation 4-km Static 8-km Static Adaptive

NC STATE UNIVERSITYScaling Adaptive MM5 to FranklinScaling Adaptive MM5 to Franklin

Processors

Z Grid points

Vertical Spacing

Min Time Step

Wall clock

Memory

Number of points 10^6

Np N1 N2 N3 1 2 3 N z z dt3 Days GBi n

Baseline 1 109 163 121 45 15 5 80 0.26 6.67 8.82 1.9 4.2E+004 X 4 X 1 16 109 163 121 45 15 5 80 0.26 6.67 0.83 0.12 4.2E+00

16 X 16 X 1 256 109 163 121 45 15 5 80 0.26 6.67 0.05 0.01 4.2E+00

4 X 4 X 1 16 436 652 484 11 3.8 1.3 80 0.26 1.67 35.28 1.9 6.8E+01

16 X 16 X 1 256 1744 2608 1936 2.8 0.9 0.3 80 0.26 0.42 141.13 1.9 1.1E+03

4 X 4 X 4 64 436 652 484 11 3.8 1.3 320 0.065 1.67 35.28 1.9 2.7E+028 X 8 X 8 512 872 1304 968 5.6 1.9 0.6 640 0.033 0.83 70.57 1.9 2.2E+0316 X 16 X

16 4096 1744 2608 1936 2.8 0.9 0.3 1280 0.016 0.42 141.13 1.9 1.7E+04

8 X 8 X 8 512 872 1304 968 5.6 1.9 0.6 640 0.033 0.83 45.01 1.21 9.9E+0216 X 16 X

16 4096 1744 2608 1936 2.8 0.9 0.3 1280 0.016 0.42 64.47 0.87 6.4E+0316 X 16 X

16 x 4 16384 1744 2608 1936 2.8 0.9 0.3 20480 0.001 0.42 162.07 2.14 7.8E+04

Nest Grid points Nest Spacings

Scaling to higher resolution

Scaling to higher resolution for all directions, larger in nest 3.

Scaling to higher resolution for all directions

NC STATE UNIVERSITYAdaptation plus PetascaleAdaptation plus Petascale

• Adaptive meshing will move specific resolutions upwards on the previous chart

• Dynamic adaptation may provide a more efficient alternative to standard nesting– Targeted solution dependent weight function determines

automatically where resolution is needed rather than a predetermined nest structure

– Initial budget mapped uniformly unto processors– No mesh boundary errors (however, adaptation is not without

error)

• Unfortunately, present NWP codes are unlikely to scale to full use of a petascale resource due to a combination of factors

NC STATE UNIVERSITYEvaluation of MM5 ResultsEvaluation of MM5 Results

• Filtering/dissipation in code tends to reduce or eliminate structure/frequencies needed for prediction

• Solution does not converge as mesh is refined- S. Koch, NOAA

• Approximately 3-1 reduction in vertical spacing improves structure but still appears over damped. Benefit due to local mesh refinement difficult to assess– LES resolution of turbulence scales not yet achieved

(Terra Incognita – Wyngaard)• WRF-ARW shares basic MM5 approach with

updated algorithms- has 8 identifiable dissipation sources

2nC

NC STATE UNIVERSITYNumerics- MM5Numerics- MM5

• Horizontal integration scheme- time centered explicit (leapfrog)– Neutrally stable for all , for central space– MM5 filtering (Asselin, 1972) for any variable α:

Where for all conditions

– Staggered grid with averaging

1CFL

)2(ˆ 1tt1ttt 1.0

)(O 2

NC STATE UNIVERSITYNumericsNumerics– Vertical coordinate

– Where

– The turbulence model provides an alternative Eddy viscosity

NC STATE UNIVERSITYNumericsNumerics

• Vertical integration scheme- semi-implicit (Klemp and Wilhelmson, 1978)– horizontal results held constant,

With divergence damping added

NC STATE UNIVERSITYAssessment of Damping Assessment of Damping

• prediction derived from local state and spatial variation- spatial filtering is then the issue

• Asselin filter used to stabilize Leapfrog in MM5

Where for all conditionsThis can be expressed as

Assuming linear advection

2nC

)2(ˆ 1tt1ttt 1.0

42

22tt tO

ttˆ

42

22tt tO

xCFLˆ

NC STATE UNIVERSITYAssessment of Damping Assessment of Damping

Which becomes

This implies that spatial damping due to the Asselin filter remains constant relative to the mesh if and CFL remain constant as the mesh spacing is reduced

This filter contributes to mathematical non-convergence

24t1ii1i

2tt x,tO2CFLˆ

NC STATE UNIVERSITYAssessment of Damping Assessment of Damping

Divergence Damping (Skamarock and Klemp, 1992)– Essentially a modification of the normal stress term in the

diagonal of the stress tensor (“Normal Stress Damping”, McRae, 1976)

– Simplified analysis as in Durran – x momentum

where

Discretizing the first term gives

y

v

x

u

xF

x

P

t

uxu

xu

xx

tx

001.02

x

21ii1i xO2001.0

NC STATE UNIVERSITYAssessment of Damping Assessment of Damping

This is a spatial damping that remains constant relative to the mesh as the mesh spacing is reduced

This damping contributes to mathematical non-convergence

Furthermore, divergence damping-– Inserts a modified normal stress into the Euler equations

with an unscaled coefficient of the order of an eddy viscosity

– Interacts with turbulence models/parameterizations resulting in non-physical shear layers

– Biases the pseudo- incompressibility formulation

NC STATE UNIVERSITYEvaluation of MM5 ResultsEvaluation of MM5 Results

• Assessing sensitivity of convergence to individual dissipations time consuming and problematical

• Needed- A technique for assessing the net effect of all of the dissipations applied in the course of the integration– A posteriori ( forensic ) analysis of the solution

• The mesh is a band pass filter and determines the number of Fourier terms available for constructing the solution

x2

2k

L2

2

NC STATE UNIVERSITYTotal Assessment of DampinfgTotal Assessment of Dampinfg

Use Fourier analysis of the solution to ascertain, approximately, how much of the spectrum theoretically resolved by the mesh remains in the solution (ignoring aliasing)

– Sample the solution in a direction in which resolution is important

– Use discrete FFT to obtain approximate frequency versus amplitude distribution

– Infer overall filtration

?kL2

2

NC STATE UNIVERSITYSamplingSampling

x

z

-10000 0 10000 20000

0

5000

10000

15000

20000

25000

30000

NC STATE UNIVERSITYSampled FunctionSampled Function

u

z

0 10 20 30 40 50

5000

10000

15000

20000

NC STATE UNIVERSITYFFT OutputFFT Output

1/

u

-0.02 -0.01 0 0.01 0.02-2

0

2

4

6

8

10

12

x=25mx=50m

NC STATE UNIVERSITY

• Three coordinate dynamic grid adaptation in conjunction with physically based turbulence modeling results in demonstrated improvement in optical turbulence prediction

• However, damping and filtering in present NWP codes limit the improvement

• Some damping types, as used, contribute directly to mathematical non-convergence and may lead to non-physical shear layers, thereby damaging further the optical turbulence prediction

Concluding RemarksConcluding Remarks

NC STATE UNIVERSITY

• Achieving full advantage from use of Petascale computing resources will likely require major changes in the current NWP code equations, algorithms and their implementation

Concluding RemarksConcluding Remarks

NC STATE UNIVERSITY

• We are grateful for the many helpful conversations with the people of AFRL/VSBYA, ARL/WSMR, NWRA, DRI, NCSU MEAS, NCAR and others

Concluding RemarksConcluding Remarks