Data Assimilation: from Meteorology to Space Weather and Applications

B. Khattatov, M. Murphy, M. Gnedin, J. SheffelB. Khattatov, M. Murphy, M. Gnedin, J. SheffelFusion Numerics Inc; Fusion Numerics Inc; V. YudinV. Yudin, National Center for Atmospheric Research , National Center for Atmospheric Research

Tim Fuller-Rowell, Tim Fuller-Rowell, NOAA/SEC – CU/CIRESNOAA/SEC – CU/CIRES

Data Assimilation: from Data Assimilation: from Meteorology to Space Meteorology to Space

Weather and ApplicationsWeather and Applications

Data AssimilationData Assimilation Origins of data assimilation reside in estimation and Origins of data assimilation reside in estimation and

optimal control theories, developed primarily as pure optimal control theories, developed primarily as pure mathematical conceptsmathematical concepts

Arguably, it was the first computationally practical Arguably, it was the first computationally practical estimation method, Kalman filter, that allowed to put men estimation method, Kalman filter, that allowed to put men on the moonon the moon

It became evident in weather forecasting that without a It became evident in weather forecasting that without a practical framework for combining imperfect models and practical framework for combining imperfect models and observations (Kalman filter being one), forecasting observations (Kalman filter being one), forecasting becomes largely uselessbecomes largely useless

Data assimilation methods developed in meteorology Data assimilation methods developed in meteorology has been recently used in other areas of Earth scienceshas been recently used in other areas of Earth sciences

BackgroundBackground One such area is chemical data assimilation, that aims at One such area is chemical data assimilation, that aims at

studying composition, pollution, and emission sourcesstudying composition, pollution, and emission sources Arguing that analogies exist between neutral atmosphere Arguing that analogies exist between neutral atmosphere

composition and ionospheric ion/electron content, we composition and ionospheric ion/electron content, we proposed to the US Air Force to build a practical proposed to the US Air Force to build a practical assimilative ionospheric modelassimilative ionospheric model

Two other similar projects were under way in the US, Two other similar projects were under way in the US, one lead by Utah State University, the other by one lead by Utah State University, the other by JPL/Caltech JPL/Caltech

Our effort has been funded by the US Air Force for a Our effort has been funded by the US Air Force for a number of years by about $2MM and was relatively number of years by about $2MM and was relatively successful successful

Primary Components of Data Primary Components of Data AssimilationAssimilation

A forward modelA forward model An observational operator: An observational operator: HH xx yy Methods of combining a forward (forecasting) model Methods of combining a forward (forecasting) model

state with observations:state with observations: Sequential (e.g. Kalman filter)Sequential (e.g. Kalman filter) Variational (e.g., 4-D Var) Variational (e.g., 4-D Var) EnsembleEnsemble

Methods for statistical validation of the system Methods for statistical validation of the system performance and tuning assimilation parameters (e.g., performance and tuning assimilation parameters (e.g., OmA & OmF analysis, OmA & OmF analysis, 22 analysis) analysis)

Sequential MethodsSequential Methods

Sequential methods aim at constraining the model state Sequential methods aim at constraining the model state with most recent data, usually at each model time stepwith most recent data, usually at each model time step

Once the data has been used, the “memory” of Once the data has been used, the “memory” of incorporated data is contained in the model error incorporated data is contained in the model error covariance matrixcovariance matrix

Both constrained model state and the covariance matrix Both constrained model state and the covariance matrix are evolved forward in time using the forecasting model are evolved forward in time using the forecasting model for the state and some variant of the Kalman filter for the for the state and some variant of the Kalman filter for the covariance matrixcovariance matrix

At the next time step the process repeatsAt the next time step the process repeats

Fusion Numerics IncFusion Numerics Inc

Kalman FilterKalman Filter( )a

t t t x x K y HxT T 1( )t t

K B H HB H O R

T T -1 - ( )at t t t t B B B H HB H O R HB

Δ ( , ) at t tt x M x

T at t t B L B L Qd

dt t

t

x

Lx

In the assimilation mode every 10 minutes model electron densities are corrected with data from GPS reference network.


Receiver Bias EstimationReceiver Bias Estimation A major problem with GPS measurements of A major problem with GPS measurements of

ionospheric electron content is inter -frequency ionospheric electron content is inter -frequency hardware biases.hardware biases.

These biases can be as large or larger than the These biases can be as large or larger than the measurements themselves, resulting, for measurements themselves, resulting, for example, in negative electron content.example, in negative electron content.

The biases change in time in response to The biases change in time in response to changing environmental conditions (T, humidity, changing environmental conditions (T, humidity, etc)etc)

Biases are currently estimated in the same Biases are currently estimated in the same (augmented) Kalman filter that is used to (augmented) Kalman filter that is used to assimilate observations.assimilate observations.


Bias EstimationBias Estimation

ˆ y y b

1

2

3

1

2

3

...

ˆ ...

...

x

x

x

b

b

b

xAugment control state with biases:

ˆ ˆˆ ˆ ˆ( )at t t x x K y HxApply Kalman filter:

Statistical Validation Statistical Validation TechniquesTechniques

Observations-minus-Analysis statistics (OmA, the Observations-minus-Analysis statistics (OmA, the smaller the better, must be Gaussian & unbiased)smaller the better, must be Gaussian & unbiased)

Observations-minus-Forecast analysis (OmF, same, but Observations-minus-Forecast analysis (OmF, same, but will be larger than OmA)will be larger than OmA)

22 analysis – verifies that the implied and the true model analysis – verifies that the implied and the true model error growth rates are approximately the sameerror growth rates are approximately the same

Other application specific techniques….Other application specific techniques….

The developed operational ionospheric modeling and The developed operational ionospheric modeling and assimilation system consists of :assimilation system consists of :

- core ionospheric forecasting model solves plasma - core ionospheric forecasting model solves plasma mass, momentum and energy conservation equations mass, momentum and energy conservation equations on a global 3-D grid.on a global 3-D grid.

- empirical models of electric/magnetic fields, and - empirical models of electric/magnetic fields, and neutral composition and windsneutral composition and winds

- data assimilation component corrects model - data assimilation component corrects model simulation with GPS ground station data via a large-simulation with GPS ground station data via a large-scale suboptimal Kalman filter. scale suboptimal Kalman filter.

- data fetching components, housekeeping modules, - data fetching components, housekeeping modules, time synchronization modules, visualization, etctime synchronization modules, visualization, etc


Forward Ionospheric ModelForward Ionospheric Model New code developed by Fusion NumericsNew code developed by Fusion Numerics Object-oriented, written in Object-oriented, written in C++ C++ ~100,000 lines of code, ~100 classes~100,000 lines of code, ~100 classes Solves plasma momentum, mass, and energy Solves plasma momentum, mass, and energy

conservation equations.conservation equations. OO++, H, H++, He, He++, NO, NO++, N, N++, O, O22

++, N, N22++

Fixed 3-D grid in magnetic coordinates.Fixed 3-D grid in magnetic coordinates. Can use real-time measured solar activity from Can use real-time measured solar activity from

NOAA Space Environment System.NOAA Space Environment System.


A Part of the Model Grid in A Part of the Model Grid in Geographic CoordinatesGeographic Coordinates

High-Resolution Local GridHigh-Resolution Local Grid


Examples of Model FieldsExamples of Model Fields

There are approximately 150 different variables at each of ~1,000,000 model grid points.

IGS Stations Used in the SystemIGS Stations Used in the System

Total Electron Content MapTotal Electron Content Map

High-Resolution Local TECHigh-Resolution Local TEC

ValidationValidation A subset of the available IGS stations is A subset of the available IGS stations is

systematically withheld from the assimilation and systematically withheld from the assimilation and used for validation.used for validation.

The validation stations are rotated at random The validation stations are rotated at random every 24 hours.every 24 hours.

Slant TEC computed in the system is then Slant TEC computed in the system is then compared with slant TEC from the validation compared with slant TEC from the validation stations.stations.

Global average RMS error is ~1-5 TEC unitsGlobal average RMS error is ~1-5 TEC units

Daily Validation StatisticsDaily Validation Statistics


Jicamarca ComparisonsJicamarca Comparisons

A GPS Positioning Engine Relying on the A GPS Positioning Engine Relying on the Assimilation System Ionospheric Assimilation System Ionospheric

SpecificationsSpecifications

Positioning Results With and Positioning Results With and Without WAASWithout WAAS

Forecasting IssuesForecasting Issues In meteorology forecast is primarily an initial In meteorology forecast is primarily an initial

value problemvalue problem

Only under quiet conditions ionosphere can be Only under quiet conditions ionosphere can be reasonably forecasted for hours or even daysreasonably forecasted for hours or even days

Proper forecast must rely on driver forecast, Proper forecast must rely on driver forecast, which is notoriously hard (e.g., geomagnetically which is notoriously hard (e.g., geomagnetically efficient CMEs)efficient CMEs)

Data Latencies IssuesData Latencies Issues In meteorology data can be incorporated in the In meteorology data can be incorporated in the

models at several hour long intervalsmodels at several hour long intervals Ionospheric conditions can change dramatically Ionospheric conditions can change dramatically

on scale of secondson scale of seconds Most IGS data are delivered in near-real time, Most IGS data are delivered in near-real time,

but are freely available to the public with but are freely available to the public with latencies of days to hourslatencies of days to hours

New protocol and software developed by BKG New protocol and software developed by BKG (NTRIP) makes it easier for operators of (NTRIP) makes it easier for operators of individual (e.g., non-IGS) GPS stations to make individual (e.g., non-IGS) GPS stations to make their data available in real timetheir data available in real time

Current StateCurrent State The developed assimilation system is currently The developed assimilation system is currently

operational and is being used by the US Air Force operational and is being used by the US Air Force Research LabResearch Lab

It has also been licensed by the European Space It has also been licensed by the European Space Agency (ESA), used by a prominent Asian government Agency (ESA), used by a prominent Asian government for studies of a GPS augmentation system, used by for studies of a GPS augmentation system, used by several large GPS equipment manufactures for several large GPS equipment manufactures for evaluating ionospheric impact on GPS, and attracted evaluating ionospheric impact on GPS, and attracted interest from government-associated organizations in interest from government-associated organizations in China, Russia, South Korea, and India. China, Russia, South Korea, and India.

Thank you for your attention!Thank you for your attention!

Always looking for new ways and new Always looking for new ways and new partners to use our system for research, partners to use our system for research, educational and commercial purposeseducational and commercial purposes

Contact info: [email protected] Contact info: [email protected]


GPS Receiver Bias EstimationGPS Receiver Bias Estimation

http://www.fusionnumerics.com/ionospherehttp://www.fusionnumerics.com/ionosphere

In the ideal world, what would we need to build an ionospheric now- or In the ideal world, what would we need to build an ionospheric now- or fore-casting system?fore-casting system?

• A model of electrical and magnetic fields : 3-D Ex, Ey, Ez, Bx, A model of electrical and magnetic fields : 3-D Ex, Ey, Ez, Bx, By, Bz in the vicinity of the EarthBy, Bz in the vicinity of the Earth

• A model of photon energy spectrum ranging from gamma rays A model of photon energy spectrum ranging from gamma rays to infraredto infrared

• A model of external 3-D particle precipitation fluxA model of external 3-D particle precipitation flux• A coupled 3-D neutral wind modelA coupled 3-D neutral wind model• A coupled 3-D neutral chemical composition modelA coupled 3-D neutral chemical composition model• A coupled 3-D ionospheric model accounting for composition, A coupled 3-D ionospheric model accounting for composition,

momentum, and energy of the ionosphere on scales of up to 100 momentum, and energy of the ionosphere on scales of up to 100 meters to include Raleigh-Taylor instabilitiesmeters to include Raleigh-Taylor instabilities

• Possibly a wave propagation model (s)Possibly a wave propagation model (s)• Enough observations to constrain all these modelsEnough observations to constrain all these models• Data assimilation modules that can use these observations Data assimilation modules that can use these observations • Lots of talented peopleLots of talented people• Lots of moneyLots of money

Variational TechniquesVariational Techniques

Variational methods aim at finding a model state that Variational methods aim at finding a model state that provides a best fit between the model trajectory and all provides a best fit between the model trajectory and all observations available over a certain time intervalobservations available over a certain time interval

As such, the model trajectory inside that time window As such, the model trajectory inside that time window tends to be smooth and exactly satisfying the model tends to be smooth and exactly satisfying the model equationsequations

Covariance matrices are not explicitly evolved and are Covariance matrices are not explicitly evolved and are usually computed by some other methodsusually computed by some other methods

The time window then “slides” forward in timeThe time window then “slides” forward in time

Sequential vs. Variational Sequential vs. Variational TechniquesTechniques

For linear forward models and observations with For linear forward models and observations with Gaussian error statistics, properly implemented Gaussian error statistics, properly implemented sequential and variational techniques are equivalentsequential and variational techniques are equivalent

For a particular non-linear systems and non-Gaussian For a particular non-linear systems and non-Gaussian statistics, one method or another might be more statistics, one method or another might be more accurateaccurate

How to evaluate performance of a given assimilation How to evaluate performance of a given assimilation strategy?strategy?

Data Assimilation: from Meteorology to Space Weather and Applications

Documents

observations kalman

chemical data assimilation

model time steponce

sheffelfusion numerics

recent data

augmented kalman filter

assimilation mode

kalman filtervariational