Institut für Umweltphysik/Fernerkundung Physik/Elektrotechnik Fachbereich 1 Overview about new processor baseline H. Bovensmann, G. Lichtenberg, R. Snel,

Institut für Umweltphysik/Fernerkundung Physik/ElektrotechnikFachbereich 1

Overview about new processor baselineH. Bovensmann, G. Lichtenberg, R. Snel, C. Lerot, T Fehr

(on behalf of SQWG)ENVIVAL-LIFE Meeting, 6.12.2010, Bremen

Overall Goals SQWG

• The main goal for the SQWG is establishing L1 and L2 high data quality for the operational data products of the full SCIAMACHY data set, which allows substantial contributions to long term atmospheric data sets in the areas of:

» the development of the O3 layer

» global air pollution » Global Greenhouse gas observations» Solar-Terrestrial interactions

• In addition, the data SCIAMACHY data set shall allow to investigate new scientific methods and new products.

SCIAMACHY QWG

• SCIAMACHY Quality Working Group– Inter-Agency Activity: ESA-DLR-NSO– Maintenance and development of the ESA operational

SCIAMACHY processors and tools• Level 1b / Level 2

• SCICAL

• SciaL1C

– Established 2006– Extension 2011-2012 currently prepared

• SQWG Statement of Work currently finalised (2011/12)

• SCIAMACHY ESL contract: Proposal received from DLR (2010-2012)

– Coordinator: Heinrich Bovensmann

SQWG TeamIUP: S. Noel, K. Bramstedt, P. Liebing, M. Buchwitz, C. von Savigny, A.

Richter, A. Rozanov, F. Wittrock, M. Vountas, A. Kokhanovsky, K. U. Eichmann, M. Weber, J.P. Burrows, H. Bovensmann

SRON: R. van Hees, P. van der Meer, J.M. (Matthijs) Krijger, R. Snel, I. Aben

BIRA: C. Lerot, M. van Roozendael

KNMI: L.G. (Gijsbert) Tilstra, P. Stammes

DLR IMF: G. Lichtenberg, A. Doicu, S. Gimeno Garcia, F. Schreier, M. Meringer, D. Sherbakov, S. Slijkhuis, B. Aberle, K. Kretschel, S. Hrechanyy, M. Gottwald

ESA: T. Fehr

SERCO: A. Dehn, G. Brizzi

• Level 1 IPF 7.04 compliant with ENVISAT 2010+ mission extension– No product format changes– No algorithm changes wrt IPF 7.03– CFI 5.8.1 implemented

• Level 2 Off-Line Processor SGP 5.01– Product format substantially changed. – Additional trace gases:

• Nadir columns of SO2, BrO, H2O, OClO and CO and BrO Limb profiles

• Limb Cloud indicators for cloud presence and type

• Improved Absorbing Aerosol Index.

• Level 2 “Fast-Delivery” processing with SGP 5.01 (<24h) at D-PAC based on Level 1b NRT and predicted ADFs

Operational SCIA Processors activated Feb. 2010


Level 1 v7

Level 0-1 processor (IPF L1 v7)

Stray light matrix channel 2 introduced

Writing of limb mesosphere states to limb MDS introduced

On-board error of writing scan encoder values corrected in processor

SciaL1c

Version 2.1 delivered October 2008

New:

M-factor handling

Bug fixes

ASCII output

Build environment improved

Degradation Correction: M-Factor’s• L1b Version 7 m-factors:

– NRT:• Quasi-operational forward-processing V7 running since Feb 2010 at IUP.• Variable limits in m-factor quality checks implemented by ESA• M-factor plots are included in the bi-monthly report

– Reprocessing:

• L1b monitoring data 08/2002 – 01/2010 completely provided by DLR.• M-factors for V7 re-processing are complete and delivered to ESA.

• Documentation, m-factor data base, plots & applicator s/w are also available via m-factor website:

http://www.iup.uni-bremen.de/sciamachy/mfactors

• M-factors for L1b version 6.0x (08/2002 – 01/2010), still available too.


Level 2 v5

Level 1b-2 Version 5 – Product Overview(Blue: New, Green: Improved)

Nadir UV/VIS:O

3, NO

2, BrO (BIRA, SDOAS/GDP4)

SO2 (volcanic & anthropogenic), OClO (IUP, algorithm using

SDOAS/GDP4 formalism),

H2O (IUP, AMC DOAS)

Nadir SWIRCO (IMF, BIRRA)

Nadir Clouds & AAI:Cloud Top Height & Optical Thickness (IUP, SACURA)

Nadir Cloud Fractions (IMF, OCRA)

Absorbing Aerosol Index (KNMI)

Limb:O

3, NO

2, BrO Profiles (IMF: operational version, IUP: retrieval

settings)

Cloud flagging (IUP, SCODA)


Nadir Products

Nadir O3 total columns

• Algorithm Heritage: SDOAS (BIRA)

• Basic algorithm settings as L2 v3.01

• Retrieval settings:

– Fit window: 325 – 335 nm

– Daily calibrated ESM irradiance spectra (D0).

– Radiances fully calibrated.

– M-factors.

– Cloud parameters from OCRA/SACURA.

• Expected quality / known issues:– Small temporal trend (up to

0.5%/year), minimised by applying m-factors at least in the tropics.

Impact of m-factors on total O3 (%)

SCIAMACHY OL 3.01- reference (%)

See also C. Lerot et al., Atmos. Meas. Tech., 2, 87-98, 2009

O3: SCIAMACHY trend m-factor impact

• Trend with respect to GOME1:– systematic trend of -4%/decade

for nearly all latitudes w.r.t. GOME

• This trend does not dependent on algorithm (see Lerot et al., 2009, Loyola et al., 2009)

– m-factor corrections improves situation in the tropics (see also Bramstedt et al., 2009)

– But: m-factors do not completely remove the observed trend in SCIAMACHY total ozone

L1: -all calibrations on -solar reference (ESM diff.) L1 w./m-factor: -all calibrations on, but no etalon

M. Weber

Nadir NO2 total columns

• Algorithm Heritage: SDOAS (BIRA)

• Basic algorithm settings as L2 v3.01


– Fit window: 426.5 – 452.5 nm

– Daily not calibrated ASM irradiance spectra (A0).

– Radiances extracted without radiometric calibration (no m-factors)

Higher fit quality and equivalent NO2 columns

– Cloud parameters from OCRA/SACURA• Expected quality / known issues:

– To be validated …

Nadir BrO total columns

• New implementation

• Algorithm Heritage: SDOAS (BIRA) Slant column retrieval as SGP/SDOAS AMF calculation uses new BrO profile stratospheric climatology

developed by BIRA used (Theys et al. ACP 2009)

• Retrieval settings:– Fit window: 336 – 351 nm

– Daily not calibrated ASM irradiance spectra (A0)


– Cloud parameters from OCRA/SACURA

• Expected quality / known issues:– small difference in VCDs of about 2x1012 molec cm-2 on average

between the reference algorithm and operational processor

– Stratospheric AMF not representative for trop. enhanced BrO

Nadir OClO slant columns• New implementation

• Algorithm Heritage:

– Slant column retrieval as SGP/SDOAS– Settings from IUP




– SZA cut-off = 95°

• Expected quality / known issues:– Small but relatively constant low bias of 3x1013

molec cm-2

– polynomial degree will be adapted to minimise bias

Note: OClO only present in chlorine activated polar vortex at SZA > 90°

Nadir SO2 total column



• Slant column retrieval as SGP/SDOAS• Retrieval and AMF settings from IUP

• Retrieval settings:• Fit window: 315 – 327 nm

• Daily not calibrated ASM irradiance spectra (A0)

• Radiances extracted without radiometric calibration (no m-factors)

• SCD post-processing step: reference sector subtraction• daily subtraction of reference sector SCD (Pacific) to

compensate for SZA and ozone sensitivity of retrieval etc.

• SCD is converted to VCD using simulated AMF’s for • “anthropogenic pollution”

• “volcanic eruption” scenarios

SO2: Volcanic Columns

• Globally, there are several artefacts:

• Strongly negative values at high latitudes

• Sea – land contrast• Very large values in

winter high latitudes• Some scattered high

values• Artefacts in IUP Bremen

product have similar pattern but much smaller values

=> Product under revision

SAA

SAA

Nadir H2O total column

Precision estimate (w.r.t. ECMWF)• 3 - 5% for columns >0.6 g/cm2 • small columns: up to 10%• Absolute precision always better than 0.2 g/cm2



– Air Mass Corrected (AMC)-DOAS (Noël et al., 2007)



– Radiances calibration: memora effect, leakage, wavelength (no m-factors)

• Key product entries: – Vertical column density (VCD), error in VCD, air

mass factor (AMF) correction

• Expected quality / known issues:– Average bias -0.2 g/cm2 w.r.t. ECMWF

Nadir CO column

• New implementation• Algorithm Heritage:

– BIRRA (direct non-lin. least square fit to radiance, Schreier et al. 2009)


– Fit window: 2324 – 2335 nm (channel 8)

• Expected quality / known issues:• highly sensitive to DBPM• Over sea signal too low, numerical effects can play a role• Most likely use case: Means over many orbits• Mean values will depend on filtering of data• Current version: not to be used• Algorithm and settings under revision


Limb Products

Limb Changes

• The following changes were made to the Limb retrievals

– New Product Limb cloud flags

– New Product BrO Profile

– Limb Retrievals recognise the cloud height from cloud flag product, i.e. the retrieval starts at the first cloudfree height found above the minimum height

• Note: small trend in pointing knowledge over time may affect trends in all limb products – under continues investigation

V3.01 V4

Limb O3 Profile

• Optimised Retrieval

• Algorithm Heritage: Optimal Estimation (IMF, Doicu et al.) Optimisation of retrieval settings IUP

• Retrieval settings: Spectral range 520 – 590 nm Fine altitude grid (33 layers instead 24) 4th order polynomial Picard iteration – Limb radiance ratio w.r.t. 46 km TH– LOWTRAN aerosol model (background conditions) used in RTM– Retrieval starts above clouds using cloud parameters from SCODA

• Expected quality / known issues:– Bias as obseved in v3.01 O3 should be much smaller

– Clouds have impact on O3 around tropopause

• Validation ongoing

Limb O3 improvements vs. LIDAR

L2 v4.4 L2 v5L2 v3.01

Comparison: Anne Gijsel, RIVM via ESA VALID / EQUAL project

Limb NO2 Profile

• Optimised Retrieval

• Algorithm Heritage: Optimal Estimation (Doicu et al. Optimisation of retrieval settings IUP

• Retrieval settings: Optimised settings

Fine altitude grid (33 layers instead 24) Picard Iteration Extended spectral range 420 – 470 nm Reduced regularization parameter

– Limb radiance ratio w.r.t. 43 km TH

– Cloud parameters from SCODA

– Retrieval starts above clouds

• Expected quality / known issues:– To be validated

Limb BrO Profile

• New Retrieval

• Algorithm Heritage: Optimal Estimation (Doicu et al. Optimisation of retrieval settings IUP

• Retrieval settings:– Spectral range 337 – 357 nm

– Limb radiance ratio w.r.t. 35 km TH

– Cloud parameters from SCODA

– Retrieval starts above clouds

• Expected quality / known issues: deviations on average up to 40% were found above with a large standard

deviation For not too large SZA‘s and in certain latitude bands on

average difference < 20%– Some „data gaps“ – under investigation

Limb Cloud Flagging• New Product

• Algorithm Heritage: SCODA using colour index ratios to identify

strong scattering layers in limb (IUP, Eichmann et al.)

• New Limb MDS introduced:• FLAG: 0 = no cloud 1 = patially

cloudy2 = fully cloudy 3 = bad

data• MAXVAL: max. color index ratio,• MAXHEIGHT: tangent height (in km) where

MAXVAL is found• MAXHEIGHTIDX: index of MAXHEIGHT• each for three different cloud types

– WCL: normal water clouds,– ICL: ice clouds– PSC: polar stratospheric clouds.

• Expected quality / known issues:– To be validated


Summary/Outlook

From Level 2 V 3.01 to V 5.01Product Improvements

Level 2 V 3.01 V5.01

2007 2010

Nadir Products

Absorbing Aerosol Index

quality not sufficient improved algorithm and usage of degradation corrections

Ozone total column

slight trend over time (< 0.5% Per year),

GDP 4

degradation correction taken into account;

smaller trend expected

NO2 offset removed reference spectra

BrO VCD

SO2 VCD, volcanic and pollution

OClO SCD

H2O VCD

CO / xCO VCD xCO quality tbc

Cloud parameter (Cloud fraction, Cloud Top Height)

OCRA/SACURA improvements due to degradation correction;

new minimum reflectance data base improved OCRA CF

From Level 2 V 3.01 to V 5.01 Product Improvements

Level 2 V 3.01 V5.01

2007 2010

Limb Products

Ozone profile TH offset removed, maximum of 4 limb

O3 profiles per tangent height

improved forward model, optimised

retrieval settings, => substantially smaller low bias; clouds and

aerosol improvements lower stratosphere

NO2 profile TH offset removed improved forward model, optimised

retrieval settings => improved lower strat.;

clouds taken into account

BrO profile product newly implemented

Limb cloud flagging

product implemented, verfied and used in O3, NO2 and BrO retrieval

Preliminary Validation• SCIAVALIG Meeting 07/08 September 2010, BIRA

– In general good agreement of the SCIAMACHY products and available validation data

• Ozone Limb profiles are clearly improved

• New products are considered of good or sufficient good quality

• SO2 slant and anthropogenic columns show product deficits.

• CO column retrievals need an optimisation

– SCIAMACHY Level 2 reprocessing interrupted– Improved baseline delivery expected February 2011

• OClO, CO and SO2 improvements

– ESA/SQWG decision to postpone reprocessing re-start until availability of corrected baseline

– SCIAMACHY delta-Validation Report made available:http://envisat.esa.int/earth/www/object/index.cfm?fobjectid=7279

Level 2 Version 5.0x (1. Quarter 2011)

• Nadir OClO: bug-fix in degree of polynomial will reduce bias in SC

• Nadir SO2: bug-fix in reference sector data base routine

• Nadir CO: bug-fix in

• Limb O3: impact of aerosol a-priori under investigation

• Limb BrO gaps under investigation

Level 1 version IPF version 8 (2011/12)

Work in progress ...

• Changes (Verified/ To Be Verified)– Memory Effect Limb improved– Non-Linearity PMD scaling noise check introduced– Hot Pixel Check Limb improved– Polarisation for occultation switched off– SAA check for etalon switched off– Selection of individual dark states– Stray light matrix channels 3-8– Key-data and scan angle dependent degradation

Next Baseline: Level 1b-2 Version 6 (2011/12)

New in version 6 (Verified)• Porting to 64bit + current versions of libraries and compilers• AAIA improvements (use Ozone columns from retrieval)• Nadir Snow/Ice separation• VCD HCHO• VCD CHOCHO• Trop. NO2 (limb-nadir)• Improved CO• XCH4 or XCO2 (tbc)• NLC detection Limb• Ozone Profile extension to lower Mesosphere

ESA Atmospheric Mission2nd half 2013

mid-2015 (to be confirmed)

2018 ?

2017 ?

12/2014

04/2013

2018

2020

12/2013 ?

06-07/2011 !

Outlook: ESA‘s „atmospheric“ missions

Figure courtesy T. Fehr, ESA

We need to lobby for a mission extension beyond 2014 !

Coming soon: The SCIAMACHY „Biography“


The End

Institut für Umweltphysik/Fernerkundung Physik/Elektrotechnik Fachbereich 1 Overview about new processor baseline H. Bovensmann, G. Lichtenberg, R. Snel,

Documents

data sciamachy data

iup retrieval settings

slant column retrieval

sgpsdoas retrieval

bremen slide

brizzi slide

heinrich bovensmann

new products