Level 2 Ocean Salinity L2OS v610 & OTT post-processor

SMOS L2 Ocean Salinity

www.argans.co.uk

Level 2 Ocean Salinity

L2OS v610 & OTT post-processor

5 July 2013

ARGANS & SMOS L2OS ESL

www.argans.co.uk

Significant changes from v600

• OTT post-processor• OTT running average tracking L1 drift, updated daily in DPGS & reprocessing

• no more compareTBs!

• TEC from Stokes 3• impact on high TEC descending orbits; ignore L1c IRI TEC

• Multi-threading• > 2 x speed increase

• Simplified slim-line UDP; extended DAP• remove unnecessary flags & fields, make it easy for users to extract good quality data

L2OS v610 status

www.argans.co.uk

Real-time salinity retrieval requires near real-time OTT to track short term drift: while processing L1c,

L2OS v610 extracts deltaTBs for South Pacific region and writes them to AUX_DTBXY_. The OTT post-

processor runs once per day, reads AUX_DTBXY_ and a set of current deltaTBs (from AUX_DTBCUR),

writes a new set of OTTs, and updates AUX_DTBCUR. The L2OS processor always uses the latest set of

deltaTBs(DTBXY_)

current deltaTBs

(DTBCUR)

OTT post-

processor

OTTsL2OS

processorL1c

L2OSUDP & DAP

OTT post-processor

By-products:

• daily drift statistics available in AUX_DTBCUR.HDR for analysis & monitoring

• other (non-OTT) regions may be specified (in AUX_CNFOSF) for monitoring (output in DTBXY_)

www.argans.co.uk

• L2OS v610for each region in AUX_CNFOSF

apply region snapshot filter // use snapshot if enough measurements & all within region

if OTT region

apply OTT snapshot filter // ignore if TBs out-of-range, RFI, L1c error

apply OTT grid point filter // not ocean, hi/lo WS, ice

if count(snapshots) > Min_Snapshots & count(gridpoints) > Min_Grid_Points

if OTT region apply OTT measurement filter // ignore sun_point, rfi_tails

deltaTB[region] = use forward model to compute deltaTBs & statistics

write deltaTB & statistics to AUX_DTBXY_

• OTT post-processormerge all AUX_DTBXY_ from previous day with current AUX_DTBCUR

discard old sets of deltaTBs (last-in/last-out)

compute & write new OTT using current set of deltaTBs

write current set of deltaTBs & statistics to new AUX_DTBCUR

OTT post-processor overview

www.argans.co.uk

for each selected measurement

map xi,eta onto x,y grid

count[x,y] += 1

sumDeltaTB[x,y] += TBsmos – Tbmodel

sumDeltaRA[x,y] += (TBsmos – Tbmodel) / ra

sumDeltaRA2[x,y] += ((TBsmos – Tbmodel) / ra)2

flags[x,y] |= selected L1c & L2OS flags

for x = 0 to 128

for y = 0 to 128

n = count[x,y]

write n

mean_deltaTB[x,y] = sumDeltaTB[x,y] / n

write mean_deltaTB[x,y]

stdra[x,y] = sqrt(((n * sumDeltaRA2[x,y] – sumDeltaRA[x,y]2) / (n * (n – 1)))

write stdra[x,y]

write flags[x,y]

AUX_DTBXYcount[x,y]deltaTB[x,y]stdra[x,y]flags[x,y]

Statistics for fov, affov, eaffov, & filtered by flags

Flags for off-line analysis:

OTT post-processor: L2OS v610 algorithm

www.argans.co.uk

deltaTB referencemax_orbits (nominally 10)

quality thresholds (RFI, L1c errors)

ott_strategy (mean/gaussian)

ott_validity_start (first/mean/last)

List of regionsName, IDtype (OTT/REG)orbit dir (A/D/?)lat/longsnapshot thresholdsgrid point thresholds

AUX_CNFOSF <OTTPP> section

Source of geophysical reference parameters for computing deltaTBs:

0 = climatology, L1c TEC & ECMWF priors

1 = model 1 retrieved parameters

And new filters:

OTT_SNAPSHOT_FILTER

OTT_REGION_FILTER

OTT_STATS_FILTER

New OTT post-processor section in AUX_CNFOSF

www.argans.co.uk

n x regions:IDsnapshot start/stop timessnapshot start/stop ID

for 3 modelsfor 8 polarisations

4 x statisticscount[x,y]deltaTB[x,y]stdra[x,y]flags[x,y]

AUX_DTBXY_ datablock

Statistics for FOV, AFFOV, EAFFOV, & filtered by flags[x,y]: mean, median, min, max, std.

OTT post-processor components: AUX_DTBXY_

www.argans.co.uk

read all deltaTB orbits from AUX_DTBCUR // ignore

ottA/D

for each AUX_DTBXY_

if region = South Pacific OTT region & good quality

read deltaTB for this orbit

// keep n ascending & n descending deltaTBs...

trim deltaTB // discard oldest first

ottA = average(deltaTB, ascending)

ottD = average(deltaTB, descending)

for model = 1 to 3

write ottA(model) & ottD(model) to AUX_OTT1/2/3

deltaTB += ottA

deltaTB += ottD

write deltaTB to new AUX_DTBCUR // including ottA/D

Options:

number of deltaTB orbits (nominally 10+10)

mean or weighted mean (nominally mean)

filter DTBXY by quality

OTT post-processor: OSCOTT algorithm

www.argans.co.uk

Header contains quality information for each region:

Grid point quality informationtotal grid points in region, number of grid points used after filteringnumber of grid points:

too near to land/coast; rain/ice contaminationmissing ECMWF data; low WS (<3m/s), high WS (>7m/s)

Measurement quality informationtotal measurements in region; number of measurements used after filteringnumber of measurements flagged:

as sun point, sun tails, with sun/moon/galactic glintcontaminated by RFI (by L1 or L2), L1c error flags

Quality information is from forward model 1 only:

used by OSCOTT to select ‘good’ sets of deltaTBs.

Snapshot quality informationtotal snapshots in region, number of snapshots used after filteringnumber of XX/YY/XXY/YYX snapshotsnumber of snapshots with each of 4 L1c error flags setnumber of snapshots with TB out-of-range, high std/std_Stokes3/4

www.argans.co.uk

Header contains quality information for each region:

Grid point quality informationtotal grid points in region, number of grid points used after filteringnumber of grid points:

too near to land/coast; rain/ice contaminationmissing ECMWF data; low WS (<3m/s), high WS (>7m/s)

Measurement quality informationtotal measurements in region; number of measurements used after filteringnumber of measurements flagged:

as sun point, sun tails, with sun/moon/galactic glintcontaminated by RFI (by L1 or L2), L1c error flags

Quality information is from forward model 1 only:

used by OSCOTT to select ‘good’ sets of deltaTBs.

Snapshot quality informationtotal snapshots in region, number of snapshots used after filteringnumber of XX/YY/XXY/YYX snapshotsnumber of snapshots with each of 4 L1c error flags setnumber of snapshots with TB out-of-range, high std/std_Stokes3/4

www.argans.co.uk

AUX_DTBXY_ datablock

m x AUX_DTBXY + extra sets:set type (L1c, OTT...)region IDorbit direction (A/D)orbit polarisation (D/F)snapshot start/stop timessnapshot start/stop IDL1c filename

AUX_DTBCUR datablock

OSCOTT

m x AUX_DTBXYn x regions:

IDsnapshot start/stop timessnapshot start/stop ID

OTT post-processor components: AUX_DTBCUR

www.argans.co.uk

Header contains quality information statistics for ascending & descending OTTs generated by

the last OTT post-processor run:

Ascending OTT quality informationcount of orbits used to make OTTsnapshot start/stop timesfor 3 polarisations

for 8 polarisations4 x statistics (same format as AUX_DTBXY_ statistics(count, mean, median, std for fov, affov, eaffov, & filtered)

Descending OTT quality information(same fields as above)

List of L1c datasets

www.argans.co.uk

Header contains quality information statistics for ascending & descending OTTs generated by

the last OTT post-processor run:

Ascending OTT quality informationcount of orbits used to make OTTsnapshot start/stop timesfor 3 polarisations

for 8 polarisations4 x statistics (same format as AUX_DTBXY_ statistics(count, mean, median, std for fov, affov, eaffov, & filtered)

Descending OTT quality information(same fields as above)

List of L1c datasets

www.argans.co.uk

OTT post-processor preliminary results

AUX_DTBXY_ x 10

AUX_OTT1F_

deltaTB, XX pol, ascending

www.argans.co.uk

AUX_DTBXY_ x 10

AUX_OTT1F_

Median = 0.58 K

Histograms of deltaTB in XX FOV, ascending orbits, AUX_DTBCUR 20110501

www.argans.co.uk

AUX_DTBXY_ x 10

AUX_OTT1F_

std/ra, XX pol, ascending

www.argans.co.uk

AUX_DTBXY_ x 10

AUX_OTT1F_

Histograms of std/ra in XX FOV, ascending orbits, AUX_DTBCUR 20110501

www.argans.co.uk

Comparing OTT from DPGS with OTTPP: re-sampled grid

DPGS OTT (20110504) OSCOTT (20110501) delta = OSCOTT – DPGS

Differences along border-belt-suspenders & tails

www.argans.co.uk

• Which statistics should be used to select/reject DTBXY deltaTB data?• Current implementation has thresholds for maximum % measurements contaminated

by L1 RFI, L2 RFI, or with L1c error flags set

OTT post-processor issues: good AUX_DTBXY

www.argans.co.uk

• What is the optimal running average window?• Averaging many sets of deltaTBs gives better quality OTTs but with a slower response to

rapid TB drift

OTT post-processor issues: averaging window

www.argans.co.uk

OTT post-processor issues: priors or retrievals?

std(priors) std(model 1 retrievals)

deltaTB(priors-model1)

deltaTBs are computed as the difference

between forward model TBs and L1c TBs

Should we run the forward model using

priors (SSS from climatology, SST/WS

from ECMWF, TEC from L1c or A3TEC)?

www.argans.co.uk

OTT post-processor issues: xi/eta grid sampling

-1.0 xi 1.0

-0.7 xi 0.7

v600 v610

v600: 129x129 (xi ±1.0, eta ±1.0): sampling interval 0.016

v610: 129x129 (xi ±0.7, eta -0.7 to 0.4): sampling interval 0.011 in xi, 0.008 in eta

What grid sampling interval is best for OTTs?

Grid point density varies within snapshot – highest at leading edge of AFFOV

Typical total 5600 grid points, >100 along leading edge (±0.25 xi): ~0.005 sampling?

Blackman pixels ~0.04 diameter: with pixel overlap 1.4, sampling interval 0.028

Small sampling interval leads to higher noise in OTTs.

www.argans.co.uk

OTT post-processor issues: merge OTTs?

long integration time short integration time delta

OTTs are similar – should they be merged?

www.argans.co.uk

long integration time short integration time

OTTs are similar – should they be merged to reduce noise

(especially for short integration time)?

www.argans.co.uk

XXY YYX delta

OTTs are similar – should they be merged?

www.argans.co.uk

L1PP v600 20110505T040402

latitude

L1cTEC & A3TEC

Cross-verification between Matlab breadboard & v610: delta = breadboard-v610

TEC from Stokes 3: descending orbit, high TEC

www.argans.co.uk

L1PP v600 20110505T040402

latitude

L1cTEC & A3TEC

Tuning:

decreased sigma prior, increased gaussian smoothing, added outlier filtering, TEC height changed from 450 to 400 km

www.argans.co.uk

L1PP v600 20110505T040402

latitude

L1cTEC & A3TEC

L2OS retrieved TEC follows A3TEC better than L1c TEC

www.argans.co.uk

L1OP v601 20110505T040448

latitude

L1cTEC & A3TEC

L1OP v601 using consolidated VTEC

www.argans.co.uk

Ascending orbit with low TEC – noisy A3 TEC retrieval, but good match with L1c

L1PP v600 20100802T153815

L1cTEC & A3TEC

TEC from Stokes 3: ascending orbit

latitude

www.argans.co.uk

L1cTEC & A3TEC

L1PP v600 20100802T153815

latitude

Tuning:

decreased sigma prior, increased gaussian smoothing, added outlier filtering, TEC height changed from 450 to 400 km

www.argans.co.uk

L1cTEC & A3TEC

L1PP v600 20100802T153815

latitude

L2OS retrieved TEC follows A3TEC better than L1c TEC?

www.argans.co.uk

L1cTEC & A3TEC

L1OP v601 20100802T153855

L1OP v601 using consolidated VTEC

latitude

www.argans.co.uk

Faraday rotation depends on TEC altitude (not only on TEC amplitude).

Tomography using Stokes 3:first analysis shows mean altitude of electronic layer is about 390 kmelectron density not located at same altitude along the orbittomography depends on TEC prior

TEC from Stokes 3: altitude of TEC

Estimated

www.argans.co.uk

Changing TEC altitude in v610 from 450 to 400km

TEC from Stokes 3: altitude of TEC

www.argans.co.uk

Motivation

• run OTT post-processor per-day during reprocessing (same as DPGS)• average L2OS run time in DPGS/reprocessing is 2.5 – 3 hours (TBC)

• x3 speed-up with 4 threads would match SM performance (Antonio de la Fuente)

• 15% faster CPU clock in reprocessing platform

• L2OS maximum run time per half-orbit is ~370 minutes (Pacific Ocean)(on 5 year old original specification server)

• L2OS v610 uses OpenMP to multi-thread time-consuming loops• Pacific Ocean test orbit results (includes 25 minutes I/O) :

3 threads: 160 minutes (5.3G) – 2.3 x speed-up

4 threads: 144 minutes (5.1G) – 2.6 x speed-up

• average run-time (20110416) with 4 threads 124 minutes (max 166, min 77)

Multi-threading L2OS

www.argans.co.uk

Motivation

• UDP includes too many flags & fields – confusing• make it easy for users to extract good quality data from UDP

• move unnecessary UDP flags & fields to DAP

• UDP is for oceanographers, not for analysis

• make the DAP easier to use• variable length grid point records in v600 makes reading DAP grid point data very slow

• restructure DAP into a block of all grid point data (fixed size records), followed by a block of all measurement data (variable size records)

• users/ESL can read just the grid point data & ignore measurements (measurement data can still be read slowly if needed)

Summary

• new UDP only contains geophysical outputs from L2OS processor

• simplified UDP quality index & quality flags for each model

• no compatibility with existing UDP/DAP readers• requires new versions of SDV, GMT, Beam toolbox & bespoke tools

Simplified UDP & extended DAP

www.argans.co.uk

Improving the UDP

n x grid points:

Grid point dataID, latitude, longitude, footprint, time x_swath, distance to coast

Science flags (4)near landsuspect rain, ice (Acard), RFI

Geophysical retrievals (2 x 7)

SSS1, SSS2, SSS3, AcardWS, SST, TEC,

sigmas on all retrieved parameters

Product confidencequality index for SSS1/2/3/Acard

Product confidence flags (4 x 3)

High/medium/low for SSS1/2/3/Acard

RFI probabilitycounters for DGGRFI

v600 (176 flags & 57 fields) v610 (16 flags & 29 fields)

n x grid points:

Grid point dataID, latitude, longitude, footprint, time

Science flags (4 x 30)land/sea/coast, rainice (4 flags: climatology, ECMWF, TBs,

Acard)high/low wind/SSS/SST, sea state

Geophysical priorsWS & SST from ECMWF

Geophysical retrievalsSSS1/2/3/Acard & sigmas

TB42.5 (8)modelled BTs & sigmas at surface & antenna

Control flags (4 x 28)range, sigma, chi, chi2, sun/moon/gal glintmax iterations, low/min #measurements,too many outliers, convergence limit, poor retrieval, poor geophysical, RFI suspect, RFI prone

Product confidence (34)

chi, chi2, #iterations, quality for SSS1/2/3/Acardvalid, border, affov, tails, sun/moon/gal glint, iceRFI probability counters for DGGRFIx_swath

www.argans.co.uk

Fg_high_quality

High quality flag is set if Fg_ctrl_poor_retrieval = 0 & Fg_ctrl_poor_geophysical = 0 & Dg_af_fov > 130 (grid points with lots of measurements in the AFFOV). Users filtering with this flag will get the best quality, suitable for cal-val.

Fg_medium_quality

Medium quality flag is set if Fg_ctrl_poor_retrieval = 0 & Fg_ctrl_poor_geophysical = 0. Users filtering with this flag will get all the high quality retrievals, plus those with Dg_af_fov <= 130. This set is suitable for oceanography, , since we know there were no retrieval or geophysical problems.

Fg_low_quality

Low quality flag is set if Fg_ctrl_poor_retrieval = 0. Users filtering with this flag will get all the high & medium quality retrievals, plus those without any retrieval problems (but there may be geophysical problems). This is the lowest quality we would recommend to users: these retrievals have passed the chi2P/max iterations/convergence tests, so may be ok.

Proposed UDP v610 quality flags (per model)

www.argans.co.uk

n x grid points:

ID, lat/longpriors & sigmas (4 x 14)

posts & sigmas (4 x 14)

oor flags (4 x 27)

fields (8)

m x measurements:

flags (4 x 30)

fieldsdeltaTB (x4)

TBgal (x2)

n x grid points:

ID, lat/longpriors & sigmas (12)

posts & sigmas (24)

science flags (4 x 30) control flags (4 x 28)

oor flags (4 x 27)

fields (45)

n x grid points:

IDm x measurements:

snapshot IDxi, etaflags (4 x 30)

fieldsdeltaTB (x4)

TBgal (x2)

Fixed block size grid pointsVariable

block size grid points

Variable block size

measurements

Improving DAP readabilityv600 v610

Extra fields & flags from

v600 UDP

v600 priors, posts & sigmas (112 fields) allow for 7 parameters per model – only 5+4+3+2 parameters retrieved: 50% unused.

For a large DAP (eg South Pacific) with typically 150k grid points & 120 measurements/grid point:

v600 = 360M (70M grid points + 290M measurements)

v610 = 400M (50M grid points + 350M measurements)

www.argans.co.uk

L1OP v601 & L2OS v610

32 ascending & 25 descending L1OP v601 orbits processed using L2OS v610, deltaTBs extracted from AUX_DTBXY_

Orbits don’t intersect OTT region, so deltaTB statistics extracted for South Pacific test region 122 (30S to 0, 160W to 120W)

Some correlation, but:climatology for 122OTT shifted (late)

v601 reduces long term drift?

www.argans.co.uk

L1OP v601 & L2OS v610: RFI test orbit 20100709T034234

L1OP v504 REPR L1OP v601Fm_L1c_RFI set mostly by Fm_RFI_XX & Fm_RFI_YY

www.argans.co.uk

L1OP v601 & L2OS v610: RFI test orbit 20100709T034234

L1OP v504 REPR L1OP v601(using L2OS v600 L1OP v504 REPR OTTs)

For RFI (& other) tests we need to ensure OTTs match L1c...

www.argans.co.uk

L1OP v504 & v601: RFI test orbit 20100802T153855

L1OP v504 REPR L1OP v601Region ID 126 - Equatorial Ocean ±10: small islands?

>1K delta in mean/median

www.argans.co.uk

L1OP v504 REPR L1OP v601 delta = v504 – v601

v601 has larger spatial bias than v504 – why?

www.argans.co.uk

L1OP v504 REPR L1OP v601Region ID 126 - Equatorial Ocean ±10: RFI @ 5N

www.argans.co.uk

L1OP v504 REPR L1OP v601 delta = v504 – v601

www.argans.co.uk

L1OP v504 REPR L1OP v601Improved Stokes 3 & 4

www.argans.co.uk

Verification scheduled August/September

• requires at least 1 month reprocessed L1c to verify (May 2011)• OTT post-processor – run at ARGANS

• TEC from Stokes 3 – run on G-POD?

• multi-threading

Remaining work before delivery

• Simplified slim-line UDP; extended DAPneed specification of weighted distance-to-coast (TBC)

• New galactic map LUTs?

Schedule• delivery mid-September, FAT early October?

L2OS v610 status summary

Level 2 Ocean Salinity L2OS v610 & OTT post-processor

Documents

An Overview of the L2OS ESL WorkPlan NR & ESLs. A document.....

Report on Salinity Assessment and Salinity Management Plan

KODAK EASYSHARE V610

About L2OS v6 improvement wrt L2OS v5 N. Martin – J.L....

Www.argans.co.uk SMOS L2 Ocean Salinity Level 2 Ocean...

Www.argans.co.uk SMOS L2 Ocean Salinity Level 2 Ocean...

Salinity Sensor

Outlier detection in L2OSPP - usefulness in RFI detection-.....

Rhizobium Salinity

Www.argans.co.uk SMOS L2 Ocean Salinity Level 2 Ocean...

Level 2 Ocean Salinity L1 -> L2OS tools

Www.argans.co.uk SMOS L2 Ocean Salinity L2OS QWG #4: RFI 7-9...

Salinity Indicator

Mapping Soil Salinity Using Soil Salinity Samples and...

Salinity BoundaryConditions

IBM Worklight V6.1.0 Getting Startedpublic.dhe.ibm.com ›.....