Page 1
The HYPERNETS project is funded by the European Union’s Horizon 2020 research and innovation programme under grant agreement No 775983.
This communication represents only the authors’ views. The European Union is not liable for any use that may be made of the information contained therein.
(c) HYPERNETS Consortium, 2018 (RBINS, TARTU, SU, CNR, NPL, GFZ, CONICET)
NPL - Commercial
Hyperspectral in-situ surface
reflectances from HYPERNETS
Pieter De Vis,
Sam Hunt, Clemence Goyens, Morven Sinclair,
Sarah Taylor, Agnieszka Bialek, Chris Maclellan
& HYPERNETS team
Page 2
(c) HYPERNETS Consortium, 2018 (RBINS, TARTU, SU, CNR, NPL, GFZ, CONICET)(c) HYPERNETS Consortium, 2018 (RBINS, TARTU, SU, CNR, NPL, GFZ, CONICET)(c) HYPERNETS Consortium, 2018 (RBINS, TARTU, SU, CNR, NPL, GFZ, CONICET)NPL - Commercial
Content
• HYPERNETS introduction
• Hypernets_processor + example data
• Uncertainty budget + application
• Conclusions
Page 3
(c) HYPERNETS Consortium, 2018 (RBINS, TARTU, SU, CNR, NPL, GFZ, CONICET)(c) HYPERNETS Consortium, 2018 (RBINS, TARTU, SU, CNR, NPL, GFZ, CONICET)(c) HYPERNETS Consortium, 2018 (RBINS, TARTU, SU, CNR, NPL, GFZ, CONICET)NPL - Commercial
Calibrationmonitoring LED
Radiometer
Pan & Tilt
HYPERNETSmulti-head
hyperspectralRadiometer
HYPERNETSAutonomous
System
810
1
1
2
1
HYPERNETS water
validation network
phase 1
HYPERNETS land
validation network
phase 11
Validation of surface reflectance at all spectral bands of all optical missions inc.Sentinel-2A&B, Sentinel-3A&B, MODIS-A&T, VIIRS, Landsat-8, Pléiades-2A&B,
PROBA-V, CHRIS, ENMAP, PRISMA, SABIAMAR, etc. ... + nanosats
The Idea
Design a new
“low cost” hyperspectral radiometer
for use in
federated networks of water and land sites
Measure reference surface reflectances for multi-mission satellite validation
Page 4
(c) HYPERNETS Consortium, 2018 (RBINS, TARTU, SU, CNR, NPL, GFZ, CONICET)(c) HYPERNETS Consortium, 2018 (RBINS, TARTU, SU, CNR, NPL, GFZ, CONICET)(c) HYPERNETS Consortium, 2018 (RBINS, TARTU, SU, CNR, NPL, GFZ, CONICET)NPL - Commercial
[MERIS 3rd reprocessing data validation report, ACRI, 2012]
Data courtesy of PIs (D. McKee, K. Ruddick, D. Siegel, S. Kratzer) and AERONET-OC PIs (G.
Zibordi, G. Schuster, S. Kratzer, B. Gibson), matchup using MERMAID
In situ
The Motivation for automated hyperspectral
Water reflectance 490nm
Sa
telli
te (
ME
RIS
)
AUTOMATED Data acquisition HYPERSPECTRAL Instrument
Sentinel-2A/B spectral response
Page 5
(c) HYPERNETS Consortium, 2018 (RBINS, TARTU, SU, CNR, NPL, GFZ, CONICET)(c) HYPERNETS Consortium, 2018 (RBINS, TARTU, SU, CNR, NPL, GFZ, CONICET)(c) HYPERNETS Consortium, 2018 (RBINS, TARTU, SU, CNR, NPL, GFZ, CONICET)NPL - Commercial
Instrument development
3 LAND
(VISNIR+SWIR)
4 WATER
The first set of HYPSTAR instruments has been produced and are currently being
tested in the field
Calibrations traceable to SI through NMI-calibrated lamp and reflectance panels
Rugged pc drives the
instrument, pan tilt,
communication with
server, …
Page 6
(c) HYPERNETS Consortium, 2018 (RBINS, TARTU, SU, CNR, NPL, GFZ, CONICET)(c) HYPERNETS Consortium, 2018 (RBINS, TARTU, SU, CNR, NPL, GFZ, CONICET)(c) HYPERNETS Consortium, 2018 (RBINS, TARTU, SU, CNR, NPL, GFZ, CONICET)NPL - Commercial
Hypernets Processor Introduction
Common processor for the land and water networks to process raw data to surface reflectance, developed as Python module.
There are two main use cases for the hypernets_processor module.
• Automated processing of network data for distribution to users.
• Ad-hoc sequence processing, for example for testing instrument operation in the field.
Page 7
(c) HYPERNETS Consortium, 2018 (RBINS, TARTU, SU, CNR, NPL, GFZ, CONICET)(c) HYPERNETS Consortium, 2018 (RBINS, TARTU, SU, CNR, NPL, GFZ, CONICET)(c) HYPERNETS Consortium, 2018 (RBINS, TARTU, SU, CNR, NPL, GFZ, CONICET)NPL - Commercial
Hypernets nomenclature
• Scan: a single measurement of the full spectrum
• Series: number of scans (typically 10) made with the same
azimuth and zenith angle
• Sequence: combination of series with different angles
files description
spe raw data from instrument
L0 raw data in netcdf
L1a calibrated scans
L1b calibrated series (scans averaged)
L1c calibrated series with coincident irradiance and radiance
measurements
L2a surface reflectances
Page 8
(c) HYPERNETS Consortium, 2018 (RBINS, TARTU, SU, CNR, NPL, GFZ, CONICET)(c) HYPERNETS Consortium, 2018 (RBINS, TARTU, SU, CNR, NPL, GFZ, CONICET)(c) HYPERNETS Consortium, 2018 (RBINS, TARTU, SU, CNR, NPL, GFZ, CONICET)NPL - Commercial
From raw to L1a for land and water network
Page 9
(c) HYPERNETS Consortium, 2018 (RBINS, TARTU, SU, CNR, NPL, GFZ, CONICET)(c) HYPERNETS Consortium, 2018 (RBINS, TARTU, SU, CNR, NPL, GFZ, CONICET)(c) HYPERNETS Consortium, 2018 (RBINS, TARTU, SU, CNR, NPL, GFZ, CONICET)NPL - Commercial
Raw data (VNIR+SWIR)
Page 10
(c) HYPERNETS Consortium, 2018 (RBINS, TARTU, SU, CNR, NPL, GFZ, CONICET)(c) HYPERNETS Consortium, 2018 (RBINS, TARTU, SU, CNR, NPL, GFZ, CONICET)(c) HYPERNETS Consortium, 2018 (RBINS, TARTU, SU, CNR, NPL, GFZ, CONICET)NPL - Commercial
Land network: L1a
Page 11
(c) HYPERNETS Consortium, 2018 (RBINS, TARTU, SU, CNR, NPL, GFZ, CONICET)(c) HYPERNETS Consortium, 2018 (RBINS, TARTU, SU, CNR, NPL, GFZ, CONICET)(c) HYPERNETS Consortium, 2018 (RBINS, TARTU, SU, CNR, NPL, GFZ, CONICET)NPL - Commercial
Land network: L1a to L2a
Page 12
(c) HYPERNETS Consortium, 2018 (RBINS, TARTU, SU, CNR, NPL, GFZ, CONICET)(c) HYPERNETS Consortium, 2018 (RBINS, TARTU, SU, CNR, NPL, GFZ, CONICET)(c) HYPERNETS Consortium, 2018 (RBINS, TARTU, SU, CNR, NPL, GFZ, CONICET)NPL - Commercial
Land network: L1b
Page 13
(c) HYPERNETS Consortium, 2018 (RBINS, TARTU, SU, CNR, NPL, GFZ, CONICET)(c) HYPERNETS Consortium, 2018 (RBINS, TARTU, SU, CNR, NPL, GFZ, CONICET)(c) HYPERNETS Consortium, 2018 (RBINS, TARTU, SU, CNR, NPL, GFZ, CONICET)NPL - Commercial
Land network: L1a to L2a
Page 14
(c) HYPERNETS Consortium, 2018 (RBINS, TARTU, SU, CNR, NPL, GFZ, CONICET)(c) HYPERNETS Consortium, 2018 (RBINS, TARTU, SU, CNR, NPL, GFZ, CONICET)(c) HYPERNETS Consortium, 2018 (RBINS, TARTU, SU, CNR, NPL, GFZ, CONICET)NPL - Commercial
Land network: L2A
- Temperature correction currently missing
- BRDF measurements can be made
Page 15
(c) HYPERNETS Consortium, 2018 (RBINS, TARTU, SU, CNR, NPL, GFZ, CONICET)(c) HYPERNETS Consortium, 2018 (RBINS, TARTU, SU, CNR, NPL, GFZ, CONICET)(c) HYPERNETS Consortium, 2018 (RBINS, TARTU, SU, CNR, NPL, GFZ, CONICET)NPL - Commercial
Uncertainties: MC methodGUM Supplement 1 – Monte Carlo Methods defines three main stages of uncertainty evaluation:
• Formulation: • Define Y, X, f
• Assign PDF
• joint PDF and
correlation matrix S
• Generate sample of
draws 𝑋𝑖 from these
joint PDF
• Propagation• Propagate each draw 𝑋𝑖 through the measurement function to get 𝑌𝑖 = 𝑓(𝑋𝑖)
• Together 𝑌𝑖 give the PDF for 𝑌
• Summarizing• Use the PDF for Y to obtain the expectation of Y, the standard uncertainty u(Y)
and the covariance between the different values in Y.
Page 16
(c) HYPERNETS Consortium, 2018 (RBINS, TARTU, SU, CNR, NPL, GFZ, CONICET)(c) HYPERNETS Consortium, 2018 (RBINS, TARTU, SU, CNR, NPL, GFZ, CONICET)(c) HYPERNETS Consortium, 2018 (RBINS, TARTU, SU, CNR, NPL, GFZ, CONICET)NPL - Commercial
MC method for HYPERNETS
• MC method implemented in NPL uncertainty propagation python package
• Correlations w.r.t. wavelength taken into account
• Gaussian PDF are assumed
• Draw samples from joint PDF using Cholesky decomposition method:
• First generate uncorrelated samples 𝑍𝑖 from gaussian
with mean X and std u(X)
• Calculate Cholesky decomposition 𝑅: 𝑅𝑇𝑅 = 𝑆 𝑋
• Correlate samples: 𝑋𝑖 = 𝑋 + 𝑅𝑇(𝑍𝑖−𝑋)
Page 17
(c) HYPERNETS Consortium, 2018 (RBINS, TARTU, SU, CNR, NPL, GFZ, CONICET)(c) HYPERNETS Consortium, 2018 (RBINS, TARTU, SU, CNR, NPL, GFZ, CONICET)(c) HYPERNETS Consortium, 2018 (RBINS, TARTU, SU, CNR, NPL, GFZ, CONICET)NPL - Commercial
MC method for HYPERNETS
L1A L2A
𝑌 calibrated (ir)radiances surface reflectances
𝑋 • Gains and non-linear from the calibration files
• DN and darks in the raw data
coincident radiances and
irradiances from L1C file
𝑓𝑓1 = 𝑔𝑎𝑖𝑛 𝜆 ∗
𝐷𝑁 𝜆 − 𝑑𝑎𝑟𝑘 𝜆
𝑐𝑛𝑜𝑛−𝑙𝑖𝑛𝑒𝑎𝑟 𝐷𝑁 𝜆 ∗𝑡𝑖𝑛𝑡1000
+ 0 𝑓4 = 𝜋𝐿
𝐸
PDF • Many (16) contributions to the PDF of the gain
and its correlation matrix from lab instrument
calibration (e.g. lamp, distance, panel …)
• Uncertainty of DN calculated from std
between scans
• Further contributions to be added (e.g.
temperature correction)
Gaussian with uncertainties
and correlation matrices
from L1C product
Page 18
(c) HYPERNETS Consortium, 2018 (RBINS, TARTU, SU, CNR, NPL, GFZ, CONICET)(c) HYPERNETS Consortium, 2018 (RBINS, TARTU, SU, CNR, NPL, GFZ, CONICET)(c) HYPERNETS Consortium, 2018 (RBINS, TARTU, SU, CNR, NPL, GFZ, CONICET)NPL - Commercial
Uncertainty budget
Three different types of uncertainties:• Random uncertainties (from std between scans)
• Systematic uncertainties (from calibration coefficient unc)
• Correlated systematic between rad and irr (cancel out in L2)
• Independent systematic
Propagated from product to product using MC
Each product has for each relevant variable:• Uncertainties for each scan/series for each of the three types
• One correlation matrix for each of the three types
Page 19
(c) HYPERNETS Consortium, 2018 (RBINS, TARTU, SU, CNR, NPL, GFZ, CONICET)(c) HYPERNETS Consortium, 2018 (RBINS, TARTU, SU, CNR, NPL, GFZ, CONICET)(c) HYPERNETS Consortium, 2018 (RBINS, TARTU, SU, CNR, NPL, GFZ, CONICET)NPL - Commercial
Land network: L1b uncertainty
Page 20
(c) HYPERNETS Consortium, 2018 (RBINS, TARTU, SU, CNR, NPL, GFZ, CONICET)(c) HYPERNETS Consortium, 2018 (RBINS, TARTU, SU, CNR, NPL, GFZ, CONICET)(c) HYPERNETS Consortium, 2018 (RBINS, TARTU, SU, CNR, NPL, GFZ, CONICET)NPL - Commercial
Land network: L2 uncertainty
Page 21
(c) HYPERNETS Consortium, 2018 (RBINS, TARTU, SU, CNR, NPL, GFZ, CONICET)(c) HYPERNETS Consortium, 2018 (RBINS, TARTU, SU, CNR, NPL, GFZ, CONICET)(c) HYPERNETS Consortium, 2018 (RBINS, TARTU, SU, CNR, NPL, GFZ, CONICET)NPL - Commercial
Correlation matrices (L2A)
Page 22
(c) HYPERNETS Consortium, 2018 (RBINS, TARTU, SU, CNR, NPL, GFZ, CONICET)(c) HYPERNETS Consortium, 2018 (RBINS, TARTU, SU, CNR, NPL, GFZ, CONICET)(c) HYPERNETS Consortium, 2018 (RBINS, TARTU, SU, CNR, NPL, GFZ, CONICET)NPL - Commercial
Uncertainty application: TOA vicarious calibration of S2
• No suitable test data available yet
we use simulated data based on ASD data taken at Gobabeb
• Compare to Sentinel-2A overpass at
roughly the same time of day
• Before the HYPERNETS data can be compared to satellite observations, we apply the following steps:
• Read/select satellite data and HYPERNETS data for appropriate angle
• Apply the atmospheric correction based on radiative transfer modelling
• Atmospheric parameters taken from RadCalNet at time of overpass
• Convolve the TOA spectrum with the satellite spectral response function
• Propagate uncertainties
• Compare
Page 23
(c) HYPERNETS Consortium, 2018 (RBINS, TARTU, SU, CNR, NPL, GFZ, CONICET)(c) HYPERNETS Consortium, 2018 (RBINS, TARTU, SU, CNR, NPL, GFZ, CONICET)(c) HYPERNETS Consortium, 2018 (RBINS, TARTU, SU, CNR, NPL, GFZ, CONICET)NPL - Commercial
Vicarious calibration uncertainties
• Metrological approach is used to propagate uncertainties through a given measurement function using MC.
• Measurement functions for RT model and spectral integration are defined in python using NPL packages
• Three uncertainty components propagated:• Random uncertainties from L2A product• Systematic uncertainties from L2A product (including
covariance matrix)• Uncertainties on atmospheric parameters from RadCalNet
(systematic)
Page 24
(c) HYPERNETS Consortium, 2018 (RBINS, TARTU, SU, CNR, NPL, GFZ, CONICET)(c) HYPERNETS Consortium, 2018 (RBINS, TARTU, SU, CNR, NPL, GFZ, CONICET)(c) HYPERNETS Consortium, 2018 (RBINS, TARTU, SU, CNR, NPL, GFZ, CONICET)NPL - Commercial
Results
Page 25
(c) HYPERNETS Consortium, 2018 (RBINS, TARTU, SU, CNR, NPL, GFZ, CONICET)(c) HYPERNETS Consortium, 2018 (RBINS, TARTU, SU, CNR, NPL, GFZ, CONICET)(c) HYPERNETS Consortium, 2018 (RBINS, TARTU, SU, CNR, NPL, GFZ, CONICET)NPL - Commercial
Uncertainty Results
Page 26
(c) HYPERNETS Consortium, 2018 (RBINS, TARTU, SU, CNR, NPL, GFZ, CONICET)(c) HYPERNETS Consortium, 2018 (RBINS, TARTU, SU, CNR, NPL, GFZ, CONICET)(c) HYPERNETS Consortium, 2018 (RBINS, TARTU, SU, CNR, NPL, GFZ, CONICET)NPL - Commercial
Conclusions
• HYPERNETS is developing a network of hyperspectral
instruments for multi-mission satellite validation
• First test sites now deployed and providing data
• Hyperspectral surface reflectance measurements will be
available over a range of water and land surface types and
can be used for BRDF measurements
• Detailed uncertainty budget available for every product
based on Monte Carlo uncertainty propagation
• Measurement function and uncertainty budget will contain
additional contributions after lab characterization of next batch