National Remote Sensing Data Sets - IMOSimos.org.au/fileadmin/user_upload/shared/IMOS General/BWCN/king.pdf · National Remote Sensing Data Sets Edward King | IMOS SRS Facility 29

National Remote Sensing Data Sets

Edward King | IMOS SRS Facility 29 November 2012

MARINE & ATMOSPHERIC RESEARCH + WEALTH FROM OCEANS FLAGSHIP

Production and Access (SRS sub‐facilities SST + AODAAC + OceanColour)

Back in the dark ages (2007)

National Remote Sensing Data Sets; Production & Access | Edward King2 |

• 21st Century remote sensing:

•What could we possibly do better?• Multiple copies of data sets, overlapping but exclusive, or duplicated• Incompatible formats, data stored on tape• Local data sets – associated with local reception|applications|customs• Each recalibration means data from overseas becomes obsolete• Data invisible or uncatalogued• So even if people could find data sets, it wasn’t so much that people wouldn’t share data – they just couldn’t• and, we were already in the low TB, with 100s of TB foreseeable

• At the coal face – it wasn’t so much the despair, but the hope!

The IMOS Opportunity

• IMOS/SRS investments in this area:1. National approach to SST by the BoM, an operational agency2. A catalogue system – the “AODAAC”


• Infrastructure support for research• Chance to do something we wouldn’t normally spend research $ on• To make research easier/faster/possible

• And then in 2010, EIF funding, including advances in eResearch3. National approach to Ocean Colour


Sea Surface Temperature (Helen Beggs & Bureau Team)

• Whole of Aust. SST using AVHRR• built upon previous CSIRO effort to compile a stitched national archive (all reception stations), motivated by USGS Global LAND 1km project

• 8TB & 100,000+ passes in 2007

• Contribution to Global GHRSST• Inherited strong standards• Best practices

• 1992 to the present

• Spent (what seemed like) a long time getting correct procedures in place for calibration and getting GHRSST standards to settle.

• Adopted netCDF (CF‐compliant) format throughout – portable, self‐documenting, archive quality data representation

NOAA AVHRR SST: Products

multi-swath, single sensor multi-swath, multi-sensor, multi-day (3-day)

multi-swath, multi-sensorsingle swathSwath SST

N18: 04:01:33

N18: 2011-04-30 04:01:33

N18: 04:01:33N18: 05:42:09

N19: 02:01:49N19: 03:40:20N18: 04:01:33N19: 05:22:46N18: 05:42:09

L3U

L3C

L3S

L3S

L2PL1B

Swath BT

GHRSST SST products in netCDF format

GHRSST SST

Products

Description

L2P Geolocated SST from individual sensors over a single orbit/scene including additional error, quality

flag and data fieldsL3 Composite, gridded, SST products (no

interpolation)L3U – Uncollated (single swath)

L3C – Collated (single sensor, multiple-swaths)L3S – Super-collated (multiple sensors)

L4 Gridded, gap-free, SST analyses created by statistically interpolating multiple L2/L3 products

Australian SST today

• L2P is a foundational product – calibrated to high accuracy, geolocated*, quality flags, consistently formatted, and online

• Everything else (L3x+) can be built from it• You can look at the L3U data on http://oceancurrent.imos.org.au/

Presentation title | Presenter name7 |

• Contemporary data available, processing back to 1992 proceeding apace

• Higher level products (RAMSSA, ReefTemp etc)• You can make your own products too if you want

• Other sensors – MTSAT (geostationary) addressing new types of problems (high time res.)

http://oceancurrent.imos.org.au/

AODAAC (Australian Oceans Distributed Archive & Access Centre) CMAR

• Designed to be an automatic data catalogue with enhanced retrieval capability (ie. software & standards on top of archives).• Based on earlier TPAC crawler code• Suffered from– Inheriting JAVA codebase– Fanciful aspirations of naïve project team – (who were all trying to learn JAVA)– Massive de‐scoping in first 2 years in order to achieve working prototype– Hiatus following retirement of project leader after 2 years of development– Role conflict with eMII, leading to delays (but a blessing in disguise)


Which is…


• Gridded data exposed via net • Catalogue automatically • Enable spatio‐temporal search• Permit data subset requests• Provide aggregated data

• It’s a web‐based data interface for gridded data

Status


• 2011/12 has seen major design revision, refactor and upgrade

• (while waiting for eMII portal)• Has completed nearly all internal testing

• Documentation and transfer to eMII is underway

• Prototype appears in eMII demo portal (see AODN newsletter)

• Should go live Dec/Jan• Is being adopted by TERN/AusCover for gridded data

T

Ocean Colour (CSIRO + GA)

• EIF funding in 2010 brought• Investment into national eResearch facilities• An opportunity for IMOS to plan additional work

• The Ocean Colour subfacility has sought to meet Climate & Bluewater node needs by providing locally processed and calibrated ocean colour data (ie Chl) for whole of Australia• Local processing = no download from overseas upon recalibration• Capacity to implement multitude of algorithms• Full space and time resolution• Full set of channels/radiances/Rrs for– Cal/val– Algorithm development (both in blue and coastal waters)


Ocean Colour ‐MODIS


• Base data for Aust. ~40TB from 2000• Overlap with terrestrial users• Partner with TERN/AusCover• Utilise National Compute Infrastructure (NCI) CPU+Storage

• And build a national processing system to meet widest possible range of needs.

• Real time merging of MODIS data from throughout Aust.• Supplemented <6hrs with NASA data• Operating since Aug 2011 (see OceanCurrent)• Standard production includes L1B (split point for TERN/IMOS)• Full set of L2 ocean products ‐ equiv. to SST L2P (once validated)

Ocean Colour ‐MODIS• Data since 2011 are online now• New version of processing code is being used to generate whole Aqua archive (online Dec).

• L2 dataset forms basis for • Matchup DB for cal/val• Chl record using std. algorithms


• Platform for testing and developing new algorithms and products (eg. PP and Tricodesmium, Phytoplankton Functional Types)

• Supports major projects – eg eReefs, Kimberley, Great Aust. Bight • NCI storage & services enable easy access and sharing• NCI CPUs permit reprocessing of all products/channels at full res. with updated cal in weeks, rather than downloading 100s of TB from US

Ocean Colour ‐ SeaWIFS

• Forerunner to MODIS, 1997‐2010• Rehabilitate CMAR+AIMS+WA merged data• Deliver as per MODIS• L1B (geolocated reflectances)• L2 (full suite of channels, products)

• 3600 days x 4 passes – 9 days @ NCI• Consistent formatting/metadata• Is now online• Remapped products soon in netCDF format• Additional local products and cal/val work as per MODIS


Conclusion• Has taken a while to get things in place• Starting to see the utility of these data• Both SST and Chl are prominent in OceanCurrent• Uptake in projects (eReefs etc)• No more downloading recalibrated products• Everything is online locally for all of Australian region, for everyone/all users• SeaWIFS+MODIS data can support physics‐based algorithm development for coastal waters

• Format and metadata standards are providing a platform for unified tool development/re‐use

• Long AODAAC journey looks to be coming to a successful culmination in the IMOS portal. It should make life easier for users of these big time series.

• The landscape is very different now from 2007. Talk to us about your data needs.


Thank youMarine & Atmospheric ResearchEdward KingSRS Facility leader

t +61 3 6232 5334e [email protected] www.csiro.au

MARINE & ATMOSPHERIC RESEARCH + WEALTH FROM OCEANS FLAGSHIP

National Remote Sensing Data Sets - IMOSimos.org.au/fileadmin/user_upload/shared/IMOS General/BWCN/king.pdf · National Remote Sensing Data Sets Edward King | IMOS SRS Facility 29

Documents