Sponsored by the National Science Foundation GENI I&M Workshop NetCDF and Local Data Manager (LDM) Mike Zink November 4, 2010 .

Sponsored by the National Science Foundation

GENI I&M Workshop

NetCDF and Local Data Manager (LDM)

Mike ZinkNovember 4, 2010

http://geni.cs.umass.edu/visehttp://geni.cs.umass.edu/dicloud

http://www.geni.net

Sponsored by the National Science Foundation 2November 3, 2010

Overview

Sponsored by the National Science Foundation 3November 3, 2010

What is NetCDF

NetCDF is a set of software libraries and machine-independent data formats that support the creation, access, and sharing of array-oriented scientific data.

Uses HDF5 as data storage layer.

• Also provides read-only access to some HDF4, HDF5 archives.

• Parallel I/O for high performance computing.• Developed and maintained by Unidata at UCAR

Sponsored by the National Science Foundation 4November 3, 2010

Who uses NetCDF?

• NetCDF is widely used in University Earth Science community.

• Used for Intergovernmental Panel on Climate Change (IPCC) data sets.

• Used by NASA and other large data producers.• Used for NEXRAD and CASA radar data

(moments)

Sponsored by the National Science Foundation 5November 3, 2010

NetCDF Example

5

netcdf chickasha.ok-20090527-103631 { dimensions: Radial = UNLIMITED ; / / (1289 currently) Gate = 426 ; variables: double Azimuth(Radial) ; Azimuth:Units = "Degrees" ; double Elevation(Radial) ; Elevation:Units = "Degrees" ; double GateWidth(Radial) ; GateWidth:Units = "Millimeters" ; double StartRange(Radial) ; StartRange:Units = "Millimeters" ; int StartGate(Radial) ; StartGate:Units = "Unitless" ; int Time(Radial) ; Time:Units = "Seconds" ; double TxFrequency(Radial) ; TxFrequency: Units = "Hertz" ; double TxLength(Radial) ; TxLength:Units = "Seconds" ; double TxPower(Radial) ; TxPower:Units = "dBm" ; int AfcSet(Radial) ; AfcSet:Units = "Unitless" ; int GcfState(Radial) ; GcfState:Units = "Unitless" ; float Reflectivity(Radial, Gate) ; Reflectivity:Units = "dBz" ; float Velocity(Radial, Gate) ; Velocity:Units = "MetersPerSecond" ; float SpectralWidth(Radial, Gate) ; SpectralWidth:Units = "MetersPerSecond" ; float DifferentialReflectivity(Radial, Gate) ; DifferentialReflectivity: Units = "dB" ; float DifferentialPhase(Radial, Gate) ; DifferentialPhase:Units = "Degrees" ; float CrossPolCorrelation(Radial, Gate) ; CrossPolCorrelation:Units = "Unitless" ; float NormalizedCoherentPower(Radial, Gate) ; NormalizedCoherentPower:Units = "Unitless" ; float SpecificPhase(Radial, Gate) ; SpecificPhase:Units = "DegreePerKm" ; float HPropagationPhase(Radial, Gate) ; HPropagationPhase:Units = "Radians" ; float VPropagationPhase(Radial, Gate) ; VPropagationPhase:Units = "Radians" ; int GateFlags(Radial, Gate) ; GateFlags:Units = "BitField" ; float CorrectedReflectivity(Radial, Gate) ; CorrectedReflectivity: Units = "dBZ" ; float CorrectedDifferentialReflectivity(Radial, Gate) ; CorrectedDifferentialReflectivity:Units = "dB" ; // g lobal attributes: :NetCDFRevision = "$Id: mdmd.c,v 1.33 2008-02-28 00:11:57 junyent Exp $" ; :RadarName = "chickasha.ok" ; : Latitude = 35.031226436707 ; :Longitude = -97.9566907121578 ; :Height = 353.986269181594 ; :MccId = 124335663 ; :NumGates = 426 ; :S canFlag = 1 ; :S canType = 2 ; :S canId = 29137 ; : PosAccel = 60. ; : PosVelocity = 12. ; :AntennaGain = 37. ; :AntennaBeamwidth = 1.8 ; }

    float Reflectivity(Radial, Gate) ;        Reflectivity:Units = "dBz" ;    float Velocity(Radial, Gate) ;        Velocity:Units = "MetersPerSecond" ;    float SpectralWidth(Radial, Gate) ;        SpectralWidth:Units = "MetersPerSecond" ;    float DifferentialReflectivity(Radial, Gate) ;        DifferentialReflectivity:Units = "dB" ;    float DifferentialPhase(Radial, Gate) ;        DifferentialPhase:Units = "Degrees" ;    float CrossPolCorrelation(Radial, Gate) ;        CrossPolCorrelation:Units = "Unitless" ;    float NormalizedCoherentPower(Radial, Gate) ;        NormalizedCoherentPower:Units = "Unitless" ;    float SpecificPhase(Radial, Gate) ;        SpecificPhase:Units = "DegreePerKm" ;    float HPropagationPhase(Radial, Gate) ;        HPropagationPhase:Units = "Radians" ;    float VPropagationPhase(Radial, Gate) ;        VPropagationPhase:Units = "Radians" ;    int GateFlags(Radial, Gate) ;        GateFlags:Units = "BitField" ;    float CorrectedReflectivity(Radial, Gate) ;        CorrectedReflectivity:Units = "dBZ" ;    float CorrectedDifferentialReflectivity(Radial, Gate) ;        CorrectedDifferentialReflectivity:Units = "dB" ;

Sponsored by the National Science Foundation 6November 3, 2010

OpeNDAP Client for NetCDF

OPenDAP (http://www.opendap.org/) is a widely supported protocol for access to remote data

Defined and maintained by the OPenDAP organization

Designed to serve as intermediate format for accessing a wide variety of data sources.

Client is now built into netCDF C library.

Sponsored by the National Science Foundation 7November 3, 2010

Local Data Manager (LDM)

• LDM software acquires data and shares them with other nodes

• Data product is treated as a opaque unit, thus nearly any data can be relayed

• LDM can handle data from:– National Weather Service "NOAAport channel 3" data

streams– NEXRAD radar data– lightning data from the National Lightning Detection

Network– GOES satellite imagery

Sponsored by the National Science Foundation 8November 3, 2010

LDM Features

• User configurable– execute an arbitrary program with the data product as

input

• Supports distributed processing– Data ingest functions can be separated from storage

and use functions– LDM servers may be configured to relay subsets of

data among themselves, distributing the storage and processing around the network.

• Extensible– New decoders can be added easily

• Event-driven– Handles data as it arrives

Sponsored by the National Science Foundation 9November 3, 2010

Demo Data Flow• Dynamic end-to-end Nowcasting

– Mapping Nowcast Workflows onto GENI

Archival Storage

Radar Nodes

“raw” live data

Upstream LDM feed

archived netcdf data

Nowcast Processing

aggregatedmulti-radar data

Post to Web

Nowcast images for display

NetCDF over LDM

LDM server

Storage

NetCDF over LDM

LDM server

Mer

ge

NetCDF over LDM

LDM server

Now

casting

Sponsored by the National Science Foundation 10November 3, 2010

Multi-radar NetCDF Data

Nowcast Processing

1. DiCloud Archival Service (S3)2. LDM Data Feed (EC2)

“raw” live data

Generate “raw” live dataViSE/CASA radar nodes

http://stb.ece.uprm.edu/current.jsp

Use proxy to track usage-based spending on Amazon and enforce quotas and limits

http://geni.cs.umass.edu/vise/dicloud.php

1. Ingest mulit-radar data feeds2. Merge and grid multi-radar data2. Generate 1min, 5min, and 10min Nowcasts3. Send results over NLR to Umass4. Repeat

ViSE views steerable radars as shared, virtualized resources

http://geni.cs.umass.edu/vise

Nowcast images for display