Geoscience Data Analysis and Visualization Tools from NCAR HDF/HDF-EOS Workshop XI Nov 6-8, 2007 David Brown
Jan 13, 2016
Geoscience Data Analysis and Visualization
Tools from NCAR
HDF/HDF-EOS Workshop XI
Nov 6-8, 2007David Brown
HDF/HDEOS Workshop XI, November 2007
Topics• Two interfaces: the same capabilities (mostly)
– NCL - a self-contained scripting language– PyNGL and PyNIO - Python modules
• Quick overview: visualization and analysis• Data model • State of our I/O component development• Enabling access to HDF 5, HDFEOS 5, and
NetCDF 4
Brief HistoryBrief HistoryNCAR NCAR
GraphicsGraphicsHLUsHLUsNCLNCLNIONIO
PyNGLPyNGLPyNIOPyNIO
PyNIO and PyNGLJuly 2006
(Numeric 24.x/NumPy 1.0)
PyNIO2005
PyHLU2003
PyNGL 2004
Original FortranLate 60s
GKS/CGM/Fortran 77/UNIX conversion1980s NCAR
Graphics
1992: NCARG ConferenceC interface
Mid 90sHLUs
Mid 90s
NCLMid 90s
GSUN - Late 90s
1995: NCL Conference
2000-present: 30+ NCL Workshops
NIO2005
HDF/HDEOS Workshop XI, November 2007
NCAR Command Language (NCL)A scripting language designed for
scientific data analysis and visualization
1. Simple, robust file input and output
2. Hundreds of analysis functions
3. Visualizations (2D) are world class and highly customizable
http://www.ncl.ucar.edu/
HDF/HDEOS Workshop XI, November 2007
NCL: Data analysis
• Array-based syntax and operators• Hundreds of functions- Spherical harmonics- Scalar and vector regridding- Vertical interpolation- EOFs
• Many tailored to geosciences• Most handle missing data• Can call C and Fortran routines
HDF/HDEOS Workshop XI, November 2007
NCL: Visualization
• High-quality and customizable visualizations• Contours, XY, vectors, streamlines• Maps with most common map projections• Handles data on regular and irregular grids,
triangular meshes• Specialized scripts for skew-T, wind roses,
histograms, Taylor diagrams, panels, bar charts• GSUN interface: simplifies visualization • Over 1,400 visualization options available
HDF/HDEOS Workshop XI, November 2007
HDF/HDEOS Workshop XI, November 2007
PyNGL - Python module• Python NCL Graphics Library • Python version of popular GSUN interface• Same publication-quality graphics as NCL• Utilizes existing Python modules and
development tools (swig, Numeric, NumPy)• Contains some climate-specific data
analysis functions• Extensive and updated documentation
http://www.pyngl.ucar.edu/
HDF/HDEOS Workshop XI, November 2007
NCL Data Model
• Based on netCDF 3 • Language variables can have:• Named dimensions• Attributes• Coordinates variables
• A convention in NetCDF but a language feature in NCL
• 1d array with the same name as a dimension• Contains coordinate values for dimension elements
HDF/HDEOS Workshop XI, November 2007
Supported file format I/O• One function reads all supported data
formats: • NetCDF (now including NetCDF4 classic),
HDF4, HDF-EOS 2, GRIB 1 and 2• Writes NetCDF and HDF4
• All file formats massaged into same model: file variables have basically the same features as internal variables
• Flat name space: HDF group names are appended to variable names
HDF/HDEOS Workshop XI, November 2007
PyNIO - Python module• Same I/O library as NCL (libnio)• Reads and writes same formats as NCL• Same NetCDF-like view of all formats• But interfaces with NumPy for data access• Modeled on the Scientific NetCDF module
– NioVariable type is a reference to the variable in the file (metadata is attached to variable)
– Dereferencing it results in a NumPy variable with no metadata
• *Numpy is a Python array processing module
http://www.pyngl.ucar.edu/Nio.shtml
HDF/HDEOS Workshop XI, November 2007
2D Coordinates• Traditional NetCDF coordinate variables are
1D vectors• However, satellite data and model data on
modern grids require 2D coordinate grids• 2D coordinate variables created on the fly for
GRIB grids and HDFEOS Grid-type data• Added value variables that allow data to be
plotted on any map projection
HDF/HDEOS Workshop XI, November 2007
Enabling NetCDF 4 ‘classic’ • Recompile with NetCDF 4 and HDF 5 1.8
beta (we also included szip)• Add options for specifying format and
compression level• Tests using ncl_convert2nc on GRIB files
show reduction in file size by ~1/2 over GRIB.• Caveats:
– Beta version NetCDF 4 not yet supported on some architectures (64 bit mainly)
– Not supported for OPeNDAP NetCDF client library
HDF/HDEOS Workshop XI, November 2007
Path to HDF, HDFEOS 5 and full NetCDF 4 support
• size_t for dimension sizes to enable large variables on 64-bit hosts
• Add support for more atomic types (int64) • Support for groups in the file context• User control of chunking• Compound and variable length data types• Components to read HDF 5 and HDFEOS 5• Current NetCDF module extended to handle v4• More support for aggregation
HDF/HDEOS Workshop XI, November 2007
Current status• NCL 5.0.0 released this week
– NCSA-style Open Source license (finally)– Binaries available under slightly more restrictive
license (because of included 3rd party software)
• PyNGL/PyNIO 1.2.0 available– Supports NumPy 1.0.x and Python 2.5– Binaries (easy installation) available for:
• Various flavors of Linux, Mac OSX, Cygwin
• Other Unix systems, 32 and 64 bit
Distribution of PyNGL/PyNIO sites
Distribution of NCL sites
HDF/HDEOS Workshop XI, November 2007
Support
• Websites with extensive documentation
• Tutorials
• Hundreds of examples with downloadable scripts
• Active email lists– [email protected], [email protected]
• Hands-on training workshops
HDF/HDEOS Workshop XI, November 2007
Download locations
• NCL
• PyNGL and PyNIO
http://www.pyngl.ucar.edu/download.html
http://www.ncl.ucar.edu/download.html
HDF/HDEOS Workshop XI, November 2007
Questions? Email me: [email protected]
HDF/HDEOS Workshop XI, November 2007
Resources (a.k.a. attributes)
• Same control mechanism for NCL and PyNGL
• Detailed control of the appearance of a visualization
• Example using POP grid ocean current data
HDF/HDEOS Workshop XI, November 2007
HDF/HDEOS Workshop XI, November 2007
HDF/HDEOS Workshop XI, November 2007
HDF/HDEOS Workshop XI, November 2007
HDF/HDEOS Workshop XI, November 2007
HDF/HDEOS Workshop XI, November 2007
NCL and PyNGL visualization examples
• scatter and line plots, contours, vectors, skew-t, meteograms, maps, wind barbs, bar charts, streamlines, trajectories, filled polygons, paneled visualizations
Skew-T graphicCourtesy ofDennis Shea
A Skew-T plot is used by
meteorologists to analyze data from
a balloon
sounding.
Based on a visualizationof Adam Phillips
Based on a visualizationof Joel Norris (Scripps)
using dummy data
Data from Climate Analysis SectionChristian Guillemot
THORPEX - an international research and development program responding to the weather related challenges of
the 21st century to accelerate improvements in the accuracy of 1-day to 2- week high impact weather
forecasts for the benefit of society, the economy and the
environment.
GRIB 2 TIGGE (THORPEX Interactive Grand Global Ensemble) data
Japan Meteorological Agency
Courtesy Jeff YinNCAR CGD
Image courtesy of Nan Rosenbloom, CGD
First two map databases built-in; high-resolution available as simple download
Christophe Cassou (CNRS/CERFACS)
This grid could be described as a tripole grid that is further modified by the arbitrary displacement of some portions of the grid to achieve finer resolution over areas of interest (typically, ocean areas).
Courtesy Mark Stevens, NCAR CGD
Grid from Tom GrossNOAA/NOS/CSDL/MMAP