VAPoR: A Discovery Environment for Terascale Scientific Data Sets Alan Norton & John Clyne National Center for Atmospheric Research Scientific Computing.

Supercomputing • Communications • Data

NCAR Scientific Computing Division

VAPoR: VAPoR: A Discovery Environment forA Discovery Environment forTerascale Scientific Data SetsTerascale Scientific Data Sets

Alan Norton & John Clyne Alan Norton & John Clyne National Center for Atmospheric ResearchNational Center for Atmospheric Research

Scientific Computing DivisionScientific Computing Division

Supercomputing • Communications • Data

NCAR Scientific Computing Division Alan [email protected]

VAPoR project overviewVAPoR project overview

GOALGOAL: Improve ability of earth sciences researchers to : Improve ability of earth sciences researchers to analyze and explore complex dynamics found in vast data sets analyze and explore complex dynamics found in vast data sets resulting from high-resolution, 4D (3D space x time) numerical resulting from high-resolution, 4D (3D space x time) numerical simulations of fluid flow in the oceans, atmosphere and sunsimulations of fluid flow in the oceans, atmosphere and sun

Open Source software development effort aimed at bridging Open Source software development effort aimed at bridging gap between visualization research and application tool gap between visualization research and application tool availabilityavailability

Funded by the NSF ITR program, VAPoR is a collaborative Funded by the NSF ITR program, VAPoR is a collaborative effort between NCAR’s Visualization and Enabling Technology effort between NCAR’s Visualization and Enabling Technology Section, U.C. Davis’ Institute for Data Analysis and Section, U.C. Davis’ Institute for Data Analysis and Visualization, and Ohio State University’s, Department of Visualization, and Ohio State University’s, Department of Computer and Information SciencesComputer and Information Sciences

Supercomputing • Communications • Data

NCAR Scientific Computing Division Alan [email protected]

The numbers – an example from solar compressible The numbers – an example from solar compressible convectionconvection 512x512x2048512x512x2048 5 variables 5 variables

(u,v,w,rho,temp)(u,v,w,rho,temp) ~500 time steps saved~500 time steps saved 9TBs storage9TBs storage Six months compute time Six months compute time

required on 112 IBM SP required on 112 IBM SP RS/6000 processors RS/6000 processors (blackforest)(blackforest)

Three months for post-Three months for post-processingprocessing

Data may be analyzed for Data may be analyzed for several yearsseveral years

Image courtesy of Joseph Mendoza, NCAR/SCD

Supercomputing • Communications • Data

NCAR Scientific Computing Division Alan [email protected]

Our approachOur approach

Exploit power of multiresolution data representation Exploit power of multiresolution data representation combined with progressive data access and efficient combined with progressive data access and efficient region sub-setting to enable user to make region sub-setting to enable user to make speed/quality tradeoffs speed/quality tradeoffs [[Pascucci and Frank, SC2001; Clyne, VIIP2003]Pascucci and Frank, SC2001; Clyne, VIIP2003]

Tightly couple traditional, non-visual (MatLab, IDL) Tightly couple traditional, non-visual (MatLab, IDL) analysis with highly interactive, advanced visualization analysis with highly interactive, advanced visualization [Clyne and Rast, VDA2005][Clyne and Rast, VDA2005]

Develop domain-specific application for numerical Develop domain-specific application for numerical simulation of fluid flowsimulation of fluid flow

Emphasize Emphasize usability usability for scientistsfor scientists

Supercomputing • Communications • Data

NCAR Scientific Computing Division Alan [email protected]

A visual comparison between coarsened and original dataA visual comparison between coarsened and original data

1283 5123

Supercomputing • Communications • Data

NCAR Scientific Computing Division Alan [email protected]

Combining visual and non-visual analysis toolsCombining visual and non-visual analysis tools

When visualization and quantitative tools are When visualization and quantitative tools are appropriately combined, result is a powerful appropriately combined, result is a powerful environment for scientific discoveryenvironment for scientific discovery

Couple existing quantitative tools (e.g. IDL, Matlab, Couple existing quantitative tools (e.g. IDL, Matlab, NCL) with VAPoR’s advanced visualization capabilitiesNCL) with VAPoR’s advanced visualization capabilities

Benefits of integration with existing analysis toolsBenefits of integration with existing analysis tools– Avoids duplication of effortAvoids duplication of effort– Lower development timeLower development time– Leverage what scientists already knowLeverage what scientists already know

Supercomputing • Communications • Data

NCAR Scientific Computing Division Alan [email protected]

Integrated analysis and visualization systemIntegrated analysis and visualization system

VtkHigh quality rendering

VAPoRInteractive visual browsing

IDLData manipulation &

analysis

Multiresolution data access

Disk Array

Supercomputing • Communications • Data

NCAR Scientific Computing Division Alan [email protected]

Software Design PrioritiesSoftware Design Priorities

Functional and usability requirements prioritized by scientific usersFunctional and usability requirements prioritized by scientific users Cross-platform (Linux, Irix, Windows, Mac,…)Cross-platform (Linux, Irix, Windows, Mac,…) Enable side-by-side comparison of multiple time-varying Enable side-by-side comparison of multiple time-varying

visualizationsvisualizations– Unique approach to management of multiple parameter setsUnique approach to management of multiple parameter sets

Exploit high-performance graphics cardsExploit high-performance graphics cards– GPU programmingGPU programming

Incorporate recent advances in data visualizationIncorporate recent advances in data visualization– Volume renderingVolume rendering– Flow visualizationFlow visualization– Isosurface generationIsosurface generation– Color/transparency mappingColor/transparency mapping

XML data descriptionXML data description– Encourage wider usage of multi-resolution data representationEncourage wider usage of multi-resolution data representation

Supercomputing • Communications • Data

NCAR Scientific Computing Division Alan [email protected]

DemonstrationDemonstration

Load multiresolution data into VAPoR applicationLoad multiresolution data into VAPoR application Combine global and local parameter settingsCombine global and local parameter settings Edit transfer function(s)Edit transfer function(s) Interactive region selectionInteractive region selection AnimationAnimation Export to IDLExport to IDL Use IDL for analysisUse IDL for analysis Visualize analysis resultsVisualize analysis results

Supercomputing • Communications • Data

NCAR Scientific Computing Division Alan [email protected]

Future PlansFuture Plans

Incorporate visualization techniques based on scientists’ Incorporate visualization techniques based on scientists’ needsneeds– Nonuniform gridsNonuniform grids– Vector field visualizationVector field visualization– IsosurfacesIsosurfaces– Contour planesContour planes

Understand effect of lossy data compressionUnderstand effect of lossy data compression– Error analysis and error visualization Error analysis and error visualization – Obtain bounds on degradation of analysis resultsObtain bounds on degradation of analysis results– Measure the effect of performing analysis on lower-resolution Measure the effect of performing analysis on lower-resolution

datadata Improve access to terabyte datasetsImprove access to terabyte datasets

– Multiresolution data output as a byproduct of the simulationMultiresolution data output as a byproduct of the simulation

Supercomputing • Communications • Data

NCAR Scientific Computing Division Alan [email protected]

SummarySummary

NSF funded collaboration with Davis and Ohio stateNSF funded collaboration with Davis and Ohio state Domain specific application aimed at improving productivity Domain specific application aimed at improving productivity

of turbulence researchers (though other groups may benefit)of turbulence researchers (though other groups may benefit) Scientists are target user group (not visualization specialists)Scientists are target user group (not visualization specialists) Multiresolution data representation exploited for handling Multiresolution data representation exploited for handling

large datalarge data Quantitative and advanced visualization methods combinedQuantitative and advanced visualization methods combined Usability is key to adoption by scientistsUsability is key to adoption by scientists

I can directly testify to the crucial importance of this [effort] to our I can directly testify to the crucial importance of this [effort] to our numerical work... to develop interactive tools that can access and numerical work... to develop interactive tools that can access and analyze the flow in its entirety, and visualize it in complex ways analyze the flow in its entirety, and visualize it in complex ways including parallelized algorithms of perspective volume rendering, including parallelized algorithms of perspective volume rendering, superposition of scalar and vector fields, and multi-resolution access superposition of scalar and vector fields, and multi-resolution access to the flows.to the flows.

Annick Pouquet, Director of ESSLAnnick Pouquet, Director of ESSL