Analyzing Multidimensional Scientific Data in ArcGIS Nawajish Noman Deng Ding
Analyzing Multidimensional Scientific Data
in ArcGIS
Nawajish Noman
Deng Ding
Outline
• Part I
- ArcGIS Platform
- Scientific Multidimensional Data
- Ingest and Data Management
• Part II
- Data Exploration and Visualization
- Analysis
- Extending Analytical Capabilities using Python
• Part III
- Publishing and Sharing Services
- Consuming Services
Desktop Web Device
Server Online Content
and Services
Portal
Available on the Open Web . . .
. . . and On-Premises
ArcGIS An Integrated Web GIS Platform
Providing Mapping, Analysis, Data
Management, and Collaboration
Identity
Security
Scientific Multidimensional Data
• Stored in netCDF, GRIB, and HDF formats
• Multidimensional
• Ocean data
Sea temperature, salinity, ocean current
• Weather data
Temperature, humidity, wind
• Land
Soil moisture, NDVI, land cover
ArcGIS
direct ingest
data management
visualizationanalysis
share
Scientific Data in ArcGIS - Vision
Challenges
variety of formats
volume & velocity
redundancy
portability
scalability
reproducibility
integration
standards
accessibility
Analyze ShareManage
• Data is represented as
• Raster
• Feature
• Table
• Direct read
• Exports GIS data to netCDF
Representing Scientific Data in ArcGIS
• Directly reads netCDF file using
- Make NetCDF Raster Layer
- Make NetCDF Feature Layer
- Make NetCDF Table View
• Ingest OPeNDAP Service
- Output dynamic multidimensional raster
- Support Sub-setting
• Scientific data formats are supported in mosaic dataset
- netCDF
- HDF
- GRIB
Ingesting Scientific data in ArcGIS
Climate and Forecast (CF) Conventionhttp://cf-pcmdi.llnl.gov/
Initially developed for
• Climate and forecast data
• Atmosphere, surface and ocean model-generated data
• Also for observational datasets
• CF is now the most widely used conventions for geospatial netCDFdata. It has the best coordinate system handling.
• Current version 1.6
• You can use Compliance checker utility to check a netCDF file.
http://cf-pcmdi.llnl.gov/conformance/compliance-checker/
CF Convention
• Geographic Coordinate Systems (GCS)
- X dimension units: degrees_east
- Y dimension units: degrees_north
• Projected Coordinate Systems (PCS)
- X dimension standard_name: projection_x_coordinate
- Y dimension standard_name: projection_y_coordinate
- Variable has a grid_mapping attribute.
- CF 1.6 conventions currently supports thirteen predefined coordinate systems (Appendix F: Grid Mappings)
• Undefined
- If not GCS or PCS
• ArcGIS writes (and recognizes) PE String as a variable attribute.
NetCDF and Coordinate Systems
Demo: Ingest
What about Aggregation?
• Create a seamless multi-dimensional cube from
- files representing different regions
- files representing different time steps/slices
• Mosaic dataset supports multiple files and variables, normalize time and depth
Multidimensional Mosaic Dataset
mosaic
table
raster pixels
Representing multivariate collection of multidimensional rasters in ArcGIS
mosaic dataset
spatially-indexed catalog
multi-resolution, multivariate, multidimensional
reduce storage redundancy & pixel resampling
defines information products
on-demand processing
HDF
GRIB
netCDF
d-aware rasters
Tabular View
Tabular view of items in a multivariate multidimensional mosaic dataset
Raster Shape Variable StdTime StdZ
… … Temperature 3/22/2016 -10
… … Temperature 3/23/2016 -10
… … Temperature 3/24/2016 -10
… … Salinity 3/22/2016 -10
… … Salinity 3/23/2016 -10
… … Salinity 3/24/2016 -10
… … Temperature 3/22/2016 -20
… … Temperature 3/23/2016 -20
… … … … …
• Supports netCDF, HDF and GRIB- Spatial Aggregation
- Temporal Aggregation
- On-the-fly analysis
• Serve as Multidimensional- Image Service
- Map Service
- WMS
• Supports direct ingest
• Eliminates data conversion
• Eliminates data processing
• Improves workflow performance
• Integrates with service oriented architecture
Scientific data support in Mosaic Dataset
Demo: Management
Behaves the same as any layer or table
• Display
- Same display tools for raster and feature layers will work on multi-
dimensional raster and feature layers.
• Graphing
- Driven by the table just like any other chart.
• Animation
- Multi-dimensional data can be animated through time dimension.
• Analysis Tools
- Will work just like any other raster layer, feature layer, or table. (e.g.
create buffers around points, reproject rasters, query tables, etc.)
Using Scientific Data in ArcGIS
• Slicing
• Temporal animation using Time Slider
• Dimensional animation using Range Slider
• Predefined renderer
Visualization of Scientific Data
Time = 1
141 241 341
131 231 331
121 221 321
111 211 311
441
431
421
411
142 242 342
132 232 332
122 222 322
112 212 312
442
432
422
412
143 243 343
133 233 333
123 223 323
113 213 313
443
433
423
413
Y
X
Time
Changing Time Slice
Animating through Time using Time Slider
Animating through Depth using Range Slider
• New Vector Field renderer for raster
- Supports U-V and Magnitude-direction
- Dynamic thinning
- On-the-fly vector calculation
• Eliminates raster to feature conversion
• Eliminates data processing
• Improves workflow performance
Visualization of Raster as Vectors
Demo: Visualization
• Hundreds of analytical tools available for raster, features, and table
• Temporal Modeling
- Looping and iteration in ModelBuilder and Python
Spatial and Temporal Analysis
• Several analytical functions are available out of the box
• Functions are chained together to create complex model
• Used to perform on-the-fly analysis
• Extend analytical capability using Python Raster Function
On-the-Fly Processing using Raster Functions
Modeling with Raster Function Template (RFT)
• A scientific model = a raster function template
• Windchill = 35.74 + 0.6215T - 35.75(V^0.16) + 0.4275T(V^0.16)
• RFT
- Created using
- Contains Query, GroupName, Variables
- Set Mosaic dataset properties
- Used during adding raster data
• netCDF4-python is included in 10.3/Pro
• Read and write netCDF file
• Conversion time values to date
• Multi-file aggregasion
• Compression
• SciPy
• Python Raster Function
https://www.unidata.ucar.edu/software/netcdf/workshops/2012/netcdf_python/netcdf4python.pdf
Python Package: netCDF4-Python, SciPy
Create Space-Time Cube & Emerging Hot Spot Analysis
Analyzing Multidimensional Scientific Data in ArcGIS
Supplemental tools
• OPeNDAP to NetCDF
• Make NetCDF Regular Point Layer
• Make NetCDF Station Point Layer
• Make NetCDF Trajectory Point Layer
• Describe Multidimensional Dataset
• Get Variable Statistics
• Get Variable Statistics Over Dimension
• Multidimensional Zonal Statistics
• Multidimensional Zonal Statistics As Table
http://blogs.esri.com/esri/arcgis/2013/05/24/introducing-the-multidimension-supplemental-tools-2/
Python – Extending Analytical Capabilities
• Marine Geospatial Ecology Tools (MGET)
• Developed at Duke Univ.
• Over 180 tools for import management,
and analysis of marine data
http://mgel.env.duke.edu/mget
• Australian Navy tools
(not publicly available)
Community Developed Tools
Demo: Analysis
Desktop Web Device
Online
Content and
Services
Server
multivariate multidimensional
mosaic dataset
Disseminating
professional
geospatial analysts
• Map Service (supports WMS)
- Makes maps available to the web.
• Image Service (supports WMS)
- Provides access to raster data through a web service.
• Geoprocessing Service
- Exposes the analytic capability of ArcGIS to the web.
Sharing / WMS Support (for multi-dimensions)
Sharing Your Map as Map Services
• File ≻ Share As ≻ Service
• Features overlaid on rasters
• Static vs Dynamic map service
• Multidimensional WMS
Sharing Data & Information Products as Image Service
• Mosaic Dataset ≻ Share As Image Service
• Pixels & item metadata
• On-demand server-side processing
• Raster models: predefined or client-specified
• Multidimensional info, filtering, vector fields
Enable access to a dynamic representation of your information product as an image service
Publishing a WMS on ArcGIS Server
• Enable WMS capabilities on Service Editor or Manager
Multi-dimensional data support in WMS
• getCapabilities
- Supports time, elevation and other dimensions (e.g. depth)
• getMap
- Returns map for any dimension value
&DIM_<dimensionName>=<value>&
- Supports CURRENT for time dimension
&TIME=CURRENT&
• getFeatureInfo
- Returns information about feature for any dimension value
Multi-dimensional WMS in ArcMap
• Supports WMS layer like any other layer
• Animates a time enabled WMS layer using time-slider
• Slices for any dimension value are accessible with ArcObjectsPublic Sub UpdateWMSServiceLayerDimensionValue()
'UID for wms service layer type
Dim pUid As New uid
pUid = "{27ABB9EC-7A26-4cf8-8BD4-70EC1D274E17}"
Dim pWMSMapLayer2 As IWMSMapLayer2
'calling a function to find the layer from active dataframe
Set pWMSMapLayer2 = GetLayer(pUid, "myWMSLayer")
'setting values to dimensions
Dim pDimNameValues As IPropertySet
Set pDimNameValues = New PropertySet
pDimNameValues.SetProperty "Depth", "500" 'dimension#1
pDimNameValues.SetProperty "T1", "500" 'dimension#2
Set pWMSMapLayer2.DimensionValues = pDimNameValues
'calling a function to redraw the layer
RefreshActiveDataFrame
End Sub
ArcGIS Online: The Living Atlas
www.arcgis.com
The Living Atlas
• Imagery
• Basemaps
• Demographics & Lifestyle
• Boundaries & Places
• Landscape
• Community Maps
• Transportation
• Urban Systems
• Earth Observations
• Historic Maps
Services of Scientific Data
• MODIS data
- MODIS land cover 2000-2011
- MODIS Vegetation Analysis
- MODIS Greenland Sea Ice
• Live NOAA wind service
• NASA Global Land Data Assimilation (GLDS)
- Soil moisture
- Evapotranspiration
- Snow pack
• More
Online Imagery content that can be directly used:
SUBHEAD INFORMATION
Demo: Publishing
Consuming Scientific Data Services
• ArcGIS Desktop/Pro/Server
• Web Map Viewer
• Web Applications
• Story maps
• Operational Dashboard
Corrected Reflectance – Consumed in ArcGIS Desktop
WMS in Dapple Earth Explorer
Tell the story of your scientific data – Create Story Maps
http://dtc-sci01.esri.com/DeadZoneStoryMap/
Demo: Consumption