Hydrologic Information System for the Nation Ilya Zaslavsky Spatial Information Systems Lab San Diego Supercomputer Center UCSD EPA talk, February 1, 2010.

Hydrologic Information System

for the NationIlya Zaslavsky

Spatial Information Systems LabSan Diego Supercomputer Center

UCSD

EPA talk, February 1, 2010

http://his.cuahsi.orghttp://hiscentral.cuahsi.orghttp://hydroseek.nethttp://river.sdsc.edu/ucsddashhttp://wron.net.au/DemosII/Modules/ODMKMLGatway.aspxhttp://maxim.ucsd.edu/mapmaker/storet.aspxhttp://water.sdsc.edu:7788/demo/CUAHSI/index.html

San Diego Supercomputer Center• Founded in 1985, as one of the five

original supercomputer centers, funded by the National Science Foundation

• 400 employees• Advanced research in high-

performance computing and networking

• R&D and cyberinfrastructure projects: in neuroscience, geology, astronomy, environmental sciences, molecular biology, hydrology

SDSC building on UCSD campus

SDSC Spatial Information Systems Lab

Research and system development• Services-based spatial information

integration infrastructure, CI projects• Mediation services for spatial data, query

processing, map assembly services• Long-term spatial data preservation• Spatial data standards and technologies for

online GIS (SVG, WMS/WFS, markup and services)

• Support of spatial data projects at SDSC and beyond

Mediator

LegendGenerator

MapAssembler

Ontology

…

GRID SERVICESFOR MAP INTEGRATION

Mediator

LegendGenerator

MapAssembler

Ontology

…

GRID SERVICESFOR MAP INTEGRATION

services

In Geosciences (GEON, CUAHSI, CBEO,…)

Spatial web services

FederalAgencies

Figure 1.26 The Geography Network.

ESRICounty spatial data and toxicant information

Telesis, other localNon-profits

CA state

WSDL

WSWSDL

WSWSDL

WSWSDL

WSWSDL

WSWSDL

WS

Student projects

The CHI ME Model

In regional development (NIEHS SRP, CRN…)

In Neurosciences (BIRN, CCDB)

http://spatial.sdsc.edu/

Contact: [email protected]

Consortium of Universities for the Advancement of Hydrologic Science, Inc.

An organization representing more than one hundred United States universities, receives support from the

National Science Foundation to develop infrastructure and services for the advancement of hydrologic

science and education in the U.S.http://www.cuahsi.org/

122 US Universities as

of July 2008

CUAHSI HIS: NSF support through 2012 (GEO)

Partners:Academic: 11 NSF hydrologic observatories, CEO:P projects, LTER, CZO…Government: USGS, EPA, NCDC, NWS, state and localCommercial: Microsoft, ESRI, KistersInternational: Australia, UKStandardization: OGC, WMO (Hydrology Domain WG); adopted by USGS, NCDC, Army Corps of Eng

An online distributed system to support the sharing of hydrologic data from multiple repositories and databases via standard

water data service protocols; software for data publication, discovery, access and integration.

What is the CUAHSI HIS?UT-Austin, SDSC/UCSD, Utah State U, Idaho State U, Drexel U, U of So. CarolinaPI: D. R. Maidment (UT-Austin)

Rainfall & SnowWater quantity

and quality

Remote sensing

Water Data

Modeling Meteorology

Soil water

Sources of Observations Data

Observation Stations

Ameriflux Towers (NASA & DOE) NOAA Automated Surface Observing System

USGS National Water Information System NOAA Climate Reference Network

Map for the US

Build a common window on water data using web services TOWARDS INTEGRATED WATER INFORMATION

9

Getting Water Data (the old way)Different Query Pages Different Query Responses

http://his.cuahsi.org/odmdatabases.html

CUAHSI Observations Data Model

Web Pages versus Web Serviceshttp://www.safl.umn.edu/ http://his.safl.umn.edu/SAFLMC/cuahsi_1_0.asmx

Uses Hypertext Markup Language (HTML) Uses WaterML (a Markup Language for water data)

Standard Water Data Services• Set of query

functions

• Returns data in WaterML

NWIS Daily Values (discharge), NWIS Ground Water, NWIS Unit Values (real time), NWIS Instantaneous Irregular Data, EPA STORET, NCDC ASOS, DAYMET, MODIS, NAM12K, USGS SNOTEL, ODM (multiple sites)

Next Step: WaterML 2.0; OGC Hydrology Domain Working Grouphttps://lists.opengeospatial.org/mailman/listinfo/hydro.dwg

http://external.opengis.org/twiki_public/bin/view/HydrologyDWG/WebHome

International Standardization of WaterML

13

OGC/WMO Hydrology Domain Working Grouphttp://external.opengis.org/twiki_public/bin/view/HydrologyDWG/WebHome

Towards an agreed upon - feature model- observations model- semantics

Expressed as WaterML 2.0By organizing - Interoperability Experiments and pilots,

standard design activities, webinars…

First OGC/WMO HydroDWG workshop : at Ispra, Italy, March 15-18, 2010

Test bed HISServers

Central HIS servers

ArcGIS

Matlab

IDL, R

MapWindow

Excel

Programming (C#, VB..)

Desktop clients

Customizable web interface

(DASH)

HTML - XMLW

SD

L - S

OA

P

Modeling (OpenMI)

Global search (Hydroseek)

Water Data Web Services, WaterML

Con

trol

led

voca

bula

ries

Met

adat

aca

talo

gs

Ont

olog

y

ET

L

serv

ices

HIS LiteServers

External data providers

Deployment to test beds

Other popular online clients

ODM DataLoader

Streaming Data Loading

Ontology tagging (Hydrotagger)

WSDL and ODM registration

Data publishing

ODMTools

Server config tools

HIS CentralRegistry & Harvester

Hydrologic Information System Service Oriented Architecture

HIS

Des

kto

p

15

• Built for data– Storage– Loading– Analysis– Publication

• HIS Software free of charge

HIS Server Real-time Sensors

WaterOneFlowWeb Service

Data Archives

Outside Users, HIS Central, HydroDesktop

Local Users

ODM Tools

HIS

Ser

ver

SQL Server

Observations DataModel Database

HIS Central – Catalog and Search

16

Managing Varying Semantics

Nitrogen: e.g. NWIS parameter # 625 is labeled ‘ammonia + organic

nitrogen‘, Kjeldahl method is used for determination but not mentioned in

parameter description. In STORET this parameter is referred to as Kjeldahl

Nitrogen.

And: Dissloved oxygen

acre feet acre-feet

micrograms per kilogram

micrograms per kilgram

FTU NTU

mho Siemens

ppm mg/kg

In measurement units…

In parameter names…

Semantic Tagging of Harvested Variables

20

• Service registry and metadata catalog– Networks– Sites– Variables– Search Keywords

• Does not store actual observation data

• Example: GetSitesInBox query function

HIS Central ServicesHICentral

Web Service

GetValues Requests Per Day from HIS Central

All Services NWIS0

200

400

600

800

1000

1200

1400

1600

February - De-cember 2008January - October 2009

22

HydroDesktop Capabilities

• Add shapefiles to map• Change symbology and

labels• Print and export map

• GIS toolbox

GIS

• Search for data• Download data

• Display time series• Export data

Hydrology

Hydroseekhttp://www.hydroseek.net

Supports search by location and type of data across multiple observation networks including NWIS, Storet, and academic data

Visualization and Analysis of Large Datasets► Tiled wall

► OLAP cubes for repositories► OLAP cubes for catalogs

EPA STORET water quality repository

USGS NWIS catalog:

measurement totals

for selected nutrients over

decades

How we work with agencies on web service access to observational data

1. Establish an agreement with the agency on joint development of water data services, identify agency partners (ideally, with time to support joint work)

2. Identify the scope of the service, databases to be exposed, and access control, assign network and vocabulary codes

3. Map semantics of the service to WaterML semantics, and verify with the agency

4. Include discussion of agency data and interoperability issues in the context of OGC Hydrology Domain WG (if needed)

5. Develop a first draft of the web service

6. Unit testing, over a series of validation cases developed jointly with the agency

7. Harvest an observations metadata catalog for agency data, to be housed either at SDSC or at the agency

8. Develop a procedure for catalog updates

9. Register the water data service at HISCentral (including mapping of variables to ontology terms), and test it using HydroSeek and HydroExcel. Document the service

10. Review and test the service together with the agency, for possible approval as “operational”

Federal Agency Water Data Services at HISCentral (10/09)

Network Name Site Count Value Count Earliest Observation Notes

NWISDV 29894 274762525 1/1/1900 WaterML-compliant GetValues service from NWIS, catalog ingested

EPA 362645 94349967 1/1/1900 SOAP wrapper over WQX services, catalog harvested

NWISUV 11185 77935488 60 DAYS WaterML-compliant GetValues Service, catalog ingested

NCDC ISH 11555 3000000* 1/1/2005 WaterML-compliant GetValues service from NCDC, catalog harvested

NCDC ISD 24770 18165478 1/1/1892 WaterML-compliant GetValues service from NCDC, catalog harvested

NWISIID 369148 15501245 1/9/1867 SOAP wrapper over NWIS web site, catalog harvested

NWISGW 827200 8325633 1/1/1900 SOAP wrapper over NWIS web site, catalog harvested

RIVERGAGES 2206 263101295 1/1/2000 WaterML compliant REST services from Army Corps of Engineers

Integration with other infrastructures

• Technical: with real time data management middleware:– OpenSource DataTurbine:

http://geo.sdsc.edu/jnlp/usgs.jnlp

• Communities:– Superfund Basic Research

Program (NIEHS)– CZOs (Boulder Creek,

Stroud Water Research Center)– South East Asia/Malaysia(http://seahydro.um.edu.my/search)

CUAHSI Water Data Services

28

Map Integrating NWIS, STORET, & Climatic Sites 47 services

15,000 variables1.8 million sites9 million series

4.3 billion data values

The largest water datacatalog in the world

Summary• CUAHSI HIS = Cyberinfrastructure for managing and publishing

observational data– Supports many types of point observational data– Overcomes syntactic and semantic heterogeneity using a standard data model

and communication language, controlled vocabularies, and ontology management (created a comprehensive ontology of hydrologic parameters)

– End-to-end infrastructure: from publishing water data to discovery, access, retrieval, analysis

– Supports a national network of observatory test beds, and links to most important federal water data repositories

– Maintains national registry of services (the largest in the world)

• WaterML is a standard language for consistently communicating water observations data from academic and government sources.– already adopted by several federal agencies– Further enhanced through WMO/OGC Hydrology Domain Working Group

• Already deployed at multiple locations; free and open source

Working with EPA Data• Wrapping WQX services;

http://river.sdsc.edu/wiki/CUAHSI%20WebServices.ashx • Ingesting STORET• OLAP over STORET, and web based mapping

application: http://maxim.ucsd.edu/mapmaker/storet.aspx

• Ontology work:– Semantic wiki: http://water.sdsc.edu:7788/demo/CUAHSI/index.html

– Recent ontology:

How we can collaborate

• Regular catalog updates• Development on water quality data services• Portal for accessing hydrologic time series• Support of hydrologic data center (proposal to NSF?)• Ontology development• Mechanism:

– MOU between EPA and CUAHSI on water data services– Involvement in HydroDWG Water Quality IE and AIP-3

• http://external.opengis.org/twiki_public/bin/view/HydrologyDWG/WebHome