Ceclia Deluca - An Architecture for Many-Component Regional Hydrologic Modeling

An Architecture for Many-Component Regional Hydrologic Modeling

Cecelia DeLucaNOAA Environmental Software Infrastructure and Interoperability (NESII)

NOAA ESRL/University of ColoradoCollaborating team: Rocky Dunlap/CIRES, Dan Rosen/CIRES,

Scott Peckham/University of Colorado, Wei Yu/NCAR, Dave Gochis/NCAR,Sujay Kumar/NASA, Mike Ek/NOAA EMC, Jiariu Dong/NOAA EMC

3rd CUAHSI Conference on HydroinformaticsJuly 17, 2015

Bridging Disciplines and Approaches

The climate and (to a lesser extent) weather communities have engaged in many-component (5-50 component) modeling efforts for more than a decade

These efforts established practices and have evolved tools that could be leveraged or extended by the hydrologic community

Modeling efforts at the National Water Center have the potential to advance the representation of hydrology in coupled modeling systems used for a variety of purposes

Motivates efforts to bridge disciplines and experiment with connections of modeling frameworks with support from the NSF Earth System Bridge project

Federal Coupled Modeling System Examples

NEMS: NOAA Environmental Modeling System Next-generation operational prediction for

weather through seasonal time scales Some applications in operations now, some in

development

NASA GEOS-5 Global Circulation Model and ModelE Research in data assimilation techniques and

utilization of satellite measurements Seasonal forecasting, climate forecasting, creation

of reanalysis data sets

CESM: Community Earth System Model Research into all aspects

of the climate system National and international

assessments, includingparticipation in theIntergovernmental Panelon Climate Changeassessment reports

Navy Forecast Systems (COAMPS, NavGEM) Research and operational weather forecasting in

support of military operations and national security

Surface winds fromCOAMPS Navy model

Earth System Prediction Suite ESPS COUPLED MODELING SYSTEMS

NEMS and CFS COAMPS NavGEM GEOS-5 ModelE CESM

Model Driver ATMOSPHERE MODELS GSM NMMB CAM FIM GEOS-5 Atmosphere ModelE Atmosphere COAMPS Atmosphere NavGEM

NEPTUNE

OCEAN MODELS MOM5 HYCOM NCOM POP POM SEA ICE MODELS CICE KISS OCEAN WAVE MODELS WW3 SWAN LEGEND Components are NUOPC compliant and the technical correctness of data

transfers in a coupled system has been validated.

Components and coupled systems are partially NUOPC compliant. From Theurich et al. 2015,in submission

The Earth System Prediction Suite is a collection of federal and community models and components that use the Earth System Modeling Framework (ESMF) with conventions called the National Unified Operational Prediction Capability (NUOPC) Layer

ESMF standard component interfaces enable major U.S. centers to assemble systems with components from different organizations, and test a variety of components more easily.

The multi-agency Earth System Prediction Capability (ESPC) supports adoption efforts

The Earth System Modeling Framework (ESMF) was initiated in 2002 as a multi-agency response to calls for common modeling infrastructure.

ESMF provides: high performance utilities, including

grid remapping, data communications, and model time management

an architecture for model construction

ESMF has become a standard for federal research and operational models in climate, weather, and space weather.

Earth System Modeling Framework

https://www.earthsystemcog.org/projects/esmf/

Metrics:~6000 downloads

~100 components in use

~3000 individuals on info mailing list

~40 platform/compilers regression tested nightly

~6500 regression tests

~1M SLOC

ESMF Standard Component Interfaces

APIs in Fortran, C, and Python

All ESMF components have the same three standard methods (these can have multiple phases)

- Initialize- Run- Finalize

Each standard method has the same simple interface:

call ESMF_GridCompRun (myComp, importState, exportState, clock, )

Where:myComp points to the componentimportState is a structure containing input fieldsexportState is a structure containing output fieldsclock contains timestepping information

Interfaces are wrappers and can often be introduced in a non-intrusive and high performance way, i.e. ESMF is designed to coexist with native model infrastructure

HOMME Cubed Sphere Grid with PentagonsCourtesy Mark Taylor of Sandia

ESMF Grid RemappingUniquely fast, reliable, and general interpolation weights computed in parallel in 3D spaceSupported grids: Logically rectangular and unstructured grids Global and regional grids 2D and 3D grids GIS formats through link with OpenClimateGIS (more on this)Supported interpolation methods: Nearest neighbor, higher order patch recovery, bilinear and 1st order conservative methodsOptions for straight or great circle lines, masking, and a variety of pole treatmentsMultiple ways to call ESMF grid remapping: Generate and apply weights using the ESMF API, within a model Generate and apply weights using ESMPy, through a Python interface Generate weights from grid files using ESMF_RegridWeightGen, a command-line utility

FIM Unstructured Grid Regional Grid

Somesupportedgrids

OpenClimateGIS

OpenClimateGIS (OCGIS) is a Python-based, open source software library enabling dynamic access to and manipulation of climate data

Its goal is to overcome barriers of usability of climate projections in adaptation planning and resource management Translate out of climate data formats to GIS-friendly formats Select geographical regions of interest Select times/levels of interest Compute application-relevant indices Convert to end-user and analysis-ready formats Provide comprehensive metadata

Recently integrated with ESMPy for easier grid remapping from catchment geometry to model grid

http://www.earthsystemcog.org/projects/openclimategis/

ESMPy + OpenClimateGIS

NFIE- inspired example Subset high resolution climate precipitation data to local scale and then regrid to

hydrologic sub-basins

Source data: CF formatted precipitation data file for the continental United States (nldas_met_update.obs.daily.pr.1990.nc)

Output: Multi-dimensional precip values (including time) on a subset of 3 catchment basins in region of interest after generation and application of conservative interpolation weights

NFIE Example Workflow

Using ESMPy and OpenClimateGIS a Python workflow was created to subset and conservatively regrid output from daily precipitation data (NetCDF) and a hydrologic sub-basin shapefile

Output shapefile contains the full time series of regridded precipitation data and all original attributes (i.e. ComID)

More information on the NFIE workflow example is available here: http://bit.ly/1Hqj3sy

The National Unified Operational Prediction Capability (NUOPC) is a consortium of operational weather prediction centers and their research partnersThe NUOPC Layer adds to ESMF: Definitions for model component interactions during

Initialize, Run, Finalize (I/R/F) for interoperability even with complex initialization interactions

Extensible, generic components that implement standard phases of the I/R/F

A field dictionary that supports multiple conventions and aliases, used for matching fields between components

Component Explorer and Compliance Checker tools that report component incompatibilities detected during run-time

Static compliance checking and code generation via the Cupid Integrated Development Environment (IDE)

NUOPC Interoperability Layer

ESMF API

NUOPC API

https://www.earthsystemcog.org/projects/nuopc/

calls and supplements

NUOPC introduces a set of pre-fabricated model component templates for building coupled systems

NUOPC wrappers or caps contain translations of native data structures (e.g. grids, field data, time quantities) into ESMF data structures.

The National Unified OperationalPrediction Capability

Connectors

Mediator

Driver

Model

NUOPC Generic Components

Simple driver (a) and schematic of Navy regional model COAMPS (b)

Cupid Development and Training Environment

NASA-funded, Eclipse plug-in Cupid parses NUOPC-based code and outlines coupled model structure

Shows NUOPC compliance issues and generates code templates to satisfy NUOPC compliance

NOAA Environmental Modeling System

https://www.earthsystemcog.org/projects/nuopc/

The NOAA Environmental Modeling System (NEMS) is a rapidly evolving coupled modeling system that will be the basis of critical predictive systems at NOAA, including 10 day, six week, and 9 month forecasts

Global and regional applications Components will include

- Atmosphere- Ocean- Land- Wave- Sea ice- Coastal/storm surge- Hydraulics- Ionosphere- Aerosol/chemistry

NEMS Status Current

Model:ICE

Model:OCN

Model:ROF

Model:WAV

AtmosphereGSMNMMas chosen

IceCICEas chosen

OceanPOPMOM5MOM6HYCOM

HydraulicsWRF-Hydro

WaveWaveWatch III

Model:ICE

Model:OCN

Model:HYD

Model:WAV

Current status: white=coupled, grayed out=in progress, solid=not started

Med

iato

r

nems

Med

iato

r

NUOPC Driver

Model:GLC

Aerosol/ChemGOCARTCMAQ

Model:CHM

Model:ATM

IonosphereIPE

Model:ATM

MediatorMediator

Model:GLC

LandLISModel:

LND

Model:ROF

CoastalADCIRCModel:

CST

Model:OCN

Model:IPM

NEMS Planned 12/2015

Model:ICE

Model:OCN

Model:ROF

Model:WAV

AtmosphereGSMNMMas chosen

IceCICEas chosen

OceanPOPMOM5MOM6HYCOM

HydraulicsWRF-Hydro

WaveWaveWatch III

Model:ICE

Model:OCN

Model:HYD

Model:WAV

Anticipated end of calendar year 2015: white=coupled, grayed out=in progress, solid=not started

Med

iato

r

nems

Med

iato

r

NUOPC Driver

Model:GLC

Aerosol/ChemGOCARTCMAQ

Model:CHM

Model:ATM

IonosphereIPE

Model:ATM

MediatorMediator

Model:GLC

LandLISModel:

LND

Model:ROF

CoastalADCIRCModel:

CST

Model:OCN

Model:IPM

NEMS Mediator Mediator transforms and transfers data can be multiple or specialized Multiple coupling periods, now includes slow (ocean) and fast

(atmosphere and ice) coupling periods The NEMS mediator performs the following functions:

Connects fields whose standard names match Accumulates and averages fields as necessary Merges fields with a generic merge method that allows for

weighting Performs custom coupling operations, along with unit

transformations Performs interpolation (fluxes are mapped bilinearly, states

conservatively, higher order also available)

More information about the mediator:http://cog-esgf.esrl.noaa.gov/projects/couplednems/mediator_design

Sample NEMS Configure File################################# NEMS Run Time Configuration File # ################################# MED #med_model: nemsmed_petlist_bounds: 60 65

#ATM#atm_model:

gsmatm_petlist_bounds: 0 31Processor layout

# OCN #ocn_model: mom5ocn_petlist_bounds: 32 55

# ICE #ice_model: ciceice_petlist_bounds: 56 59

Processor layout

Colors show actions performed by: Connectors (->) Mediator (MED) Models

(@) indicates coupling timesteps

# Run Sequence #runSeq::

@7200.0OCN -> MEDMED MedPhase_slowMED -> OCNOCN

@3600.0MED MedPhase_fast_beforeMED -> ATMMED -> ICEATMICEATM -> MEDICE -> MEDMED MedPhase_fast_after

@@

::

Assembling NEMS Modeling Applications

NEMS AppBuilder: Enables users to construct a specific, versioned modeling application

from a versioned set of model components and configuration files from multiple locations.

Helps to ensure that changes made to the different applications are coordinated as they get checked back into the NEMS repository.

Implemented using SVN externals, can be used with git repositories

More about the AppBuilder:http://cog-esgf.esrl.noaa.gov/projects/couplednems/appbuilder

The AppBuilder is based on low-level terminal-based Unix utilities for maximum ease of use and portability. A command-line version will be available shortly.

Running NEMS Applications

Component sets (compsets): A labeling system that distinguishes different run configurations for

many-component modeling applications. Labels are associated with scripts that pull together all the files and

inputs needed to run the specified configurations.- standard runs can be set up easily and consistently- effective way to implement regression testing across a coupled

system with many possible combinations of components Each modeling application is typically associated with multiple

compsets.

More about compsets:http://cog-esgf.esrl.noaa.gov/projects/couplednems/compsets

Using CompsetsRunning Compsets:./NEMSCompsetRun [COMPSET_LIST_FILE]

Compset syntax:caselabel_architecture_model1[%opt1[%opt2[...[%optN]]]]_model2[...]_..._modelN[...]

Where the file has the format:### List of compsets ###########################

AMIP_sbys_gsm ! stand-alone GSM - fake example2009_nems_gsm_cice_mom5 ! NEMS mediator coupled GSM-CICE-MOM52011_sbys_gsm%wam ! stand-alone GSM run as WAM2011_sbys_gsm%wam%ndsl ! stand-alone GSM run as WAM with NDSL

Supported compsets:http://cog-esgf.esrl.noaa.gov/projects/couplednems/supported_compsets

NEMS Seasonal 0.1 Milestone A first version (0.1) of the NEMS global seasonal prediction was completed in

June 2015. Three-way coupled atmosphere-ocean-ice system - GSM (T126) -MOM5 (1

deg) CICE (1 deg)

SST after 5 days of a 15 day run Focus so far has been on technical correctness Model initialization is not fully in place Next steps focus on adding grid resolutions and initializing from CFSR so

comparative runs with last operational model can begin

Image courtesy of Fei Liu, NOAA CIRES

Next Milestones

August 2015 NEMS regional 0.1 milestone WRF-Hydro hydrology and Noah-MP (via the NASA Land Information

System) running side by side uncoupled with other active NEMS components (still global atmosphere and ocean, CICE ice model)

Important to have ability to validate each component standalone! Currently WRF-Hydro is capped with NUOPC and running in NEMS (with

a minor I/O glitch) and LIS is in the process of being capped December 2015 NEMS regional 0.2 milestone WRF-Hydro hydrology and Noah-MP or Noah (via the NASA Land

Information System) running coupled with other NEMS components (target regional NMMB atmosphere and regional HYCOM ocean, CICE ice model)

May 2016 NEMS regional 0.3 milestone Flexible, but likely ADCIRC added to coupled system Potential for experimental connection to CSDMS components

Conclusion

A prototype many-component system is being constructed at NOAA that includes a whole Earth set of coupled components, using ESMF/NUOPC Layer infrastructure

An initial delivery of a seasonal application shows reasonable behavior and is entering an evaluation phase

Next NEMS milestones include a regional application, and bring in WRF-Hydro, a separate land component, and a coastal/storm surge model

This regional system can serve as an exploratory testbed for both technical and scientific aspects of this complex coupling, and a bridge to connect hydrologic and climate/weather communities

Thank you!

Any questions?

Slide Number 1Slide Number 2Slide Number 3Slide Number 4Slide Number 5ESMF Standard Component InterfacesESMF Grid RemappingOpenClimateGISESMPy + OpenClimateGISNFIE Example WorkflowSlide Number 11Slide Number 12Slide Number 13Slide Number 14Slide Number 15Slide Number 16Slide Number 17Slide Number 18Slide Number 19Slide Number 20Slide Number 21Slide Number 22Slide Number 23Slide Number 24Slide Number 25