Parma 2016-05-17 - JGrass-NewAGE - Some About The State of Art

The JGrass-NewAGE system essentials Concepts

Rigon R.

Arpae, Parma, 17 Maggio 2017

Giu

sep

pe

Pen

on

e

!2

R. Rigon

Introduction

MODELS

Mad

emoi

selle

Ros

e, c

a. 1

820.

Eug

ène

Del

acro

ix.

!3

DataParametersEquations

Mass, momentum and energy conservation. Chemical transformations

Forcings and observables

Equation’s constant. In time! In space they are usually heterogeneous

Models we are talking about are computer applications

In the past they were built as monolithic programs

R. Rigon

Which kind of models

!4

I - Once a model, design and implemented as a monolithic software entity, has been deployed, its evolution is totally in the hands of the original developers. While this is a good thing for intellectual property rights and in a commercial environment, this is absolutely a bad thing for science and the way it is supposed to progress.

Rob

bed

from

a C

CA

pre

sent

atio

n

R. Rigon

The old way

!5

II - Independent revisions and third-party contributions are nearly impossible and especially when the code is not available. Models falsification (in Popper sense) is usually impossible by other scientists than the original authors.

III- Thus, model inter-comparison projects give usually unsatisfying results. Once complex models do not reproduce data it is usually very difficult to determine which process or parameterization was incorrectly implemented.

R. Rigon

The old way

!6

Q: HOW CAN WE BE MORE “GALILEIAN” ?

A: YES, PRODUCING AND PROMOTING OPEN SOURCE MODELS. THIS HOWEVER IS NOT ENOUGH SINCE MODELS SHOULD BE STRUCTURALLY EASY TO UNDERSTAND, DOCUMENT, MODIFY, MAINTAIN, AND FAVOR PROCESSES ANALYSIS.

R. Rigon

The new way

!7

MODELLING, FOR WHO ? Which end user do you have in mind ?

Babo

on, P

apiu

s An

ubis

R. Rigon

No models for everyone

!8

Modified from Rizzoli et al., ,2005

Roles Users

Hard Coders

Soft Coders

Linkers Runners Player Viewers Providers

Prime

Other End Users

Technical

Researchers

R. Rigon

Users/Roles

!9

SO WHAT ?

R. Rigon

Solutions

!10

Component-oriented software development. Objects (models and data) should be packaged in components, exposing for re-use only their most important functions. Libraries of components can then be re-used and efficiently integrated across modelling frameworks. Yet, a certain degree of dependency of the model component from the framework can actually hinder reuse.

NEW (well relatively) MODELING PARADIGMS

Mod

ified

from

Riz

zoli

et a

l., 20

05

R. Rigon

Software Engineering Solutions

!11

R. Rigon

Components

!12

A F T E R 1 0 Y E A R S , W H Y T H E S E SOFTWARES BY COMPONENTS I N F R A S T R U C T U R E S D I D N OT EMERGE ?

R. Rigon

Existing Examples ?

TOO INVASIVE !

TOO MANY COMPUTER SCIENTISTS, TOO FEW HYDROLOGISTS ?

!13

A BLUEPRINT ?

Esch

er -

Dra

win

gs h

ands

, 194

8

R. Rigon

Infrastructure design

!14

DataParametersEquations

Mass, momentum and energy conservation. Chemical transformations

Forcings and observables

Equation’s constant. In time! In space they are heteorgeneous

Numerics, boundary and initial conditions

Data Assimilation. Data Models. Tools for Analysis.

Calibration, derivation from proxies

To Sum up

R. Rigon

Infrastructure design

!15

Decision making

EVALUATION OF STRATEGIES THROUGH MODELS

STRATEGIES FOR POLICY MAKERSDATA INTERPRETATION

EVALUATION OF STRATEGIES THROUGH MODELSEVALUATION OF STRATEGIES THROUGH

MODELS

DATA INTERPRETATIONDATA INTERPRETATION

STRATEGIES FOR POLICY MAKERSSTRATEGIES FOR POLICY MAKERS

R. Rigon

Infrastructure design

!16

PREREQUISITES General

Programming LANGUAGE NEUTRAL

PLATFORM NEUTRAL: Windows, Linux and Mac

OPEN SOURCE

TARGETED AT PERSONAL PRODUCTIVITY OF DIFFERENT USERS People come before program efficiency.

BUSINESS NEUTRAL: GPL would be fine if encapsulated in components

R. Rigon

Infrastructure design

!17

PREREQUISITES Technologies

ALLOWS WRAPPING OF EXISTING CODES BUT PROMOTES BETTER PROGRAMMING STRATEGIES

DATA BASE AWARE

DEPLOYABLE THROUGH THE WEB or as a web-server

USES MULTICORES

COMPLIANT OF STANDARDS (OGC, CUAHSI, OTHERS)

R. Rigon

Infrastructure design

!18

PREREQUISITES Documentation/Replicability

WITH TOOLS THAT HELPS DOCUMENTATION

COMPLIANT TO STANDARDS FOR DEFINING VARIABLES (e.g. VARIABLES AND PARAMETERS)

MANAGED IN A PUBLIC REVISION CONTROL SYSTEM (e.g. GIT)

HAVING A STANDARD WAY AND PLACES TO EXPOSE DOCUMENTATION

R. Rigon

Infrastructure design

The JGrass-NewAGE system essentials Deployment choices

Rigon R, Formetta G., Antonello A., Franceschi S.

Arpae, Parma, 17 Maggio 2017

Giu

sep

pe

Pen

on

e

!20

JAVA

OMS

GEOTools

JGrassTools

C/C++

Languages and infrastructures/libraries

Rigon et al.

Formetta et al., 2014

!21

JAVA

OMS

GEOTools

JGrassTools

C/C++

Python

FORTRAN

ESMF

A competing solution

Rigon et al.

Formetta et al., 2014

!22

JAVA

OMS

GEOTools

JGrassTools

C/C++

OPENMI

C#

Another competing solution

Rigon et al.

Formetta et al., 2014

!23

parameter editing, and a visual model builder. As developmentcontinued, both the OMS2 and Netbeans! API improved but alsobecame large and complex. Emerging requirements such as web-service deployment and the need to support model creation inother integrated development environments resulted in redesignof OMS2 in favor of the more flexible and “lightweight” OMS3framework design. OMS3 (David et al., 2010) provides a new APIwhich leverages existing concepts of component-based modeling.The new OMS3 architecture was driven by the need for:1) elegance and simplicity in component design; 2) performanceand scalability to support implicit multi-threading for componentexecution; 3) faster adaptation of legacy code; and 4) flexible toolextensions to implicitly integrate model calibration and sensi-tivity/uncertainty analysis methods, resulting in the utilization ofDSLs for simulation setup and execution.

As shown in Fig. 1, OMS3 is comprised of four primary architec-tural foundations including modeling resources, the system knowl-edge base, development tools, and modeling products. The OMS3core consists of an internal metadata knowledge base for model andsimulation creation. A simulation in OMS is defined as a collection ofresources (parameter sets, input data, modeling components, modelexecution method, etc.) required to produce desired modelingoutputs. The system supports harnessing metadata from varioussources including natural resources databases (e.g., land use/cover,soil), web-based data provisioning services, version control systems,and/or other code repositories, which is incorporated into theframework knowledge base that various OMS3 development toolsemploy to create modeling products. OMS3 modeling productsinclude science components and complete models, simulationssupporting parameter estimation and sensitivity/uncertainty anal-ysis, output analysis (e.g., statistical evaluation and graphical visu-alization) tools, modeling audit trails (i.e., reproducing model resultsfor legal purposes), and miscellaneous technical/user documenta-tion. As with any EMF, fully embracing the OMS3 architecturerequires a commitment to a structured model development processwhich may include the use of a version control system for modelsource code management or databases to store audit trails. Suchfeatures are important for institutionalized adoption of OMS3 butless critical for adherence by a single modeler.

3.2. Framework invasiveness and OMS3

The degree of dependency between a framework and simulationmodel code can be described as “framework invasiveness”, definedby Lloyd et al. (2011a) as the degree to which model code is coupledto the underlying framework. Framework to modeling code inva-siveness occurs due to several factors, including the use of a frame-work API consisting of data types and methods/functions whichdevelopers use to harness framework functionality, the use offramework-specific data structures (e.g., classes, types, andconstants), or the implementation of framework interfaces andextension of framework classes. Framework to application inva-siveness is a type of code coupling; object-oriented coupling(i.e., coupling between classes in an object-oriented program) hasbeen shown to correlate inversely with the likelihood of a mistake inthe code (Briand et al., 2000). Mistakes in model code negativelyimpact the functional correctness of the code, thereby reducing thefunctional aspects of code quality. Other important aspects of modelcode quality include “non-functional” quality attributes such asmaintainability, portability/reusability, and understandability. Lloydet al. (2011a) demonstrated that code invasiveness incurred byusing a modeling framework is correlated to non-functional codequality metrics. The impact of this invasiveness can be considered asthe degree of dependency imposed by the modeling framework fora specific modeling problem.

Why is a non-invasive framework approach important forOMS3? Most environmental modelers are natural resource scien-tists frequently with only self-taught experience in programmingand little or no proficiency with software architecture and design.Most environmental modeling development projects do not havethe luxury of employing experienced software engineers orcomputer scientists who are able to understand and apply complexdesign patterns, UML diagrams, and advanced object-orientedtechniques such as parameterized types, higher level data struc-tures and/or object composition. The use of object-oriented designprinciples for modeling can be productive for a specific modelingproject that has limited need for external reuse and extensibility.Extensive use of object-oriented design principles can be difficultfor scientists to adopt in that adoption often entails a steep learning

Fig. 1. OMS3 principle framework architecture.

O. David et al. / Environmental Modelling & Software xxx (2012) 1e134

Please cite this article in press as: David, O., et al., A software engineering perspective on environmentalmodeling framework design: The ObjectModeling System, Environmental Modelling & Software (2012), doi:10.1016/j.envsoft.2012.03.006

OMS

Rigon et al.

David et al., 2012

!24

curve requiring management of complex programming concepts(e.g., polymorphic execution flow in an inheritance hierarchy).Object-oriented design principles have been promoted for nearlytwo decades as a promising technology for supporting environ-mental modeling. As experience has shown, well-designed object-oriented models are difficult and time consuming to develop,particularly with a design objective to support reuse, maintain-ability, and understandability. Design of complex systems requiresexperience, anticipation of future use cases by providing extensionpoints, freedom to refactor poor aspects of a design, and adequatetime and resources. Environmental modeling perspectives,concepts, and approaches vary and are not easy to capture ina single object-oriented design. The environmental modelingcommunity maintains many legacy models still in use based onalgorithms and equations developed decades ago. What haschanged and continues to change are the hardware and softwareinfrastructures that house and deliver the output from environ-mental models. Devices such as smart phones and cloud computing(described later) are emerging technologies which non-invasiveEMFs can readily support.

EMFs can also be classified as heavyweight or lightweightbased on various design characteristics (Lloyd et al., 2011a). Theprimary difference between these framework types is how theypresent functionality to the developer. Heavyweight frameworks,e.g., traditional object-oriented frameworks such as Java’s SwingApplication Framework for graphical user interface development,provide developers with an API that is can be large and unwieldyrequiring developers to spend considerable time becomingfamiliar with before writing model code. The lightweight frame-work design approach adopted by OMS3 originated from variousweb application and enterprise frameworks (Richardson, 2006). Incontrast to heavyweight frameworks, lightweight frameworksoffer functionality to the developer using a variety of techniquesaimed at reducing the API’s overall size and developer depen-dence on the API. A lightweight EMF adapts to an existing model’ssource code, resulting in a less steep learning curve as there is nocomplex API to understand or special data types to manage. Thisprovides several practical implications for environmentalmodelers as there is no major paradigm shift since existingmodeling code and libraries are being used. Adopting and usinga lightweight framework is easier since modeling componentsused by the lightweight framework can still function and continueto evolve outside the framework.

Enabled by the lightweight and non-invasive characteristics ofthe OMS3 modeling framework, creating a modeling objectbecomes a rudimentary task as there are no interfaces to imple-ment, no classes to extend, no polymorphic methods to override,and no specialized framework-specific data types to use. OMS3 usesmetadata bymeans of language annotations to specify and describe“points of interest” amongst existing data fields and class methodsof the model. To verify the improvement in code quality of usingannotation-based components and models versus traditional APIapproaches, Lloyd et al. (2011a) conducted a study comparing theimplementation of several component-based hydrology modelswithin different languages and EMFs. They applied several codequality metrics to quantify code characteristics of the differentmodel implementations and found that the non-invasive frame-work approach of OMS3 enabled more concise model imple-mentations, in terms of number of lines of code and lower codecomplexity, for environmental model development. For example,the OMS3 implementation of the Thornthwaite model requiredonly 295 lines of code whereas other EMFs required between 450and 1635 lines of code. All of the model implementations hadidentical functionality and produced the same modeling results.Furthermore, implementation of the PRMS model in OMS3

required only 10,163 lines of code compared with 16,997 for OMS2(Lloyd et al., 2011a). Software engineering research suggests thatreduction in code size typically results in lowermodel developmentand maintenance costs over the lifetime of a model (Briand et al.,2000). Outside of the environmental modeling domain, similarsuccess stories have been observed which are currently driving thepopularity of lightweight web services frameworks such as JBossSeam and Spring (Yuan et al., 2009).

3.3. OMS3 component-based modeling concepts

Like other modeling frameworks such as OpenMI (Blind andGregersen, 2005; Gregersen et al., 2007), Common ComponentArchitecture (CCA) (Bernholdt et al., 2003), Earth System ModelingFramework (ESMF) (Collins et al., 2005), and Common ModelingProtocol (CMP) (Moore et al., 2007), OMS3 uses classes as thefundamental model building block while embracing principles ofcomponent-based software engineering for the model develop-ment process. The advantages of constructingmodular software arewell known in the software engineering field. Individual modulescan be developed using standardized interfaces supporting modulecommunication. Partitioning a system into modules typically helpsto minimize coupling, which should lead to code that is easier tomaintain.

The term component refers to self-contained, separated soft-ware units that implement independent functions in context-independent manner. In this paper, we refer to components asmodeling entities which implement a single conceptual modelingconcept. A component can be hierarchical in that it may orchestrateinteraction among other finer-grained components through the useof different categories of annotations in OMS3. Components exposeframework-relevant aspects via metadata, and each componentshould provide a sufficient level of complexity within a model’scomponent hierarchy. Fig. 2 shows the principle layout of compo-nents as supported in OMS3 including managing data flow/execu-tion phases and building component hierarchies. Like many EMFs,OMS3 provides core features including functional encapsulationsupporting isolation of individual computational aspects intocomponents, facilitation of directed data flow (input/output slots orexchange items), and management of various execution stateswithin components including “Initialize/Run/Finalize” as describedby Peckham (2008).

While object-oriented methods focus on abstraction, encapsula-tion, and localization of data and methods, their use can also lead tosimulation systems where objects are highly co-dependent. To

Fig. 2. OMS3 component architecture including data flow, execution phases, andencapsulation.

O. David et al. / Environmental Modelling & Software xxx (2012) 1e13 5

Please cite this article in press as: David, O., et al., A software engineering perspective on environmentalmodeling framework design: The ObjectModeling System, Environmental Modelling & Software (2012), doi:10.1016/j.envsoft.2012.03.006

Components again

Rigon et al.

David et al., 2012

!25

4. OMS3 applications

4.1. Cloud Services Innovation Platform (CSIP)

Recent attention has been directed towards enabling OMS3 toseamlessly scale models through the use of cloud infrastructuresand service-oriented architectures (SOAs). Cloud computing isemerging as a viable and attractive solution for scientificcomputing (Hoffa et al., 2008). The main goal is to scale parallelprocessing of components beyond individual computers to harnessnetworks of virtual computers while not requiring significant codechanges. For that purpose, the USDA-NRCS has initiated the CloudServices Innovation Platform (CSIP) (Fig. 4).

The goal of CSIP is to develop a scalable, modular, cost effective,and open deployment platform for simulation models to deliverlegacy and research simulation models as cloud-based web-services. To provide developers with a robust SOA environment,CSIP incorporates existing USDA infrastructure componentsincluding OMS3 and soil, management, climate, and other data-bases to support environmental modeling within both managedprivate clouds and public clouds (e.g., Amazon EC2 or Rackspace). Aprimary research goal for CSIP development is to gain experienceand implement sustainable strategies for model and data servicesin a cloud environment. The use of OMS3-based annotation inter-faces under CSIP accelerates the migration of legacy environmentalmodels for resultant cloud-based deployment.

Initial CSIP research has developed a web-service implementa-tion of the RUSLE2 (Revised Universal Soil Loss Equation Version 2,Foster et al., 2001) model for estimating sediment production onupland areas. This effort is in support of the USDA-NRCS Conser-vation Delivery Streamlining Initiative program in collaborationwith the USDA-ARS. CSIP is currently leveraged to run field- andwatershed-scale models in a scalable compute cloud environmentto assist the USDA Conservation Effects Assessment Program(Duriancik et al., 2008). RUSLE2 has historically been used asa Windows!-based desktop application to guide conservationplanning and inventory erosion rates over large areas. The modelprovides a reusable computational engine that can be used withouta user interface for model runs in other applications. RUSLE2’s

computational engine was integrated into an OMS3 model tosupport efficient execution and was enhanced using innovativenon-relational database approaches. The resulting RUSLE2/OMS3erosion component was embedded into a RESTful web service(Richardson and Ruby, 2007) for input data management; dataretrieval for soils, climate and management records; data conver-sion; and data caching. A single server manages access to cloud-based compute nodes. RUSLE2 model tests on the order of 100Kþmodel runs has been completed using hundreds of cloud nodes toverify the utility of a cloud-based deployment (Lloyd et al., 2011b).

To demonstrate cloud-based support for environmentalmodeling, a prototype application was developed to showcaserunning the RUSLE2 model under CSIP from an Android! mobiledevice (Fig. 5). The interactive workflow shows the parameteriza-tion of RUSLE2 erosion transects by accepting manual input orusing USGS elevation services. Mobilemapping features are utilizedto visualize location information available via global positioningsystem, transect direction, or latitude/longitude information. CSIP-based geospatial databases are queried to determine location-specific land use/land cover management options. Model runs areperformed using CSIP supported by cloud compute node(s). Uponmodel completion, the mobile device displays erosion values forthe given input parameters.

CSIP development remains ongoing. The flexibility offered byOMS3 components using the annotation-based integration methodwas essential for a RESTful web services development. RESTfulservice definition is enabled using web-service specific annotationson the OMS3 modeling components.

4.2. Water supply forecasting and watershed modeling

There are several operational and research-focused OMS3modelapplications to date. The National Water and Climate Center of theUSDA-NRCS is augmenting seasonal, regression-equation basedwater supply forecasts with shorter-term forecasts based on the useof distributed-parameter, physical process hydrologic models andan Ensemble Streamflow Prediction (ESP) methodology. Theprimary ESP model base (Leavesley et al., 2010) is built using OMS3and the PRMS hydrological watershed model. The model base willbe used to address a wide variety of water-user requests for moreinformation on the volume and timing of water availability and toimprove water supply forecast accuracy. The PRMS/ESP method-ology is a modified version of the ESP procedure developed by theNational Weather Service (Day, 1985) which uses historical orsynthesizedmeteorological data as an analog for the futurewith thetimeseries data used as model input to simulate future stream flow.

A visualization tool running under OMS3 is available forvisual display of user-selected ESP output traces. The toolperforms a frequency analysis on the peaks and/or volumes ofthe simulated hydrograph traces and displays a list of all thehistoric years used with their associated probability of exceed-ance. Different options are available in applying frequencyanalysis. One assumes that all years in the historic database havean equal likelihood of occurrence. Alternative schemes forweighting user-defined periods, based on user assumptions ora priori information, are also being investigated. El Niño, La Niña,and Pacific Decadal Oscillation (PDO) periods have been identi-fied in the ESP procedure, and these can be sorted and extractedseparately for analysis. The PRMS/ESP tool running under OMS3will provide timely forecasts for use by the agriculturalcommunity in the western United States where snowmelt isa major source of water supply.

Another modeling application currently being developed underthe OMS3 framework is the component-oriented AgES-W (Agro-Ecosystem-Watershed) model. AgES-W is a fully distributedFig. 4. Cloud Services Innovation Platform (CSIP) software architecture.

O. David et al. / Environmental Modelling & Software xxx (2012) 1e13 9

Please cite this article in press as: David, O., et al., A software engineering perspective on environmentalmodeling framework design: The ObjectModeling System, Environmental Modelling & Software (2012), doi:10.1016/j.envsoft.2012.03.006

CSIP

Rigon et al.

David et al., 2012

!26simulation of water quantity and quality in large watersheds(Ascough et al., 2010). AgES-W consists of Java-based simulationcomponents (80þ representing interception; snow processes; soilwater balance; nutrient (nitrogen and phosphorus) cycling; erosion;lateral flow and groundwater movement; and runoff concentration,flood, and chemical routing in channels) from the J2K-S (Krauseet al., 2009), SWAT, RZWQM2, and WEPP models. AgES-W simu-lates conjunctive stream flow and groundwater interaction, carriedout by HRUs which are connected by a lateral routing scheme tosimulate lateral water transport processes. This permits fullydistributed hydrological modeling of river basins. AgES-W perfor-mance for stream flow prediction was evaluated recently (Ascoughet al., in press) for the Cedar Creek Watershed in northeasternIndiana, USA, one of 14 benchmark watersheds in the USDA-ARSConservation Effects Assessment Project watershed assessmentstudy. Future plans are to enhance AgES-W for: 1) diverse croppingsystem responses to water deficits, 2) model uncertainty analysesand scaling, and 3) plant responses to atmospheric CO2. New OMS3tools currently under development to facilitate AgES-W applicationinclude HRU delineation, new sensitivity/uncertainty analysismethods and spatial visualization tools, and web-based cloudcomputing as described in the previous section.

5. Summary and conclusions

Environmental modeling frameworks streamline and accel-erate the model development and implementation process;however an initial learning curve for EMF-based modeling alwaysexists. Resource and in-kind institutional support is important forthe acceptance of an EMF, but it is up to the modeler to adopt anEMF so that it becomes an integral part of the model developmentworkflow. Apart from social and cultural barriers, EMF developersare further challenged technically to develop frameworks whichare less cumbersome for modelers to adopt. Web service anddatabase framework projects outside the modeling communityhave demonstrated that model developers will adopt a softwaredevelopment framework if it is easily understood, enablesseamless integration of existing codebases and workflows, anddoes not invalidate existing institutional software developmentpractices.

For the above reasons, designing EMFs and associated program-ming interfaces is an extremely challenging task. Numerous

modeling frameworks are currently under development worldwidewith the primary purpose of integrating existing and future envi-ronmental models into common, inter-operable, and flexiblesystems. One such framework, the ObjectModeling SystemVersion 3(OMS3), represents a persuasive choice for adoptionwith its inherentnon-invasive and scalable implementation. OMS3 developmentleverages successful framework designs and software engineeringprinciples originating from various general purpose and web-basedapplication frameworks. In OMS3, the internal complexity of theframework has been reduced by adopting a lightweight design,thereby resulting in a less steep learning curve as there are fewercomplex technical details for the model developer to absorb. Paral-lelism inOMS3 is achieved usingmulti-threadingonmulti-core CPUsand research is ongoing to further extend the scalability of OMS3 byadopting MapReduce-based large-scale distributed computingenvironments (Dean and Ghemawat, 2004) such as Hadoop!. OMS3can be considered a non-invasive modeling framework forcomponent-based model and simulation development on multipleplatforms. As shown by Lloyd et al. (2011a), the straightforwardcomponent integration structure allows rapid implementation ofnew models and an easier adaptation of existing models andcomponents. Other studies have shown that this approach leads tomodelswith less overhead and amore intuitive design. Byembracingthe use of non-intrusive language annotations for modeling meta-data specification and framework integration in favor of traditionalAPIs, OMS3-basedmodels keep their identity outside of themodelingframework. Annotations enable multi-purposing of components,which is difficult to accomplish with a traditional API design. InOMS3, annotations provide component connectivity, data trans-formation, unit conversion, and automated generated of modeldocumentation. In addition to the Java programming language, theannotation-based approach for component integration in OMS3 isalso supported for programming languages such as FORTRAN, C,Cþþ, and C#.

OMS3 introduces an extensible and lightweight layer for simu-lation description that is expressed as a Simulation DSL based onthe Groovy framework. DSL elements are simple to define and usefor basic model applications, or for more complex setups forparameter estimation, sensitivity/uncertainty analysis, etc. The useof DSLs for “programmable” configuration eliminates coreprogramming language “noise” and is efficacious for many differenttypes of modeling applications (e.g., distributed watershed

Fig. 5. CSIP/OMS3-based mobile RUSLE2 erosion model application.

O. David et al. / Environmental Modelling & Software xxx (2012) 1e1310

Please cite this article in press as: David, O., et al., A software engineering perspective on environmentalmodeling framework design: The ObjectModeling System, Environmental Modelling & Software (2012), doi:10.1016/j.envsoft.2012.03.006

CSIP

Rigon et al.

David et al., 2012

!27

Other companions

Rigon et al.

!28

http://geoframe.blogspot.com

Rigon et al.

The JGrass-NewAGE system essentials Hydrology

Arpae, Parma, 17 Maggio 2017

Giu

sep

pe

Pen

on

e

Rigon R, Formetta G. Bancheri M., Serafin F., Abera W.

!30

Kriging

• Ordinary Kriging and detrended kriging and their local versions: results are in form of raster maps or shapefiles for selected points

Based on the in situ data, it selects the best variogram (VGM) model, without any human decision, and optimises VGM parameters automatically at each time steps. Selection of VGM model is NOT efficient (so far).

What is there

Rigon et al.

Formetta, 2013

!31

• Separate rain from snow based on temperature: results are in form of raster maps or shapefiles for selected points

It can be used conjointly with calibrators and satellite (e.g. MODIS) data to obtain local estimates of the parameters.

RainSnow

What is there

Rigon et al.

Formetta et al. 2014

!32

• Implements degree-day, Casorzi-Dalla Fontana and Hocks methods: needs radiation components. Results are in form of raster maps or shapefiles for selected points

Snow

What is there

Rigon et al.

Formetta et al. 2014

!33

• Priestley Taylor, FAO and Penman-Monteith versions.

Various strategies were adopted to calibrate parameters. Only PT has been throughly tested and applied.

ET

What is there

Rigon et al.

Formetta, 2013

!34

Adige

• Implements Hymod and separation of basin area in sub-catchments numbered according to a modification of the Pfastetter algorithm.

Probably next version needs to be split apart into two or three components.

What is there

Rigon et al.

Formetta et al., 2011

!35

LWRB

SWRB

• Shortwave and longwave radiation estimation. Contains algorithms for estimating shadows according to the geometry of complex terrain. They also have parameterisation for cloud cover.

What is there

Rigon et al.

Formetta et al., 2013 Formetta et al., 2016

!36

LUCA

Particle Swarm

• Calibration tools. The first implements classic shuffle-complex evolution tools. They are part of OMS core.

What is there

Rigon et al.

David et al., 2012

!37

deSaintVenant

• Integration of de Saint-Venant 1D equation (part of Jgrasstools)

What is there

Rigon et al.

http://abouthydrology.blogspot.it/search/label/de%20Saint-Venant%20equation

!38

A - AGEs

To be checked

B- JGrass-NewAGE (https://github.com/geoframecomponents)

[Adige] BP- Backward probabilitiesClearness IndexETFP -Forward probabilities[Kriging] NetRadiationLWRB -RainSnowSWB (Simple Water Budget) SWRBSnow

C - JGrassTools (http://moovida.github.io/jgrasstools/)

More than 50 components

An index

Rigon et al.

!39

D - OMS (https://alm.engr.colostate.edu)

LUCAParticle Swarm

And the whole infrastructure for running them all

An index

Rigon et al.

The JGrass-NewAGE system essentials Case studies and Use cases

Arpae, Parma, 17 Maggio 2017

Giu

sep

pe

Pen

on

e

Abera W., Formetta G., Bancheri M., Serafin F., Abera W., Rigon R.

!41

CHAPTER 4. ESTIMATING BASIN WATER BUDGETS INPUTS WITHJGRASS-NEWAGE

water budget equation over an appropriate control volume, k:

(4.1)@Sk(t)@t

= Jk(t)+X

iQki(t)°ETk(t)°Qk(t)

for an appropriate set of elementary control volumes connected together. In Eq.(5.1),S [L3] represents the total water storage of the basin, J [L3 T°1], ET [L3 T°1], and Q[L3 T°1] are precipitation, evapotranspiration, and runoff (surface and groundwater)respectively. The Qis represent input fluxes, of the same nature of Q, coming fromadjacent control volumes.

ab

Figure 4.1: The location of the Posina basin in the Northeast of Italy (a) and DEM elava-tion, location of rain gauges and hydrometer stations, subbasin-channel link partitionsused for this modelling (b).

It is clear that Eq.(5.1) is governed by two types of terms, which can be easily identi-fied as “inputs" and “outputs". The outputs are certainly evapotranspiration, ET, anddischarges, Q, including the Qis, because they come from the assembly of control volumes.The inputs are J(t), but this term has to be split into rainfall and snowfall. Moreover,other inputs are ancillary to the estimation of outputs, in particular temperature, T andradiation Rn. Another input of the equation is the definition of the domain of integrationand its“granularity", i.e. its partition into elements for which a singe value of the statevariables is produced.

In this paper we discuss the estimation of all of these input quantities, with thescope to obtain a methodology that is generally applicable, following and expanding

56

Posina

A small (114 km2) basin in Vicenza province, flowing into the Brenta river

Abera et al.

A small basin

Abera, 2016

!42

CHAPTER 4. ESTIMATING BASIN WATER BUDGETS INPUTS WITHJGRASS-NEWAGE

method; Isaaks et al., 1989), based on removing one data point at a time and performingthe interpolation for the location of the removed point using the remaining meteo-stations.Finally, for this paper, kriging is used to generate time series of meterological forcingsfor the centroid of each HRU. These forcings, for the purposes of this paper, are keptconstant over the whole HRU area.

Figure 4.3: The Spatial interpolation component of the NewAge system (SI-NewAge).The figure shows how different components are connected together, here the variogram(semivariogram) component solves for the spatial structure of measured data in theform of an experimental variogram. The particle swarm optimization algorithm usesthe experimental variogram to identify the best theoretical semivariogram and optimalparameter sets for each time step. Lastly, Kriging uses the best semivariogram modeland optimal model parameters to estimate the meteorological data at the interpolationpoint or as a raster for a given basin.

In order to understand the effects of the theoretical semivariogram model on krigingand to compare the different kriging methods performances, we applied the following pro-cedures. Firstly, we select a single kriging type (for instance OK) and fit the experimentalsemivariogram with a single theoretical semivariogram (for instance, exponential) andestimate the best semivariogram parameters. Secondly, we perform a cross-validation foreach station, computing estimated time-series forcing values for each (removed) station.Thirdly, measured and estimated time-series forcing values are compared with GOFindices (appendix ??). Lastly, the GOF indices values calculated from 18 years of hourly

64

Calibration of Kriging parameters

Abera et al.

Schemes of work

Abera, 2016

!43

4.3. METHODOLOGY OF INPUTS ANALYSIS

(4.5) Ωrank = 1°6.

Pnk=1 D2

kn(n2 °1)

where D is the difference between the rank of the MODIS data (FSC or snow albedo)and snowfall, Js, data at the K th pair, and n is the number of observations. The higher thevalue of Ωrank, the higher the correlation between Js and snow albedo. Those parametersproducing the highest Ωrank are used to model the hourly time steps of snowfall for eachHRU.

The derivation of snow separation parameters for each HRU is possible, however, asis pertinent to the overall analysis of other components of the study, single, global andoptimized values of Eq.(4.3) parameters are derived.

Figure 4.4: The Snow separation component, outlining how the MODIS snow productsare used to calibrate the spatial snow accumulation ( Eq. 4.3). The dashed line shows theiterative (calibration) process to optimize the equation. Due to the time step differencesbetween MODIS and the separation model output, the manual calibration is preferredin this case.

4.3.4 Net Radiation

Net radiation is necessary for evapotranspiration estimation and for snow modelling. Itderives from the local difference between downwelling radiation and upwelling radiation,

67

Calibration of snow-rainfall separation

Abera et al.

Schemes of work

Abera, 2016

!44

CHAPTER 5. ESTIMATING WATER BUDGET MODELLING OUTPUTS ANDSTORAGE COMPONENT

parameters of the model can be interpreted as the mean travel time in each of the surfaceand subsurface compartments of the hydrological cycle.

The NewAge Hymod component is applied to any HRU, in which the basin is subdi-vided and the total watershed discharge is the sum of the contribution of the HRUs. Thissum can include (or not include) the delay due to routing from the HRUs outlet to thebasin outlet, but in this application we excluded it because at these scales (of around tenkilometers) travel time in channels is irrelevant (D’Odorico and Rigon, 2003). Eventuallythe Hymod component provides an estimate of the discharge at each link of the rivernetwork of the watershed, downstream to the HRUs.

ADIGE

Figure 5.2: The HYmod component of NewAge system and its input providing compo-nents. It shows how different components are connected, here kriging, SWE, ETP, andcalibration component connected with Adige to solve the runoff at high spatial andtemporal resolution. The detail discussion about each component can be referred at itsrespective section.

The first part of the simulation analysis is to evaluate the effects of four precipitationdata set generated using four krigings on the runoff calibration and modelling results.So HYMOD parameters are calibrated for all the four precipitation data sets for fiveyears (1994-1999), using LUCA as optimization tool. The simulation from 2000-2012 is

90

Calibration of the overall system

Abera et al.

Schemes of work

Abera, 2016

!45

CHAPTER 5. ESTIMATING WATER BUDGET MODELLING OUTPUTS ANDSTORAGE COMPONENT

0

1000

2000

3000

Prainfall

Psnow

Pre

cipi, J (

mm

)

0

1000

2000

94

/5

95

/6

96

/7

97

/8

98

/9

99

/00

00

/01

01

/02

02

/03

03

/04

04

/05

05

/06

06

/07

07

/08

08

/09

09

/10

10

/11

11

/12

Q

AET

S

Wate

r co

mponents

, A

ET, S (

mm

)

Hydrological years

Figure 5.11: Water budget components of the basin and its annual variabilities from1994/95 to 2011/2012. It shows the relative share (the size of the bars) of the threecomponents (Q, ET and S) of the total available water J.

104

Annual budget

Abera et al.

The idea is that JGrass-NewAGE obtain water budgetsA

bera

et a

l, in

pre

para

tion,

201

6b

!46

CHAPTER 5. ESTIMATING WATER BUDGET MODELLING OUTPUTS ANDSTORAGE COMPONENT

This could have been deduced from the data alone, However, seeing it with the otherbudget components enlighten the complexity of the interactions actually in place.

0

100

200

300

400

500

01

-201

2

02

-201

2

03

-201

2

04

-201

2

05

-201

2

06

-201

2

07

-201

2

08

-201

2

09

-201

2

10

-201

2

11

-201

2

12

-201

2

Date(month)

Q,E

T,S

(mm

/mon

th)

Q

ET

S

0

100

200

300

J (m

m/m

onth

)

Figure 5.12: The same as figure 5.11, but monthly variability for the year 2012.

106

Monthly budget (temporal)

Abera et al.

The idea is that JGrass-NewAGE obtain water budgetsA

bera

et a

l, in

pre

para

tion,

201

6b

!47

5.4. RESULTS AND DISCUSSIONS

J

80 120 160 200

Q

40 80 160

ET

20 40 60

S

Jan

Apr

Jul

Oct

−150 −100 −50 0 50

Figure 5.13: The spatial variability of the long term mean monthly water budget com-ponents (J, ET, Q, S). For reason of visibility, the color scale is for each componentseparately.

107

Monthly budget (spatial)

Abera et al.

The idea is that JGrass-NewAGE obtain water budgetsA

bera

et a

l, in

pre

para

tion,

201

6b

!48

Events

Abera et al.

But events are equally likely well reproducedA

bera

et a

l, in

pre

para

tion,

201

6b

!49

6.1. INTRODUCTION

10

20

30 40 50Long

Lat

a

8

9

10

11

12

13

36 38 40Long

Lat

1000

2000

3000

4000Elevation(m)

Lat

Station

Lake Tana

b

Figure 6.1: The geographic location of Upper Blue Nile basin in the Nile basin (a) anddigitale elevation model of the basin (b). The points in figure b are the meteorologicalstations used for this study.

Several validation studies of SREs have been conducted in the Ethiopian UBN basin(Dinku et al., 2007, 2008; Haile et al., 2013; Gebremichael et al., 2014; Worqlul et al.,2014; Romilly and Gebremichael, 2011; Hirpa et al., 2010; Habib et al., 2012). Forinstance, two comparative studies by Dinku et al. (2007) and Dinku et al. (2008) on hightemporal (less than and equal to 10 days) and spatial (less than or equal to 10) resolutionproducts shows that CMORPH, TAMSAT (Grimes et al., 1999) and TRMM 3B42 (thegauge-corrected version of TMPA products, Huffman et al. (2007)) are three SREs withgood accuracy and potentially useful for hydrological applications in the region. Dinkuet al. (2008) reported that CMORPH works better in Ethiopia than other regions ofAfrica, while Haile et al. (2013), studying the accuracy of CMORPH over a subbasin ofUBN basin for three months, found poor accuracy with respect to other regions. Morerecently, Gebremichael et al. (2014), by designing experimental rain gauges for twosummer seasons in two experimental locations (one in the lowlands and one in thehighlands) of the UBN basin, examined the accuracy of three high-resolution satelliterainfall products (CMORPH, TRMM 3B42RT - the real-time version of TMPA - andTRMM 3B42). Regarding the relationships between SREs goodness-of-fit values andtopography (particularly elevation) of the experimental sites, SREs overestimate themean rainfall rate in the lowlands and, vice versa, underestimate at the highland site.

113

Blue Nile (175000 Km2)

Abera et al.

Larger riversA

bera

et a

l, in

pre

para

tion,

201

6c

!50

CHAPTER 6. EVALUATION OF DIFFERENT SRES AND BIAS CORRECTION

In general, the effect of data length on BIAS is very small, and it is valid for all theSREs (figure 6.3, third row). For instance, the BIAS for 3B42V7 decreases from 4% for 1year evaluation to -4% in 10 years, the same level of BIAS but opposite sign. A similarslight decline in BIAS is shown for CMORPH (frm -66% to -72%) when the number ofyears in the analysis increases. The comparison of the five products using BIAS is notconsistent with the products comparison using r and RMSE (figure 6.3, the third row).For instance, SM2R-CCI (0.001) has the lowest BIAS, followed by 3B42V7 (-0.042) andCFSR (-0.06). The low BIAS of SM2R-CCI has to be attributed to the use of 3B42V7for the calibration of the parameter values of the SM2RAIN algorithm. Note that whileCMORPH is better in estimating ground-gauge rainfall using the two previous statistics(i.e., r and RMSE), it is underestimating by 72%, thus being the most biased product ofthe five SREs. This could be because CMORPH is only based on satellite products, andnot corrected using ground data as 3B42V7. TAMSAT, on average, is underestimatingrainfall by 30%.

Correlation

RMSE

BIAS

3B42V7 CMORPH CFSR SM2R-CCI TAMSAT

8

9

10

11

12

13La

tCorrelation

<0.2

(0.2,0.3]

(0.3,0.4]

(0.4,0.5]

(0.5,0.6]

(0.6,0.7]

8

9

10

11

12

13

Lat

RMSE(mm/day)

[4, 6]

(6, 8]

(8, 10]

(10, 12]

(12, 14]

>14

8

9

10

11

12

13

36 38 40 36 38 40 36 38 40 36 38 40 36 38 40Long

Lat

BIAS

(-0.9,-0.6]

(-0.6,-0.3]

(-0.3,-0.1]

(-0.1,0.1]

(0.1,0.3]

(0.3,0.6]

(0.6,1.4]

Figure 6.4: The spatial distribution of GOF values for different SREs: correlation coeffi-cient (first row), RMSE (second row) and Bias (third row).

The spatial distribution of the the three GOF values (r, RMSE, BIAS) are presentedin figure 6.4. Overall the distribution of the statistics can depict a spatial pattern, i.e., thecorrelations in the eastern and northeastern part of the basin are higher than westernand southwestern part. Similar pattern can be inferred from the RMSE and BIASstatistics that are smaller in the eastern part (the highlands), while they are higher in

124

Satellites products comparison

Abera et al.

Approached with satellite dataA

bera

et a

l, 20

16a

!51

6.5. RESULTS AND DISCUSSIONS

A.Mehal Meda B.Debre Markos C.Assosa

0

1000

2000

3000

0 100 200 300 0 100 200 300 0 100 200 300

SREs

Gauge observations

CFSR

CMORPH

SM2R-CCI

TAMSAT

3B42V7

Mean C

um

ula

tive r

ain

fall

(mm

)

Days of year

Mehal_Meda Debre_Markos Assosa

Figure 6.6: Annual mean cumulative rainfall estimations based on five SREs and gaugesdata.

these two kinds of SREs (e.g., SM2R-CCI and CMORPH or 3B42V7 or TAMSAT).Among the five SREs, TAMSAT has the highest detection capacity for lowest rainfall

intensities (91%). For all classes, TAMSAT has the highest missing rate and the highestrecorded is for the 0.1-2 mm observed rainfall class (54%), while the systematic biasfor all the classes is relatively low (figure 6.5e). The SREs detection capacity is furtherevaluated by the overall accuracy capacity, and the comparison is shown in figure 6.5f.The result confirms the confusion matrix analysis.

The time series rainfall summary analysis is useful for comparative evaluation, butdoes not provide insight into the aggregate effects of using different SREs on waterresource modelling. Figure 6.6 shows the comparison of long term (2003-2012, 10 years),mean cumulative rainfall for different SREs and measured data. A sample of threestations systematically selected to represent different ranges of elevation and spatiallocation is used in the analysis. These are Mehal Meda, Debre Markos, and Assosawhich are located at high (3084 meters), medium (2446 meters) and low (1600 meters)elevations, respectively. The spatial location of the three stations is shown in the mapsplotted in figure 6.6. Four comments can be drawn:

1. Based on the three stations, the observed long term annual rainfall shows that theeffect of elevation is masked by the rainfall climatological regime difference (Mel-

127

Big Bias

Abera et al.

Which are not always goodA

bera

et a

l, 20

16a

!52

7.4. CALIBRATION AND VALIDATION APPROACH

GRACE is a mission based on two twin satellites that measures spatiotemporal variationsof water storage that is derived from a continuous observation of the gravity field. Atthis scale, however, GRACE can still be used for constraining and validating data to themodelling solutions. Here, the performance of our modelling approach to close the waterbudget i.e. estimating storage following the characterization of all the terms, is assessedusing the GRACE estimation at the basin scale. Since the other fluxes are modeled asfunction of basin water storage, for instance Q and ET, good estimation of water storageof a model has inference to its reasonable computation of other fluxes as well (Döll et al.,2014). GRACE data is an extraordinary resource to assess the over all performance ofthe simulation, at least at the basin scale.

8

9

10

11

12

35 36 37 38 39 40long

lat

3.0

3.5

4.0

4.5

5.0Precip(mm/day)

8

9

10

11

12

35 36 37 38 39 40long

lat

1000

1200

1400

1600

1800

Precip(mm/year)a b

Figure 7.4: The spatial distribution of daily mean (a) and annual mean rainfall estimatedfrom long term data (1994-2009).

7.4 Calibration and validation approach

The precipitation data is error corrected based on the in situ observation. The Adigerainfall-runoff component, i.e HYMOD model parameter, are calibrated to fit the observeddischarge during the six years of calibration period (1994-1999) at daily time step. Basedon the approach described ET estimation, the ADIGE component is also used to calibratethe PT Æ. The simulation for each hydrological component is verified using the availablein-situ or remote sensing data as follows:

145

Final rainfall estimates

Abera et al.

but can be correctedA

bera

et a

l, 20

16a

!53

7.3. METHODOLOGY

We divide the UBN basin into 402 subbasins and channel links as shown in figure 7.2.This spatial partitioning may not be the finest scale possible, however, considering thesize of the basin, it can be considered an acceptable compromise to capture the waterbudget spatial variability.

ADIGE: Rainfall-runoff

Figure 7.3: Workflow with a list of NewAge components (in white), and remote sensingdata processing parts (gray shaded, not yet included in JGrass-NewAGE but performedwith R tools) used to derive the water budget of UBN. It does not include the componentsused for the validation and verification processes.

7.3.2.1 Precipitation J(t)

Regards to the input term of Eq. 7.1 (J(t)), the spatio-temporal precipitation, it is quan-tified based on RS-based approaches (chapter 6). Different satellite rainfall estimates(SREs) available for varied accuracy and purposes. The use of SREs and lists productsthat can be used in hydrological applications can be found elsewhere in literature (Honget al., 2006; Bellerby, 2007; Huffman et al., 2007; Kummerow et al., 1998; Joyce et al.,2004; Sorooshian et al., 2000; Brocca et al., 2014). Regardless of the recent advancementof rainfall retrieval algorithm, SREs are still subjected to significant uncertainty due tovarious factors including sensor problem, infrequent satellite overpasses, large spatio-temporal scale, and retrieval algorithm (Hong et al., 2006; AghaKouchak et al., 2009;Hossain et al., 2006).

141

The Modelling Solutioncalibration phase

Abera et al.

Schemes of workA

bera

et a

l, in

pre

para

tion,

201

6c

!54

Discharges

Abera et al.

At daily time scaleA

bera

et a

l, in

pre

para

tion,

201

6c

!55

Abera et al.

ET (spatial)A

bera

et a

l, in

pre

para

tion,

201

6c

!56

Abera et al.

The water budget (spatial)A

bera

et a

l, in

pre

para

tion,

201

6c

!57

CHAPTER 7. WATER BUDGET MODELLING OF UPPER BLUE NILE BASIN USINGJGRASS-NEWAGE MODEL SYSTEM AND SATELLITE DATA

0

100

200Pre

cip[m

m/m

onth

]

−100

0

100

01 02 03 04 05 06 07 08 09 10 11 12Months

Fluxe

s(Q

,ET,S

)[m

m/m

onth

]

ET

Q

S

Figure 7.16: Basin scale long term monthly mean Water budget components based onestimates from 1994 to 2009. It shows the relative share of the three components (Q, ETand S) of the total available water J.

160Abera et al.

The water budget (temporal)A

bera

et a

l, in

pre

para

tion,

201

6c

!58

7.5. RESULTS AND DISCUSSIONS

capability to reproduce other components well,as it is the residual terms to balance theflux dynamics.

The spatial distribution of NewAge ds/dt and GRACE based TWSC for four months(January, April, July, and October) of 2005 is shown at figure 7.12. The comparison isbased on the NewAge modelling at subbasin scale, and GRACE grid resolution of 10. Dueto the possible high leakage error introduced at high spatial resolution (Swenson andWahr, 2006), statistical comparison at subbasin level is not performed. However, focusingon maps of the sample months, some level of similar spatial and temporal pattern isrevealed (figure 7.12).

−100

0

100

200

2004 2005 2006 2007 2008 2009 2010Date

TW

SC

(mm

/month

)

NewAge

GRACE

Correlation = 0.84

Figure 7.11: Comparison between basin scale NewAge ds/dt and GRACE TWSC from2004-2009 at monthly time step.

7.5.2 Water budget closure

The water budget components (J, ET, Q, ds/dt) of 402 subbasin of UBN is simulated forduration of 1994-2009 at daily time series. Figure 7.13 is long term monthly mean waterbudget closure derived from 1994-2009. The four months (January, April, July, and Octo-

155

JGrassNewAGE—GRACE comparison

Abera et al.

Storage variationsA

bera

et a

l, in

pre

para

tion,

201

6c

!59

Adige(12000 Km2)

This is a work in progressAbera et al.

Ongoing

!60

Ongoing

Forecasting positions

arm

courtesy of Stefano Tasin

Abera et al.

!61

Abera et al.

Infos

Introduction to JGrass-NewAGE

http://abouthydrology.blogspot.it/2015/03/jgrass-newage-essentials.html

Documentation

http://geoframe.blogspot.it/

!62

Find this presentation at

http://abouthydrology.blogspot.com

Ulr

ici, 2

00

0 ?

Other material at

Questions ?

R. Rigon

http://www.slideshare.net/GEOFRAMEcafe/parma-20160517-jgrassnewage-some-about-the-state-of-art