gemstat.orggemstat.org/wp-content/uploads/downloads/DRAFT_GEMStat... · Web viewGuidance to GEMStat Data Submissions General introductions Currently, the process of data submission

Guidance to GEMStat Data Submissions

General introductionsCurrently, the process of data submission to GEMStat is handled through email communication and the exchange of Excel-tables of a given structure, which contain the submitted data. These Excel-tables serve as templates that prescribe the format and sequence in which the submitted data should be entered in order to prevent ambiguity and allow for quick processing of the data.

There are three types of Excel templates, each dedicated to a single aspect of the data submission process:

1. One template for registering “assets”, such as monitoring stations and water quality analysis methods for given water quality parameters that are then referenced in data submissions.

2. One template for submitting the actual water quality data for registered monitoring stations and combinations of water quality parameters and analysis methods.

3. One template acting as an inventory of assets already registered by the individual data provider and a catalogue of all water quality parameters and analysis methods that are available within the GEMStat database.

Table 1: Excel-templates handed out to the Data Providers for submitting data to GEMStat.

Excel-Template Role

GEMS_Register_Assets.xlsTemplate for registering assets (monitoring stations and analysis methods for given water quality parameters) with GEMStat. Referred to as: Registration Template

GEMS_Data_Submission.xls

Template for submitting water quality data to GEMStat. Referred to as:Data Submission Template

GEMS_Catalogue.xlsLook-up tables of all information necessary for submitting data to GEMStat. Referred to as: GEMS Catalogue

The Excel-files contain a number of worksheets whose tabs are colored in 4 ways. The tab-colors provide a visual hint towards the role of the underlying worksheet and are grouped as shown here:

Table 2: Roles of Excel-worksheets inside the submission templates according to the color of their tabs.

Tab-Color Role

Grey Description of columns for the individual tables or submission examples.

Green Actual forms for submission of stations, analysis methods or water quality data.

Blue Inventory of assets already registered by the Data Provider within the GEMStat database.

Orange Catalogues of all relevant assets that are registered within the GEMStat database.

In order to be able to report water quality data, the Data Provider needs to register their monitoring stations and analysis methods with the GEMStat database. In return, the GEMS/Water Data Centre assigns codes for the monitoring stations and analysis methods, which are then used by the Data Provider for reporting water quality data. The workflow for registration of assets and data submission is summarized below, followed by a more detailed description.

Summarized Data Submission WorkflowIn summary, the workflow consists of six steps (see Figure 1):

1. The GEMS/Water Data Centre sends the data submission templates to the Data Provider. One of the templates, the GEMS Catalogue (GEMS_Catalogue.xls), is pre-filled with monitoring stations and analysis methods that have been already registered by the Data Provider in the past.

2. The Data Provider screens the already registered monitoring stations and analysis methods in the blue inventory worksheets of GEMS Catalogue, to check whether they are up to date and contain all stations and methods that will be part of the next data submission.

3. If some of the monitoring stations or analysis methods of the forthcoming data submission are not listed in the inventory worksheets, or if the information on some of the stations needs to be updated, these items can be filled into the green submission worksheets of the Registration Template (GEMS_Register_Assets.xls). The filled Registration Template is then sent to the GEMS/Water Data Centre.

4. The GEMS/Water Data Centre checks the submitted items from the Registration Template and registers them within the GEMStat database. In the process, the GEMS/Water Data Centre assigns codes to the newly registered monitoring stations and analysis methods.

5. The GEMS/Water Data Centre sends an updated version of the GEMS Catalogue to the Data Provider, which now also contains the newly registered assets together with their codes.

6. The Data Provider compiles a data submission through the Data Submission Template (GEMS_Data_Submission.xls), using the codes for monitoring stations, as well as the codes for combinations of water quality parameters and analysis methods from the GEMS Catalogue and sends it to the GEMS/Water Data Centre.

Following this the submitted data is subjected to quality assurance procedures and is then imported into the GEMStat database. In the next sections, the process of registering new items and submitting water quality data is explained in more detail.

Figure 1: Data submission workflow

Detailed Data Submission Workflow

Screening of InventoriesInitially, the Data Provider gets handed out a pre-filled inventory (GEMS_Catalogue.xls) of already registered monitoring stations and analysis methods that also contains a catalogue of all analysis methods and water quality parameters available inside GEMStat.

If the Data Provider did not previously participate in the GEMS/Water Programme, the provider specific inventory is empty and only the global catalogue is filled with content. In this case, the Data Provider does not need to screen his inventory and can proceed with the next step, the registration of new assets (i.e. monitoring stations and water quality parameter specific analysis methods). However, if the country or institution of the Data Provider has submitted data to GEMStat in the past, the GEMS Catalogue will contain all reported monitoring stations and analysis methods in the worksheets that are labelled with blue colored tabs.

Screening of Registered StationsThe worksheet Station Inventory contains all the registered monitoring stations with their metadata. Having 31 columns (see Table 3), this worksheet holds the most amount of information. Figure 2 shows the first ten columns of this worksheet, filled with example data. Of the remaining columns, depending on the water body type, only a fraction of is actually in use. The columns that are not applicable for a particular entry are shaded by a pattern of hatched lines ( ). This can be seen in Figure 3, where some of the water body specific columns are shown for the example stations.

Figure 2: First columns of the Station Inventory filled with example data from stations in Canada.

Figure 3: Water body specific information of the Station Inventory, found further to the right of the content that was shown in Figure 2. Columns that are not applicable to a given water body type are labeled by crosshatching.

Table 3: Content of the Station Inventory worksheet inside the GEMS Catalogue.

Column Category Field Name Requirements

A

Assigned Station Codes

Historical GEMS Number

Historical GEMS station code of registered monitoring stations, as issued by GEMS/Water prior to the re-organization of the GEMStat database. Intended as a reference for Data Providers to make transition to new GEMS Station Numbers easier.

B GEMS Station Number

GEMS station code of registered monitoring stations, as issued by GEMS/Water after the re-organization of the GEMStat database. These codes are to be used during the data submission process to link water quality values to monitoring stations.

C General Information

Local Station Number Station code according to local station network of the Data Provider.

D Station Identifier Name of station.

E Latitude Cartesian y-Coordinates according to the Coordinate Reference System in column G.

F Longitude Cartesian x-Coordinates according to the Coordinate Reference System in column G.

G Coordinate Reference System

EPSG-code of coordinate-based reference system for Latitude and Longitude coordinates (http://spatialreference.org/ref/epsg/).

H Station Narrative Description of sampling location and special conditions.

I Country CodeISO 3166-1 ALPHA-3 code of country, where the station is located in. See column B of worksheet Countries Catalogue.

J Monitoring Type Type of monitoring activity.

K Date Station Opened Date when station was established.

L Time Zone Name of time zone where the monitoring station is located in. See column B of worksheet Timezones Catalogue.

M ElevationFor surface waters: mean elevation of water surface above mean sea level.For groundwaters: ground elevation above mean sea level.

N GRDC Number ID-code of corresponding monitoring station from the Global Runoff Data Centre, if applicable.

O

Water Body Information

Water Body Type Type of water body, at which the station is located.

P Water Body Name Name of the water body, at which the station is located.

Q Main Basin Name of major river, whose basin contains the station.

R

Lake / Reservoir

Information

Max. Depth Maximum depth of lake or reservoir.

S Lake Area Area of lake or reservoir.

T Lake Volume Volume of lake or reservoir.

U Average Retention Retention time of water in lake or reservoir.

V Area of Watershed Area of watershed draining into lake or reservoir.

W

River Information

River Width Width of river at the station during average discharge conditions.

X Discharge Average river discharge at the station based on 3-5 years of data.

Y Upstream Basin Area Watershed area upstream of the station.

Z Flows Into Name of water body that the river flows into.

AA

Groundwater Information

(Well/Spring)

Area of Aquifer Area of the aquifer.

AB Depth of Impermeable Lining in Well Length of well casing.

AC Production Zone Thickness of the layer, through which water can enter the well.

AD Mean Abstraction Rate Rate of water abstraction from the well.

AE Mean Abstraction Level Water level inside well above mean sea level during a period of normal abstraction.

The Data Provider should screen the Station Inventory for correctness of the stored information and whether it is up to date. Furthermore, the Data Provider should verify whether the inventory encompasses all stations that are designated for the reporting of water quality data to GEMStat. If stations are missing or some information is outdated, the data can be added or updated with the help of the Registration Template.

Screening of Registered Analysis MethodsThe worksheet Parameter - Methods Inventory contains all the water quality parameters on which the Data Provider has already reported data in the past, together with their specific analysis methods (Table 4). The columns contain all information that is relevant for the identification of and for discerning individual parameter / method combinations. The entries contain either the exact information as reported by the Data Provider, or represent the information of an already registered parameter / method combination that was identified by the GEMS/Water Data Centre staff as equivalent to the parameter / method combination that was actually reported by the Data Provider.

The Data Provider should screen the Parameter - Methods Inventory for correctness of the stored information, specifically the method detection limits in column J, as well as the units in column L. Furthermore, the Data Provider should make sure that all parameters intended for data submission are present with their respective analysis methods. New parameter / method combinations, as well as corrections to existing entries can be introduced with the help of the Registration Template.

Table 4: Content of the Parameter - Methods Inventory worksheet inside the GEMS Catalogue.

Column Category Field Name Description

A Historical GEMSParameter Code

Historical parameter code for individual parameter-method combinations, as issued by GEMS/Water prior to the re-organization of the GEMStat database and listed in the GEMSWater Analytical Methods Dictionary 2014.Intended as a reference for Data Providers to make transition to the new system of combined Parameter Codes and Analysis Method Codes easier.

BParameter /

MethodCode

Parameter Code

GEMS code of water quality parameters, as issued by GEMS/Water after the re-organization of the GEMStat database. These codes are to be used in combination with their accompanying Analysis Method Codes during the data submission process to uniquely identify the water quality parameter, for which an individual value is reported.

C Analysis Method Code

GEMS code of analysis methods, as issued by GEMS/Water after the re-organization of the GEMStat database. See description of column B above on the relevance of these codes for the reporting of water quality values.

D

Analysis Method

Information

Parameter Long Name Full name of water quality parameter in column B.

E Analysis Method Name Full name of analysis method in column C.

F Analysis Method Type Type of analysis method.

G Analysis Method Description

Description of analysis method, according to a Standard Method, the Standard Operating Procedures (SOPs) of your laboratory or other references.

H Method Reference Reference to a national or international standard or a literature citation describing the analysis method.

I Method Number Method number, as provided within the method reference of column I, when available.

J

Metadata

Method Detection LimitThe Method Detection Limit (MDL) of the reported analytical method, which is the smallest amount of a substance detectable by a given analysis method with 99% confidence in your laboratory.

K Rounding Number of significant decimal places.

L Unit Unit of analysis result.

Figure 4: Structure of the Parameter – Methods Inventory filled with example data from Canada.

Figure 5: Structure of the Parameter – Methods Inventory (continued from Table 4) filled with example data from Canada.

After the screening of inventories is completed, the Data Provider may find that everything is in order and no new stations or method s needs to be registered. In this case, one can directly proceed with the reporting of water quality data with the help of the Data Submission Template. For this the GEMS Station Numbers in column B of the Station Inventory, as well as the Parameter Codes and Analysis Method codes from columns B and C of the Parameter - Methods Inventory are needed in or der to assign every reported value to an individual station and water quality parameter with their accompanying analysis methods. Thus, the characteristics of analysis methods are documented for each reported water quality value.

However, if new stations or analysis methods are part of the next data submission, the Data Provider needs to report those new assets using the Registration Template, as described in the next section.

Registering AssetsWhen reporting water quality data to GEMStat, every value needs to be linked to a monitoring station and a water quality parameter, as well as the analysis method that was applied to the sample in order to obtain the reported value. This linkage is done by codes that are assigned by the GEMS/Water Data Centre to each of the above mentioned assets (i.e. stations, parameters and analysis methods). The implication of this is that every new asset, for which water quality data is to be reported, first needs to be registered within GEMStat and get a code assigned to them before data can be reported.

It is strongly suggested to report new assets with the help of the Registration Template (Register_Assets.xls) by filling in the required information for monitoring stations and water quality parameters with their accompanying analysis methods into the worksheets that are labeled by green colored tabs.

Registering New Stations and Updating of Existing Station MetadataRegistering new monitoring station or changing information on already registered stations is pretty much straightforward. Just follow the steps outlined below:

1. Open the Registration Template and switch to the worksheet Station Submission Form.2. In order to register new monitoring stations, please fill in the required information into the

columns marked with an asterisk *, while leaving column A (“GEMS Station Number”) empty. This column is only to be filled if you intend to update metadata of already registered stations.

a. When entering new stations, the entries should at least provide the information as listed in Table 5.

3. If you wish to update information on stations listed in worksheet Station Inventory of the GEMS Catalogue, please proceed as follows:

a. Copy columns B to AE from the Station Inventory to the Station Submission Form worksheet.

i. Please make sure that the copied entries start with the column “GEMS Station Number” (column B) and leave out the column “Historical GEMS Number” (column A).

b. Change the information of the individual columns as you desire. i. Please leave the entries of column A (“GEMS Station Number”) unchanged.

This information is essential for the GEMS/Water Data Centre to attribute the updated information to the correct station.

ii. Be advised that the following columns allow only certain values to be entered or chosen from a drop-down list:column H (“Country Code”), column I (“Monitoring Type”), column K (“Time Zone”) and column N (“Water Body Type”)

4. New stations may be added to empty rows. In this case, please leave column A (“GEMS Station Number”) empty and shaded like so: . This column, as mentioned above, is only to be filled with a GEMS Station Number, if information on an already registered station is to be updated.

a. New station entries should at least provide information for columns that are marked with an asterisk *. Those columns are listed in Table 5.

Table 5: Description of obligatory columns for the registration of new stations within GEMStat.

Column Field Name Requirements

C Station Identifier

Name of station.

Format: Water body name - "at" Location specifiere.g. Aare River –at Brugg

D LatitudeCartesian y-Coordinates in given Coordinate Reference System (see column F).

e.g. 47.4839

E LongitudeCartesian x-Coordinates in given Coordinate Reference System (see column F).

e.g. 8.1958

FCoordinate Reference

System

Coordinate-based reference system for Latitude and Longitude coordinates.

Value: EPSG code of reference system (from www.epsg-registry.org).e.g. 4326 for WGS84

G Station Narrative

Description of sampling location and special conditions.

e.g. Measuring station lies before the confluence of the Aare with the Limmat and the Reuss rivers.

H Country Code

ISO 3166-1 ALPHA-3 code of country, where the station is located in.

Value: Insert entry from column "ISO 3166-1 Code" in worksheet "Countries Catalogue".e.g. CHE for Switzerland

K Monitoring Type

Type of monitoring activity.

Value: Baseline, Trend, Impact or Flux

LDate

Station Opened

Date when station was established.

Format (ISO 8601): YYYY-MM-DDe.g. 1978-01-01

M Time Zone

Name of time zone for time stamps of water quality data submissions.

Value: Insert entry from column "Name of Time Zone" in worksheet "Timezones Catalogue".e.g. W. Europe Standard Time

N Water Body Type

Type of water body, at which the station is located.

Value: Lake, Reservoir, Wetland, River or Groundwater

5. Additional information for a new station may be provided in the remaining columnsa. General information regarding the water body may be provided in Columns O to Q.b. Depending on the Water Body Type in column N, additional information for the

water body specific properties may be provided in the following columns:i. Lake / Reservoir: Columns R to V

ii. River: Columns W to Ziii. Groundwater: Columns AA to AE

6. After adding new stations to the template and prior to reporting water quality data for them, you should submit the Registration Template containing the newly added stations to the GEMS/Water Data Centre ([email protected]), where the new entries will be reviewed and a GEMS station Number will be assigned to them.

7. Upon completion, the GEMS/Water Data Centre will send an updated version of the GEMS Catalogue back to you, which will contain the new stations with a GEMS Station Number assigned to them.

8. Now you can use the newly assigned GEMS Station Number to report water quality data using the Data Submission Template.

Registering New Parameter / Method CombinationsThe process of registering new water quality parameters and their analysis methods is not as straightforward as the registration of stations, simply because the documentation of analysis methods and their metadata is essential for the consistent storage of water quality data.

First of all, you need to verify whether the water quality parameter part (as well as its speciation, be it the total, dissolved or any other fraction) of the new parameter / method combination is already registered within the GEMStat database. Since the GEMStat database holds over 250 individual water quality parameter species, your particular water quality parameter is probably already in there. However, we are well aware that not all possible parameters were covered, so this might very well apply in your case.

1. Open the GEMS Catalogue template and switch to the worksheet Parameter Catalogue in order to verify, whether the desired water quality parameter is already registered in the database.

2. Look up, whether the water quality parameter that is part of your new parameter / method combination is listed there.

3. To make this easier in view of the plethora of available water quality parameters, you can use the built in filters in order to narrow down the amount of visible entries:

a. Click on the filtering control buttons in the header of a column that you want to filter to open a filter dialogue (see Figure 6).

b. Remove the checkmarks from entries that you do not want to see.c. You can repeat the filtering process for additional columns in order to further narrow

down the visible entries according to multiple criteria.d. The visible entries will now be restricted to those, where the filtered columns

contain the selected values only (see Figure 7).4. If the water quality parameter of interest is not listed, please contact the GEMS/Water Data

Centre in order to add it to the list.

Figure 6: Filter dialogue fort two columns of the Parameter Catalogue. The example shows a filtering for parameters where column “Parameter Froup 3” has the entry Nitrogen, and then extended to only show rows where column “Parameter Name” has the entries Ammonia, Kjeldahl Nitrogen and Oxidized Nitrogen.

Figure 7: Result of the filter setting in Figure 6, when applied to the Parameter Catalogue.

5. Otherwise, if you have found the water quality parameter within the list, copy its Parameter Code from column F for further use.

Following this, we kindly ask you to provide a description of the applied analysis method for this particular water quality parameter in worksheet Method Submission Form of the Registration Template, in order to register your parameter / method combination:

6. Open the Registration Template, switch to the worksheet Method Submission Form and insert the previously copied Parameter Code into column A.

7. In order to register a new analysis method for this water quality parameter, please fill in the required information (see Table 6) into the remaining columns marked with an asterisk *.

8. After adding parameter / method combinations to the template and prior to reporting water quality data for them, you should submit the Registration Template containing the newly added parameter / method combinations to the GEMS/Water Data Centre ([email protected]). The new entries will then be reviewed and an Analysis Method Code will be assigned to them.

9. Upon completion, the GEMS/Water Data Centre will send an updated version of the GEMS Catalogue back to you, which will contain the new entries together with their Parameter Codes and Analysis Method Codes in worksheet Parameter - Methods Inventory.

Table 6: Description of obligatory columns for the registration of new parameter / method combinations within GEMStat.

Column Field Name Requirements

A Parameter Code

GEMS code of water quality parameter, as listed in worksheet Parameter Catalogue of the GEMS Catalogue, for which the analysis method of the current row is reported.If your parameter is not available, please contact the GEMS/Water Data Centre."

e.g. Cr-VI

BAnalysis Method Name

English name of analysis method as reported by the laboratory.

e.g. Inductively-Coupled Plasma Mass Spectrometry

CAnalysis Method

Type

Type of method used for sample analysis. Allowed values can be selected from a drop-down list, copy-pasted or entered manually.

e.g. Spectroscopy

DAnalysis Method

Description

Description of analysis method, according to a Standard Method, the Standard Operating Procedures (SOPs) of your laboratory or other references.

e.g. Pretreat a sample, add an internal standart substance, spray it into an inductively coupled plasma through the sample introducing part, measure the ionic current in each number of mass/electric charges of the chromium and internal standard substance …

GMethod

Detection Limit

The Method Detection Limit (MDL) of the reported analytical method, which is the smallest amount of a substance detectable by a given analysis method with 99% confidence in your laboratory.

e.g. 0.005

H RoundingNumber of significant decimal places for the reported values.

e.g. 3

I Unit

Unit of analysis result. Allowed values can be selected from a drop-down list, copy-pasted or entered manually.If your unit is not available, please contact the GEMS/Water Data Centre.

e.g. mg/l

10. Now you can use the newly assigned Parameter Codes and Analysis Method Codes to report water quality data using the Data Submission Template.

Data SubmissionIn order to submit water quality data to GEMStat, we strongly suggest using the Data Submission Template. This template not only gives you the option to set the Data Policy for the whole batch of submitted data, but also provides a consistent data structure which incorporates the coding system for stations and parameter / method combinations. Additionally, the template also provides some functionality for data validation and for mitigation of input errors with the help of controlled vocabularies.

Before preparing the data submission, you have the option to change the Data Policy settings for the submitted data. These settings determine how GEMS/Water will use and disseminate the data:

1. Open the Data Submission Template and switch to the worksheet Data Policy Settings.2. You will find a list of Available Data Policy Options and their Description in columns E and F,

as well as a field to Select a Data Policy Option in column B.a. The default data policy option for data submissions is “LIMITED”

3. Click on the field in the second row of column B to open a drop down list from which you can select one of the Available Data Policy Options. You may also enter the respective value by hand.

a. An error will pop up, if you enter a value that is not defined in column E.4. All the data submitted within this particular Data Submission Template will be flagged

internally with the selected Data Policy option and treated accordingly.

After setting the Data Policy option, please proceed with preparing the actual data submission:

1. Switch to the worksheet Data Submission Form.2. Enter the water quality values in column G, while also filling in the required information into

the remaining columns marked with an asterisk * for every record.a. See Table 7 for a list of obligatory column entries.b. Please make sure that each water quality value is entered into a new row.

3. Each water quality value must be assigned to a location that has previously been registered within the GEMStat database through a GEMS Station Number in column A.

4. Each water quality value must be assigned to a water quality parameter and analysis method, which both, in tandem, have previously been registered within the GEMStat database through their Parameter Code and Analysis Method Code in columns E and F.

5. If you have particular remarks on conditions or quality regarding individual water quality values, you may include them in column J (“Remarks”)

Table 7: Description of obligatory columns for submitting water quality data to GEMStat.

Column Field Name Requirements

AGEMS

Station Number

CEMS station code, as issued by the GEMS/Water Data Centre, of the monitoring location where sample has been taken.

e.g. XMP00052

B Sample Date

Date on which sample has been taken.

e.g. 2013-09-27

C Sample Time

Time on which sample has been taken.

e.g. 17:25:00

D Sample Depth

Depth below water surface, from which sample has been taken.

e.g. 0.1

E Parameter Code

GEMS code of water quality parameter, whose value is reported in current row.

e.g. Alk-Tot

FAnalysis Method

Code

GEMS code of analysis method, which has been used to obtain the value reported in current row.

e.g. Alk-T-POT-Auto

G ValueReported value of water quality parameter in current row.

e.g. 0.01

H Value FlagsSign indicating detection limits or the quality of reported value, when applicable.

e.g. L or < (indicating that value is below the Method Detection Limit)

I Integrated Sample

If the reported value is in some way integrated over space or time (mean of consecutive samples or value of a mixed sample collected over several days), please set the appropriate flag.

e.g. T14 (indicataing that value is the result of an averaged continuous measurement – or other time integration method - over 14 days)