Climate data management

1

CLIMATE DATA MANAGEMENT

By

Prof. A. Balasubramanian CENTRE FOR ADVANCED STUDIES IN EARTH SCIENCE

UNIVERSITY OF MYSORE

MYSORE-6

2

Introduction:

Climate data can provide a great deal of

information about the atmospheric environment

that impacts almost all aspects of human

endeavour.

Most of the long-term climatological analyses

depend on a number of non-climatic factors.

3

These factors include changes in:

instruments,

observing practices,

station locations,

formulae used to calculate means, and

station environment.

The whole world depends on climatic data for

present and future developments.

4

If we measure rainfall, in order for the data to

be useful for future users, we also need to

document where and how the measurements

were made.

Meteorological data users other than the

climatological community, working in fields

like agrometeorology, engineering or

aeronautics, also benefit from good data.

Meteorological data are influenced by a wide

variety of observational practices.

5

Data depend on the instrument, its exposure,

recording procedures and many other factors.

There is a need to keep a record of all these

metadata to make the best possible use of the

data.

The twentieth century saw the routine

exchange of weather data in digital form and

many meteorological and related data centres

took the opportunity to directly capture and

store these in their databases.

6

Automatic methods of collecting and processing

meteorological data was started in the late

1950s. Today, the management of climate

records requires a systematic approach that

encompasses paper records,

microfilm/microfiche records and digital

records, where the latter include image files as

well as the traditional alphanumeric

representation.

7

The Internet is already delivering greatly

improved data access capabilities and,

providing security issues are managed, we can

expect major opportunities for data managers in

the next five to ten years. STATION IDENTIFIERS AND GEOGRAPHICAL

DATA

Data can always be associated with some place.

To do so, the station has to be identified by

names and codes and to be located into the

geographical network.

8

It is also important to clearly identify when data

started to be collected and by whom.

Station Identifiers -Name: station names

usually refer to the city or village where the

data are collected .

Type of station:

Synoptical, aeronautical, agrometeorological ,

etc.

9

Geographical Data : Climate data are associated

with geographical locations. Latitude and

longitude and Elevation above Mean Sea Level.

Local Environment, Coordinates and elevation.

Local land use/land cover:

At different scales, it is recommended to keep

track of several attributes.

At the mesoscale (1 km to 30 km) it is

important to account in the metadata for:

o Proximity and size of large water surfaces

10

o Urbanized areas

o Mountain ranges.

At toposcale ("local" scale, 100 m to 2 km)

observations are influenced by:

o Terrain slope, both steepness and direction

o Forests, crops and other roughness .

o Nearby obstacles such as trees or houses (at

airports: airplanes)

o Proximity to irrigation.

11

Landuse/ landcover classification that covers

most cases.

o Artificial surfaces: continuous urban cover;

discontinuous urban cover; industrial and

commercial areas; transportation

infrastructures; harbour areas; airports; mines,

dumps and areas under construction; artificial

green areas (non agricultural).

12

o Agricultural surfaces: non irrigated crops;

irrigated crops; rice fields and other inundated

crops; grasslands; mixed crops; agricultural-

forest systems

o Natural vegetation and open areas: deciduous

forests; evergreen forests; mixed forest; shrub

vegetation; mixed shrub and forest; natural

grasslands and prairies

13

o Wetlands:

swamp areas; peat lands; marshes; inter tidal

flat areas .

o Water bodies:

rivers and other natural water courses;

artificial water courses;

lakes and lagoons;

dams; estuaries; seas and oceans

14

Type of instruments, Instrument exposure

Depending on each meteorological element,

some additional instrument features are very

important:

o Temperature and humidity: screen (type and

size) and ventilation.

Wind direction: time and method of azimuth

alignment.

15

Wind speed: response time of anemometer and

recording chain, and how these where

determined.

o Precipitation: gauge rim diameter, rim height

above ground, presence of overflow storage,

presence of a nipher screen or other airflow-

modifying feature, presence of heating or other

means to deal with solid precipitation.

Global radiation: wavelength range transmitted

by the dome.

16

Sunshine: thresholds for automatic sunshine

recorders.

Evaporation: any coverage applied to

evaporation pan.

Data recording and transmission:

When a meteorological element is measured

with an instrument, data have to be recorded

and usually transmitted to the data management

section of the organization for checking and

archival.

17

Data Conversions:

consider data conversion algorithms.

Data Processing:

It is very important to keep information on how

the data are to be processed, validated and

transmitted to the regional or central office from

every single station.

Units: All units of measurement , analysis,

processing.

18

Calculations:

Calculations other than those made on-site by

the observers, such as time averaging (daily,

monthly and so on) of elements, can also be

performed at stations or at regional and central

meteorological offices.

19

Climate data management

1. User requirements and supporting priority

needs

It is essential to take into account the needs of

the existing, and to the extent that it is

predictable, future data users.

The Climatic data management requires

awareness of the needs of the end users.

20

At present, the key demand factors for data

managers are coming from climate prediction,

climate change, agriculture and other primary

industries, health, disaster/ emergency

management, energy, natural resource

management (including water), sustainability,

urban planning and design, finance and

insurance. The quality of climate data is

greatly influenced by how well observation

networks and systems are managed.

21

The data manager will need to operate as an

effective intermediary between the observation

manager and data user. In responding to user

needs, some of the key issues to consider in

prioritizing new/additional observations are:

National social, economic and environmental

priorities;

Data-poor regions; Poorly-observed parameters;

Regions sensitive to change; and Measurements

with inadequate temporal resolution.

22

Climate Data Management Systems:

Desirable properties- Database model

Any climate database will be based on some

underlying model of the data.

Meteorological data will be accessed

‘synoptically’, and retrievals will be of the

form “get all data for some given locations or

area and for some defined and relatively short

period”.

23

By contrast, climate applications typically

involve retrieving data for one or a few stations,

but for a long period.

Consequently, one broad approach to storing

climate data involves storing (for daily data) all

data associated with a given station and day

together.

A similar approach can be taken for hourly and

monthly data.

24

Key entry capabilities :

The data entry system should be free of

annoying defects which may slow up the data

entry operator. Ideally the forms presented on

the screen will be customizable so as to

optimize the efficiency of data entry. The

system should also, as far as possible, validate

data as they are entered - catching errors and,

where possible, suggesting alternative values.

25

Electronic input options:

As mentioned above, it is desirable for a CDMS

to have the ability to represent the full content

of the relevant WMO standard message formats

– SYNOP, CLIMAT, etc).

An associated beneficial feature is the ability to

decode these message formats directly into the

climate database.

26

Scope of quality checks on observation

values:

Checks should be applied to determine the

quality of an observation.

Data extraction:

Ideally data can be retrieved both from a GUI

interface and from a command-line interface, as

appropriate.

27

Ideally, GUI retrieval facilities should be

provided for the vast majority of users, with

query language facilities used only by a small

range of knowledgeable users who have a need

to do non-standard retrievals.

Output Options:

The system should also support a wide range of

output options.

28

Options should be provided to give access to

listings of data, tabular summaries, statistical

analyses (simple and complex), and graphical

presentations.

Security issues:

The main goals of a security policy and

associated activities are to prevent loss of, or

damage to, the CDMS and to keep data

management facilities in the best possible

condition.

29

The archives and database environment must be

secured and protected against fire, humidity,

etc;

Only protected applications permitted to a small

group of people have the right to handle data

manipulations (i.e. insert, update, delete).

All changes to data tables should have an audit

trail and the controls on access to this trail

should be in place;

30

• There should be a policy of not sharing

passwords as well as not writing down

passwords anywhere.

Passwords should be changed regularly and this

applies to all users from the database

administrator to the user who handles data

manipulation applications.

The archive database system must run behind a

firewall.

31

• Backups must be made at such intervals-

daily and weekly full backups;

A recovery plan should be drawn up

Database management and monitoring :

The aim of database management is to ensure

the integrity of the database at all times, and to

ensure that the database contains all the data

and metadata needed to deliver the objectives of

the organization, both now and into the future.

32

Typical monitoring reports would include the

number and type of stations in the database, the

quantity of data in the database grouped by

stations and by observation element types.

Documentation management :

Documentation of the processes involved in

managing and using the database is essential,

both to record the design and as operational

instructions and guidelines for the managers,

users and developers of the database.

33

Metadata documentation and management :

In order that meteorological data be useful for

future users, it is essential that an adequate set

of metadata be available.

Data acquisition, entry, storage and

archiving

Data collection should be as close to the source

as possible.

34

AWSs, including stations which have some

manual observations, and partially automatic

weather stations should collect their climate

data and error messages on site and transfer

these electronically to the CDMS, possibly via

another database system.

Manually observed data should be collected and

captured on-site and transferred as soon as

possible to the CDMS.

35

Storage and archiving of hard copy records:

All paper records should be stored in a

controlled environment to avoid deterioration

and possible destruction by temperature and

humidity extremes, insects, pests, fire, flood,

accidents or vandalism.

But before archiving, the records should be

captured in microfilm or, preferably, in

electronic image form through a digital camera

or scanner.

36

Storage and archiving of digital information:

An important job of the data manager is to

estimate data storage requirements, including

estimating future growth.

In the case of having not enough mass storage

to keep all original raw data, the oldest data

could be rolled out of the database to a slow

mass storage archiving system.

37

This would generally be a tape robot system but

nowadays it is often a DVD robot system or

something similar.

Managing original records and data rescue

Data rescue is to be comprehensively covered .

Data exchange :

Exchange of data between organisations is

essential for climatology.

38

This may cover both the storage and use of data

(and metadata) among countries in the database

and the transmission of data to global and

regional data centres.

Change management issues :

The kinds of changes that need to be managed

include: Changes to observation networks and

systems; Changes to observational practices;

39

The introduction of new data types; and

Changes in the algorithms that compute derived

data.

Scalability:

Typical issues will be the need to add extra data

types, or to cope with large volumes of extra

data.

40

System Architecture and technology

Data models in use by CDMSs

The Element Model

An Element Model represents data in tables,

having, in each row, different values of one

element observed at one station at different

times.

Advantages: It is easy to add new elements; the

data model remains the same even if a new

element is added.

41

Disadvantages:

Performance for real-time applications may be

poor; many operations on the database can be

more complex than would otherwise be the

case.

The Observation Model

An Observation Model represents data in tables

having, in each row, the values of different

elements observed at one station at a given

time.

42

For example daily data could be stored in a

Daily table.

Each row would correspond to a specific station

at a specific time.

Each column of a specific row would store the

values of the different elements observed.

Advantages:

High performance for real-time applications;

optimisation of data storage.

43

Disadvantages:

Need to update the table structure if a new

element that has not been included during the

database design stage has to be added.

The Value Model

A Value Model will represent the data values in

tables having, in each row, only one value of

one element observed at one station at a

specific time. For example, daily data could be

stored in a Daily table.

44

Each row would correspond to a specific station

for a specific element and at a specific time.

Advantages:

It is easy to add new elements, the model is

adaptable to a large range of data types.

Disadvantages:

Optimization of data storage will not be done

well, so this approach is not suitable for tables

with huge amounts of data; also shares the

disadvantages of the Element model.

45

Computer hardware and software

considerations

Complete inventory and description of the

hardware and software components currently

available: computers, network, operating

systems, DBMSs, applications in use, etc.

Current telecommunication possibilities in

the country and/or the region: International

and national telecommunication lines available:

Internet, GTS, telephone, radio, etc.

46

Functional interactions: Internal, national,

and international? Drawing up functional

schemas between the different stakeholders

within a data management operation including

internal, national and international levels are

especially useful.

Operating System: Which OS?

DBMS:

Which DBMS?

Day to day operation

47

Process and responsibility:

Each process should be described and should be

under the supervision of an identified person.