Data Standards Workflow Raw data Scripts Database Store raw data in subversion to keep track of history Stored files (netcdf) accessible through the web Extract Transform Load Charts & Maps Tools and websites Provide Add meta information Script to convert raw data into netcdf OpenEarth RawData OpenEarth OPeNDAP OpenEarth Tools
42
Embed
Data Standards Workflow Raw dataScriptsDatabase Store raw data in subversion to keep track of history Stored files (netcdf) accessible through the web.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Data Standards Workflow
Raw data Scripts Database
Store raw data in subversion to
keep track of history
Stored files (netcdf)
accessible through the web
Extract Transform Load
Charts & Maps
Tools and websites
Provide
Add meta information
Script to convert raw data into
netcdf
OpenEarthRawData
OpenEarth
OPeNDAP
OpenEarthTools
Data Standards Workflow
Raw data Scripts Database
Store raw data in subversion to
keep track of history
Stored files (netcdf)
accessible through the web
Extract Transform Load
Charts & Maps
Tools and websites
Provide
Add meta information
Script to convert raw data into
netcdf
OpenEarthRawData
OpenEarth
OPeNDAP
OpenEarthTools
Transform
• Add metadata• Store in netcdf• Save script in subversion
Add metadata
• Use the inspire meta data form to store information about the dataset.• http://www.inspire-geoportal.eu/inspireEditor.htm• Click launch editor
Transform
Turn validation on
Transform – add metadata
validation
Location in subversion
micore
File identification
Transform – add metadata
History of your data.
Transform – add metadata
quality
Please fill in limitations of use.
Transform – add metadata
constraints
Store in course/Pcnumber/inspire_description.xml
Transform – add metadata
Save metadata file1. Save metadata file (local)2. Add to subversion (local)3. Commit => metadata into subversion (remote)
Transform
• Add metadata• Store in netcdf• Save script in subversion
Store in netcdf
• What’s netcdf?• Write a script to transform data into netcdf• Using CF convention
Transform
What is netcdf
• Data format defined by unidata• Data store used for coverage data and
multidimensional data• CF Metadata convention
Transform – store in netcdf - netcdf
What is netcdf
XX
ZZ
TT
YY
• An array based data structure for storing multidimensional data
• N-dimensional coordinates systems• X coordinate (e.g. longitude)• Y coordinate (e.g. latitude)• Z coordinate (e.g. altitude)• Time dimension• … other dimensions
• Variables – support for multiple variables• Temperature, humidity, pressure, salinity, etc
% read header header = fscanf(fid, '%d', 2); year = header(1); % store year in time time(i) = year; npoint = header(2); % read data data = fscanf(fid, '%d', npoint*2); data = reshape(data, [2, npoint]); % use column vectors data = data';
Read all data% preallocate all data % (time, coastward)transectseries = NaN(3, 58);coastward_distance = NaN(58, 1);time = NaN(3, 1);% open file and get file idfid = fopen('..\raw\transect.txt');i = 1;while (~feof(fid)) % read header header = fscanf(fid, '%d', 2); year = header(1); % store year in time time(i) = year; npoint = header(2); % read data data = fscanf(fid, '%d', npoint*2); data = reshape(data, [2, npoint]); % use column vectors data = data' % store data in transect series transectseries(i,:) = data(:,2); coastward_distance(:) = data(:,1); fgetl(fid); i = i + 1;end
Transform – store in netcdf - script
Create a functionfunction transect = readtransect(filename)% preallocate all data % (time, coastward)transectseries = NaN(3, 58);coastward_distance = NaN(58, 1);time = NaN(3, 1);% open file and get file idfid = fopen(filename);i = 1;while (~feof(fid)) % read header header = fscanf(fid, '%d', 2); year = header(1); % store year in time time(i) = year; npoint = header(2); % read data data = fscanf(fid, '%d', npoint*2); data = reshape(data, [2, npoint]); % use column vectors data = data'; % store data in transect series transectseries(i,:) = data(:,2); coastward_distance(:) = data(:,1); fgetl(fid); i = i + 1;endtransect = struct('series', transectseries, … 'distance', coastward_distance, 'time', time);end
• What’s netcdf?• Write a script to transform data into netcdf• Using CF convention
Transform – store in netcdf - convention
CF convention
Standard used by USGS, NOAA, Arcgis, GDAL
Climate and Forecast (CF) Conventionhttp://www.unidata.ucar.edu/software/netcdf/docs/conventions.html
Initially developed for• Climate and forecast data• Atmosphere, surface and ocean model-generated data• Also used for observational datasets• CF is the most widely used convention for geospatial netCDF
data.
Transform – store in netcdf - convention
Improve output
• Store extra attributes• Title• Author• Standard_name
Transform – store in netcdf - convention
Transform
• Add metadata• Store in netcdf• Save script in subversion
Transform – save script
Save script1. Save script (local, using matlab
https://repos.deltares.nl/repos/OpenEarthRawData/course/PCnr/scipts/)2. Add to subversion (local)3. Commit => script into subversion (remote)