Database Server Extension for managing and querying 4D gridded spatiotemporal data Presented at the Edinburgh e-Science Institute Nov 1-2, 2005 conference on “Spatiotemporal Databases” by Ian Barrodale Barrodale Computing Services Ltd. (BCS) http://www.barrodale.com
49
Embed
Database Server Extension for managing and querying 4D gridded spatiotemporal data
Database Server Extension for managing and querying 4D gridded spatiotemporal data. Presented at the Edinburgh e-Science Institute Nov 1-2, 2005 conference on “Spatiotemporal Databases” by Ian Barrodale Barrodale Computing Services Ltd. (BCS) http://www.barrodale.com. - PowerPoint PPT Presentation
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Database Server Extension for
managing and querying 4D gridded
spatiotemporal data
Presented at the Edinburgh e-Science Institute Nov 1-2, 2005 conference on “Spatiotemporal Databases”
by
Ian BarrodaleBarrodale Computing Services Ltd. (BCS)
http://www.barrodale.com
Barrodale Computing Services Ltd. (BCS)
“At BCS we let the actual tasks that our clients are trying to accomplish guide our solutions, rather than producing software that dictates how clients can perform their work.”
Provides customized software and R&D services to technical clients
Successfully completed 450+ software development projects since incorporation in 1978
Long-term professional staff
IBM Business Partner
Major clients include:Canada - Province of BC (Elections BC, Ministry of Forests), DND,...USA - US Navy, NOAA, IBM, SPAWAR, Univ. of Mississippi,...
Barrodale Computing Services Ltd. (BCS)
Some Application Areas:
• Defense Sciences (ASW, MCM, METOC)
• Elections/Census (Geo-Spatial Database)
• Forestry (Spatial Timber Supply Models)
• Terrain Modeling (Watershed Delineation)
• Seabed Monitoring (Gas Hydrates)
Some Skill Sets:
• Mathematical Analysis• Algorithm Development• Signal & Image Processing• Modeling & Simulation• Software Engineering• Spatial Data Analysis• Spatial Database Design• Database Server Extensions• Large Dataset Management• Graphical User Interfaces• Data Visualization• Web Map Services
Complex DataSimple Data
Query
No Query File System
RelationalDBMS
ObjectOrientedDBMS
ObjectRelational
DBMS
(Simplistic) Database Classification Matrix
File Server vs. RDBMS + File Server
Files for both metadata and data
vs. RDBMS for metadata & files for data.
File Server alone:
+ Simpler.
+ Less expensive.
± Metadata stored in data file name/directory or inside gridded data file.
RDBMS + File Server:
+ Integrity checking of metadata - integrity checking of metadata can be performed by built-in RDBMS features (check constraints, triggers, etc.).
+ Efficient access to metadata - e.g., indices can be used.
+ Easier to locate gridded data of interest - e.g., complicated queries on metadata can be performed.
− Metadata separated from gridded data - data inconsistencies possible.
Object Relational DBMS
RDBMS for metadata & fileserver for data
vs. ORDBMS (metadata & data integrated).
ORDBMS:
+ Improved concurrency - concurrent users can safely query the same gridded data.
+ Composite data types - gridded data bundled with their metadata.
+ Improved integrity - ability to reject bad gridded data before it is stored in ORDBMS.
+ Database extensibility - easy addition of data types and operations.
+ Uniform treatment of data items - SQL interface can perform complex queries based on any of these data items, e.g., metadata as well as gridded data; less need for custom 3GL programming.
+ Custom data access methods - e.g., R-tree indexes.
+ Point-in-time recovery of gridded data possible.
+ Built-in complex SQL functions for gridded data operations - e.g., aggregating, slicing, subsetting, reprojecting, subsampling, ...
BCS specializes in ORDBMS ApplicationsThe current main platforms for BCS database applications are IBM Informix Dynamic Server and PostgreSQL . Object Relational Data Base Management Systems (ORDBMSs) have four features that set them apart from traditional DBMSs:
User-defined abstract data types (ADTs). ADTs allow new data types with structures suited to particular applications to be defined.
User-defined routines (UDRs). UDRs provide the means for writing customized server functions that have much of the power and functionality expressible in C.
“SmartBLOBs”. These are disk-based objects that have the functionality of random access files. ADTs use them to store any data that does not fit into a table row.
Flexible spatial indexing. R-tree indexing for multi-dimensional data enables fast searching of particular ADTs in a table.
Example: Sum the “area” (UDR) of all “lakes” (ADT) contained (R-tree) in “British Columbia” (ADT)
Query: From a given point on a stream, what is the entire area from which drainage is
received?
“SQL” Example 1: Find the area of the watershed that is upstream from where a given road crosses a given stream.
FROM streamNetwork, roadNetworkWHERE Overlap(Intersection(Box(streamElement),
Box(roadElement)), userDefinedArea);
Note:userDefinedArea is, say, a string provided by the user.UDRs:BOX - rectangle enclosing objectINTERSECTION - common areaOVERLAP - T or FWATERSHED - calculates watershed upstream from a pointAREA - calculates area
“SQL” Example 2: Find all side-scan sonar images, that are in a user-defined area, with a heading within one degree of 128.3 degrees and with an average slant range of less than 50 m.
SELECT image
FROM sonarImageArchive
WHERE Overlap(Box(image), userDefinedArea)
AND ABS(Heading(image) - 128.3) < 1.0
AND Average(SlantRange(image)) < 50.0;
Note:userDefinedArea is, say, a string provided by the user.UDRs:SLANTRANGE - calculates slant rangeAVERAGE - calculates averageHEADING - supplies heading of objectABS - absolute valueBOX - rectangle enclosing objectOVERLAP - T or F
“SQL” Example 3: In a user-defined area, overlay on a sea floor map all “West”-looking side scan sonar images of “sandy” sea floor bottom type.
SELECT Overlay(image, map)
FROM sonarImageArchive, seaFloorMapping
WHERE Overlap(Box(image), userDefinedArea)
AND Overlap(Box(map), Box(image))
AND SlantDirection(image) = “West”
AND surfaceType = “sandy”;
Note:userDefinedArea is, say, a string provided by the user,and surfaceType is a column in seaFloorMapping.UDRs:SLANTDIRECTION - calculates slant directionBOX - rectangle enclosing objectOVERLAP - T or FOVERLAY - overlays one image on another
• Gridded data occurs in meteorology, oceanography, the life sciences, non-destructive testing, exploration for oil, natural gas, coal & diamonds,…
• These datasets range from simple, uniformly spaced grid points along a single dimension (e.g., time series) to multidimensional grids containing several types of values (e.g., 4D cubes of meteorological attributes).
• Grids have typically been stored in simple files and then manipulated by programs that operated on these files. Nowadays there is increasing justification for storing and manipulating gridded data in DBMSs: the principal advantages are their ability to (i) ensure data integrity and consistency, and (ii) provide diverse users with independent and effective query-based access to these data across multiple applications and systems.
Gridded Data in Databases
• However, implementing an efficient gridded DBMS can be very challenging, particularly when it involves Binary Large Objects (BLOBs), user-defined abstract datatypes (ADTs) that encapsulate grid data structures and attributes, and user-defined routines (UDRs) with which applications can create, manipulate and access the gridded data stored in these new datatypes.
• BCS has developed an efficient technology that supports database storage, update, and fast retrieval of gridded data; it uses BLOBs, ADTs, and UDRs.
• Our first implementation of this technology was a Grid DataBlade for IBM Informix, and then a Grid Extension for PostgreSQL; we are currently developing an analogous Grid Cartridge for use with Oracle.
Gridded Data in Databases
• is designed to handle 1D, 2D, 3D, 4D (and “5D”) grids.
• stores grids using SmartBLOBS and a (user-controlled) tiling scheme that together permit very efficient generation of products (e.g., oblique slices or 1D sticks from 4D grids).
• sometimes provides more than 50-fold increases in speed of data product generation compared to the conventional approach that does not involve tiling or SmartBLOBs.
• can store the data in, and convert it between, hundreds of mapping projections.
• can handle irregularly spaced grids in any/all grid dimensions.
• can handle the presence of multiple vector and/or scalar values.
• provides several interpolation options.
• provides for convenient database loading and extraction of grid files via one form of the commonly used NetCDF format.
• provides C, Java, and SQL application programming interfaces.
• is supplied with full user/programmer documentation.
0 1 0 0 1 0 0 0) (dim_sizes 1 1 800 800)) (rules(weight)) )') FROM images i, places_of_interest p WHERE i.imageType = 'aerialPhoto' AND overlap(grdbox(i.image),grdbox(p.loc)) AND p.name = 'New Orleans';
SQL driving the Grid Fusion
SELECT GRDFuse( GRDFuseCollect(GRDPriorityGrid(image,1.0)), '((grdspec (translation -90.4 29.57 0 0) (affine_transformation 0 0 0 .001 0 0 .001 0 0 1 0 0 1 0 0 0) (dim_sizes 1 1 800 800)) (rules(weight)) )') FROM images i, places_of_interest p WHERE i.imageType = 'aerialPhoto' AND overlap(grdbox(i.image),grdbox(p.loc)) AND p.name = 'New Orleans';
UDR to resample a set of grids into a single grid.
SQL driving the Grid Fusion
SELECT GRDFuse( GRDFuseCollect(GRDPriorityGrid(image,1.0)), '((grdspec (translation -90.4 29.57 0 0) (affine_transformation 0 0 0 .001 0 0 .001 0 0 1 0 0 1 0 0 0) (dim_sizes 1 1 800 800)) (rules(weight)) )') FROM images i, places_of_interest p WHERE i.imageType = 'aerialPhoto' AND overlap(grdbox(i.image),grdbox(p.loc)) AND p.name = 'New Orleans';
Two UDRs to build a set of transient grids, associating a floating-point value with each of these grids. This floating-point value is later used to establish the relative weight of each grid’s elements in producing the fused grid. We’ve chosen each grid to have equal weight.
SQL driving the Grid Fusion
SELECT GRDFuse( GRDFuseCollect(GRDPriorityGrid(image,1.0)), '((grdspec (translation -90.4 29.57 0 0) (affine_transformation 0 0 0 .001 0 0 .001 0 0 1 0 0 1 0 0 0) (dim_sizes 1 1 800 800)) (rules(weight)) )') FROM images i, places_of_interest p WHERE i.imageType = 'aerialPhoto' AND overlap(grdbox(i.image),grdbox(p.loc)) AND p.name = 'New Orleans';
Each source grid is resampled at the same locations, using the source images’ spatial reference system, which is a Lat-Lon grid. The fused grid’s horizontal resolution is 0.001 degrees.
SQL driving the Grid Fusion
SELECT GRDFuse( GRDFuseCollect(GRDPriorityGrid(image,1.0)), '((grdspec (translation -90.4 29.57 0 0) (affine_transformation 0 0 0 .001 0 0 .001 0 0 1 0 0 1 0 0 0) (dim_sizes 1 1 800 800)) (rules(weight)) )') FROM images i, places_of_interest p WHERE i.imageType = 'aerialPhoto' AND overlap(grdbox(i.image),grdbox(p.loc)) AND p.name = 'New Orleans';
The source of the grids is a table called “images”.
SQL driving the Grid Fusion
SELECT GRDFuse( GRDFuseCollect(GRDPriorityGrid(image,1.0)), '((grdspec (translation -90.4 29.57 0 0) (affine_transformation 0 0 0 .001 0 0 .001 0 0 1 0 0 1 0 0 0) (dim_sizes 1 1 800 800)) (rules(weight)) )') FROM images i, places_of_interest p WHERE i.imageType = 'aerialPhoto' AND overlap(grdbox(i.image),grdbox(p.loc)) AND p.name = ‘New Orleans';
We use metadata stored in another column to pick only those images derived from aerial photographs.
SQL driving the Grid Fusion
SELECT GRDFuse( GRDFuseCollect(GRDPriorityGrid(image,1.0)), '((grdspec (translation -90.4 29.57 0 0) (affine_transformation 0 0 0 .001 0 0 .001 0 0 1 0 0 1 0 0 0) (dim_sizes 1 1 800 800)) (rules(weight)) )') FROM images i, places_of_interest p WHERE i.imageType = 'aerialPhoto' AND overlap(grdbox(i.image),grdbox(p.loc)) AND p.name = 'New Orleans';
A second table called “places_of_interest” is used to include only source grids that overlap a region called “New Orleans”.