Safeguarding GIS Data through Metadata Christopher Cialek & Nancy Rader Minnesota Land Management Information Center September 2004.

Safeguarding GIS Data through Metadata

Christopher Cialek

& Nancy Rader

Minnesota Land Management

Information Center

September 2004

What’s Metadata?

If you had two cans without labels, which would you eat?

Without a label, how would you know which was tuna and which was cat food?

Cat Food?

Tuna?

By the end of the Workshop . . .

Understand what metadata is; appreciate its value

Become familiar with metadata standards

Create your own metadata records

Use a search engine to find data

Know where to go for help

You will:

From: stacey stark <stark-stacey@ To: MNPLAN.MNPLANPO(rader-n) Date: 9/21/98 12:09PM Subject: Quaternary Geology layer Nancy, I am trying to learn as much as I can about the sources of the Quaternary geology layer of the MGC100 data. I do not know who or where the data was transformed / processed from the EPPL7 format – if you have any information, that would be very helpful. (log indicates user name whpa?). I am also wondering if the coverage I have might be the scanned polygon coverage before it was converted to EPPL6? The coverage name used to be qgeo - but I don't recognize the user. Would you have information? I hope I am not too much of a pain, I talked to MGS. They have never processed a polygon coverage of Quaternary geology in this area. Thank you very much. Stacey Stark GIS Specialist EPA Mid-Continent Ecology Division

From: "Tim Ogg" To: <[email protected]> Date: 8/ 3/ 00 11:53AM Subject: Re: soils documentation & status chart Hi Nancy - I have a Washington Co cover that I think came f rom the county or the SWCD. I t was residing on the computer of a former employee so I don't know exactly where it came from. The attributes on it are all wrong. Polygons are severely mislabeled all over the place. I intend to get a "good" copy f rom the county or SWCD. Where’s the best place to look? Thanks Tim

From: stacey stark <stark-stacey@ To: 'Nancy Rader' Date: Thursday, January 15, 2004 12:46 PM Subject: RE: FW: PLS QQ metadata -----Original Message----- I now know what I need to about the data. Thanks for all your help. Tracking down this info has been kind of a nightmare. After this experience I can testify to the NEED for data to have metadata along with it. It gets frustrating when you don't know what it is or where it came from. Thanks again for the help jeff

What’s Metadata?

“the information that makes data sets understandable, usable and

sharable.”

International Standards Organization

FDA Food LabelWe often use metadata without knowing it -- even a food label is an example of metadata!

What’s Metadata?

Structured format

Specific content

Necessary information

What’s Metadata?

Search metadata to find resources

in the library

Library community has developed metadata

systems to describe books

– Dublin Core

Allows you to

search by title,

author, subject…

What are Metadata Used For?

MANAGING DATABASES

COMPARING DATA SETS

FACILITATING DATA SHARING

PROVIDING TECHNICAL SPECS

FINDING DATA

Standards

Content Standard for Digital Geospatial Metadata (CSDGM)

– Established by the FGDC in 1994

– Foundation standard for the NSDI

– “Mandatory” for federal agencies

– www.fgdc.gov/metadata/contstan.html

Standards

Minnesota Geographic Metadata Guidelines (MGMG)

– Derived from the federal standard in 1998

– Simplified, but retains all required fields

– Became state standard in 1999

– www.gis.state.mn.us/stds/metadata.htm

Standards

ISO 19115 Geographic Information: Metadata

– International geospatial metadata standard

– Conceptual design approved May 2003

– Implementation schema draft under review

– Crosswalk to CSDGM: http://ogcengine.nima.mil/metadata/index.htm

Metadata Structure Examples

mandatory

mandatory, if applicable

optional

LINEAGE

Source Information

Source Citation

Source Time Periodof Content

Citation Information

Source Scale

Type of Source Media

Time Period Info

Source Currentness

Source Contribution

Source Citation Abbrev

Process Step

Process Date & Time

Source Used Citation

Process Description

Source Produced

Process ContactContact Information

CSDGM

LINEAGE

MGMG

LINEAGE

Statement

Process Step

Description

Rationale

Processor

Date & Time

Source

Description

Scale

Spatial Reference Sys

Source Citation

Source Extent

Source Step

ISO

TheMinnesotaGeographic Metadata Guidelines

The MGMG

SUBSET OF A FEDERAL STANDARD

MADE UP OF SEVEN SECTIONS

DRIVES WEB SEARCH TOOLS

USED BY OVER 100 ORGS IN MN

MN STANDARD; FGDC RECOGNIZED

Seven Sections of the Minnesota Geographic Metadata Guidelines

Minnesota State Guideline 17-1.2http://www.gis.state.mn.us/stds/metadata.htm

1Identification

2Data

Quality

3Spatial DataOrganization

4Spatial

Reference

5Entity andAttribute

6Distribution

7MetadataReference

Originator Attribute Accuracy Native Data SetEnvironment

HorizontalCoordinate Scheme

Entity and AttributeOverview

Publisher Metadata Date

Title Logical Consistency GeographicReference (Tabular)

Ellipsoid Entity and AttributeDetailed Citation

Publication Date Metadata ContactInformation

Identifier (optional) Completeness Spatial Object Types Horizontal Datum &Units

DistributorInformation

Metadata StandardInformation

Abstract Positional Accuracy Tiling Scheme Resolution Distribution Liability

Purpose Lineage Altitude Datum &Units

Transfer Format

Content Date Source Scale Depth Datum & Units Transfer Size

Currentness If Raster Ordering Instructions

Progress If Geographic Online Linkage

Maintenance andUpdate Frequency

If UTM

Spatial ExtentDescription

If State Plane

BoundingCoordinates

If County Coordinate

Keywords If User SpecifiedProjection

Constraints If Other

Contact Information

Browse GraphicInformationAssociated DataSets

Abstract

Purpose

1Identification

Originator

Title

Identifier (optional)

Content Date

Currentness

Progress

Maintenance andUpdate FrequencySpatial ExtentDescription

BoundingCoordinates

Keywords

Constraints

Contact Information


A Walk Through the Guidelines

Originator: name of organization or individual that developed the data

Title: name by which the data set is known

Abstract

Purpose

1Identification

Originator

Title


Content Date

Currentness

Progress


BoundingCoordinates

Keywords

Constraints

Contact Information


TITLE

– Too cryptic: niclcpy3

– Too general:Landuse

– Acronyms:Wisconsin DOPs

– Too detailed: Wetland Polygon Coverage Overlay for

St. Cloud (USGS Quad)

– Just right (includes theme, area, date): Minnesota Telecommunications Service

Area Boundaries, June 2002

Abstract

Purpose

1Identification

Originator

Title


Content Date

Currentness

Progress


BoundingCoordinates

Keywords

Constraints

Contact Information



Abstract: summary of what’s in the data setThis land cover data set was derived from 30

meter resolution LANDSAT Thematic Mapper

(TM) satellite imagery. Classification is divided

into 15 classes with source imagery dates

ranging from September 1991 to August 1996.

Both a raster and a vector version are

available.

Purpose: why the data set was developedLand use planning, natural resource

monitoring

Abstract

Purpose

1Identification

Originator

Title


Content Date

Currentness

Progress


BoundingCoordinates

Keywords

Constraints

Contact Information



Time Period of Content: single date that best describes when the data are current08/2004

Currentness: text describing what the Time Period date is referring to, e.g., range of dates of aerial photographyDate of source imagery (LANDSAT-5 TM,

bands 3, 4, and 5) ranges from September 20,

1991 to August 30, 1996.

Abstract

Purpose

1Identification

Originator

Title


Content Date

Currentness

Progress


BoundingCoordinates

Keywords

Constraints

Contact Information



Spatial Extent: description of the geographic area covered

(Lyon County, Minnesota)

Bounding Coordinates: the extreme north, south, east and west limits of coverage expressed in latitude and longitude values

W -95.4 E -89.5 N 49.4 S 45.5

Abstract

Purpose

1Identification

Originator

Title


Content Date

Currentness

Progress


BoundingCoordinates

Keywords

Constraints

Contact Information



Keywords: words or phrases that summarize the theme and location of the data set, together with the name of any formal list of keywords (thesaurus)

Too general: GIS, layer, surveyJust right: Feedlot, animal agriculture,hog

Constraints: any restrictions to the access or use of the data set

Access: Due to increased security measures taken after 9/11/01, this data set is no longer available online.

Use: . . . right to use these data for any internal purpose

Abstract

Purpose

1Identification

Originator

Title


Content Date

Currentness

Progress


BoundingCoordinates

Keywords

Constraints

Contact Information



Contact Information: the person who can answer questions about the content or development of the data set

Abstract

Purpose

1Identification

Originator

Title


Content Date

Currentness

Progress


BoundingCoordinates

Keywords

Constraints

Contact Information



Browse Graphic: a sample illustration of the data set

Abstract

Purpose

1Identification

Originator

Title


Content Date

Currentness

Progress


BoundingCoordinates

Keywords

Constraints

Contact Information



Associated Data Sets: information about other, related data sets that may be of interest

If you’re interested in this data set, here are others that may also interest you. NOT a list of source materials (those are described in Lineage).

For information on other air photos available for Minnesota, see www.lmic.state.mn.us/chouse/airphoto


Attribute Accuracy: qualitative or quantitative explanation of how accurately features in the data set have been described, including procedures used to assess accuracy (examples: field-checking, checkplots, frequency counts to find invalid codes)

2Data

Quality

Attribute Accuracy

Logical Consistency

Completeness

Positional Accuracy

Lineage

Source Scale


Completeness: information about selection criteria, omissions, generalization, etc.

2Data

Quality

Attribute Accuracy

Logical Consistency

Completeness

Positional Accuracy

Lineage

Source Scale

EXAMPLE: Geographic exclusion

“Data was not available for Smith Township.”


Completeness: information about selection criteria, omissions, generalization, etc.

2Data

Quality

Attribute Accuracy

Logical Consistency

Completeness

Positional Accuracy

Lineage

Source Scale

EXAMPLE: Categorical Exclusion

“Municipalities with population

under 1000 not included.”

A Walk Through the Guidelines Positional Accuracy: an explanation of what’s

known about the horizontal and vertical accuracy of the data set (can be qualitative or quantitative)

Qualitative example:

2Data

Quality

Attribute Accuracy

Logical Consistency

Completeness

Positional Accuracy

Lineage

Source Scale

Data was collected in the field and plotted on a variety of base maps. Archaeological properties visited in the past 30 years are located on USGS maps. Almost all site locations are accurate to the quarter section. Most site locations are accurate to within a quarter-quarter section. Site boundaries are poorly defined, as are site centroids.

Minnesota State Historic Preservation Office Archaeological Inventory

A Walk Through the Guidelines Positional Accuracy: an explanation of what’s

known about the horizontal and vertical accuracy of the data set

Quantitative example:

2Data

Quality

Attribute Accuracy

Logical Consistency

Completeness

Positional Accuracy

Lineage

Source Scale

Using the National Standard for Spatial Data Accuracy, this data set tested 1 foot horizontal accuracy at 95% confidence level.

City of Minneapolis (from Positional Accuracy Handbook)


Lineage: information about the sources of data used to construct the data set & processing steps applied

2Data

Quality

Attribute Accuracy

Logical Consistency

Completeness

Positional Accuracy

Lineage

Source Scale

LINEAGE RECIPE2

DataQuality

Attribute Accuracy

Logical Consistency

Completeness

Positional Accuracy

Lineage

Source Scale

1. Source Information

Data set reference

Scale

Media

Time period of content

Source contribution

Source Metadata reference

2. Processing Step

Process description

software used

organization doing the processing

Process date

3. Miscellaneous Notes

A Walk Through the Guidelines3

Spatial DataOrganization

Native Data SetEnvironmentGeographicReference (Tabular)

Spatial Object Types

Tiling Scheme

A Walk Through the Guidelines4

SpatialReference

HorizontalCoordinate SchemeEllipsoid

Horizontal Datum &Units

Resolution

Altitude Datum &UnitsDepth Datum & Units

If Raster

If Geographic

If UTM

If State Plane

If County Coordinate

If User SpecifiedProjectionIf Other


Entity & Attribute Overview: description of the information content of the data set: the features it represents (entities) and details about them (attributes). An entity might be road and the attributes that describe it might include interstate, 6 lanes, concrete.

Entity & Attribute Detailed Citation: reference to other sources of detailed information on the content of the data set; pointer to a data dictionary

5Entity andAttribute

Entity and AttributeOverviewEntity and AttributeDetailed Citation


Examples: Land Use Codes5

Entity andAttribute

Entity and AttributeOverviewEntity and AttributeDetailed Citation

Useless:

212223

Slightly Better:

AGRICULTURAL LAND

21 - Cultivated Land 22 - Pasture Land23 - Transitional Agricultural Land

A Walk Through the GuidelinesMuch Better:

AGRICULTURAL LAND

21 - Cultivated Land Cultivated land includes those areas under intensive cropping or rotation, including periods when a parcel may be fallow. It represents land planted to forage or cover crop. The units exhibit linear or other patterns associated with current or relatively recent tillage.

22 - Pasture LandLand in active pasture use. This class was discontinued and combined into 23.

23 - Transitional Agricultural LandThis category includes areas that show evidence of past tillage but do not now appear to be continuously cropped or in a crop rotation. Parcels in this unit include fields that are idle or abandoned and may or may not have been planted to a cover crop. In addition to displaying some evidence of past tillage, they usually are relatively uniform in vegetation.


Publisher: organization or individual that distributes the data set

Distributor Information: person who can answer questions about the distribution of the data set

Distribution Liability: statement of any liability assumed by the distributor

Limitations

Warranty

Liability

Redistribution Conditions

Data Delivery Terms

6Distribution

Publisher

Publication Date


Distribution Liability

Transfer Format

Transfer Size

Ordering Instructions

Online Linkage


Ordering Instructions: instructions for obtaining the data set. If applicable, instructions for acquiring data through Online Linkage element below

Online Linkage: (optional) when the data set is available online, this is the link to the Internet site where it can be downloaded

6Distribution

Publisher

Publication Date



Transfer Format

Transfer Size


Online Linkage


Two new reports available to aid in determining GIS

data distribution policy

6Distribution

Publisher

Publication Date



Transfer Format

Transfer Size


Online Linkage

Mapping the Risks:

Assessing the Homeland Security Implications of Publicly Available Geospatial Information

2004 RAND National Defense Research Institute

www.rand.org/

RAND researchers found no publicly accessible federal geospatial information deemed critical to meeting attackers’ information needs.

The researchers found only four publicly available federal databases that had information that is both useful to potential attackers and could not be obtained from other widely available sources. The four federal

databases are no longer being made public by federal agencies


Two new reports available to aid in determining GIS

data distribution policy

6Distribution

Publisher

Publication Date



Transfer Format

Transfer Size


Online Linkage

Guidelines for Providing Appropriate Access to Geospatial

Data in Response to Security Concerns

www.fgdc.gov/fgdc/homeland/FGDC_access_guidelines.pdf

A Federal Geographic Data Committee Homeland Security Working Group study investigating restrictions to geospatial data access that are

reasonable, sensible and cost effective

Purpose of a Metadata Entry Tool

Organizes metadata content

Provides help

Formats results– printed reports– webpages– Clearinghouse searches

Can it write the whole record for you?

Tools

ArcCatalog– Part of ESRI’s ArcGIS software– FGDC version is built-in– MGMG version, download at:

www.lmic.state.mn.us/chouse/arccatalog.html

DataLogr– Software independent– Free!– FGDC version, download at:

www.imagin.org/resources/datalogr.html

– MGMG version, download at: www.lmic.state.mn.us/chouse/datalogr.html

ArcCatalog

Editor– create a new record– change an existing metadata record

Stylesheet– needed to view a metadata record

MGMG Editor

7 sections

Help

FGDC Editor7 sections

Help

many sub sections for each section

MGMG Editor

FGDC Editor

Repeating screens

Lineage

Converting formats

ArcCatalog: stores records in XML format

FGDC Editor

– Use Export button to create HTML

MGMG Editor

– Use MGMG Converter to create HTML Creates hotlinks for web addresses

DataLogr

7 sections

Help

Hands-on Exercise with ArcCatalog

Part 1: Get started– Become familiar with viewing and editing screens

Part 2: Edit a record– Change the content of an existing metadata record

Part 3: Display a record– Create an HTML version of your edited record

– Compare different views of a record

Part 4: Create a new record– Evaluate an automatically created record

– Create your own starter template

How to Make this Easier…

Value metadata

Create metadata during your project

Prioritize legacy data

Use existing resources

Writing tips

Use your judgment

Share the task

Value metadata

Establish its value for yourselfand for your organization

Short-term investment long-term payoff

Metadata is no longer optional;it is part of being a GIS and IT professional

Cat Food

Create metadata during your project

When you create new data

When you change existing data

If you write metadata as you go along,at the end, it is done!

Tuna

Prioritize legacy data

What is most critical?

What are you asked about the most?

What may be lost soonest?– Information that is quickly forgotten

– Information that only one person or organization knows

Use existing resources

Guidelines and tools

Starter templates

Existing documents

Other peoples’ metadata

Writing tips

Goal: Concise but complete

Metadata records are drafts– Fill out in any order

– Modify when information changes

Get help from others– Editor

– Interviewer

OK to describe imperfections in the data– Data Quality and Purpose fields

– OK to say, “I don’t know”

Use your judgment

Some information is defined

Fixed set of choices

Fixed format

Some information is flexible

Defining data sets– Not too fine or too broad

Amount of detail– Answer the questions that you want answered

Share the task

Divide by areas of expertise

Do something…

How will you follow up after the workshop?

Imagine yourself back at your usual place of work…

Now, write down one or two specific things related to

metadata that you could do in the next week or two

Do something…

How can you follow up after the workshop?

Suggestions:

– Install one of the metadata tools on your computer

– Start documenting a data set you are currently creating

– Create your own starter template

– Use the GeoGateway to look for data you need

– Write down an obstacle and think of a way around it

– Tell a coworker about something of value you learned today

More HelpLMIC

ArcCatalog

www.lmic.state.mn.us/chouse/meta_help.html

FGDCwww.fgdc.gov/metadata/metadata.html

Questions?

Chris Cialek– [email protected]– 651-297-2488

Nancy Rader– [email protected]– 651-297-3281

Safeguarding GIS Data through Metadata Christopher Cialek & Nancy Rader Minnesota Land Management Information Center September 2004.

Documents

data slide

data set slide

subject slide

metadata standards

whats metadata

data title

search metadata

data set access