Integrated Data Management Network: Project Tracking Deanne DiPietro Data Manager, California LCC.

Integrated Data Management Network:

Project Tracking

Deanne DiPietroData Manager, California LCC

The Team

Lei Ann WilsonFort Collins Science Center

Tim KernFort Collins Science Center

Deanne DiPietroCalifornia LCC

Zhahai StewartCalifornia LCC

Matt HellerGreat Northern LCC Emily Fort

National Climate Change & Wildlife Science Center

Josh BradleyArctic LCC

DD

Goal of the IDMN Project Tracking Task

“Support cataloging and tracking of LCC funded projects, and improve the integration of project information for use across the LCC Network.”

Multiple Needs

• National level: for the National LCC to analyze what the LCC Network is funding, doing, and with what partners.

• The Public: to discover LCC projects relevant to their work

• LCC level: for LCCs to manage detailed project information including proposals, funding, contacts, deliverables, contracts.

Information Needs for the National LCC Project Catalog

• All LCC projects in one place• Links to more detailed info• Types of deliverables• Types of end users• Funding by category• Partner match• # of projects co-funded by multiple LCCs• # of projects in progress/completed• # of projects by subject area• Projects by Congressional district• And more….

Two kinds of project information

• Project Cataloging: basic, standardized information about LCC projects, with links to more detail on LCC websites

• Project Tracking: detailed information about every aspect of a project needed to manage it from start to completion; typically expands on project cataloging with deliverables, contacts, deadlines, etc.

Project Tracking LCC level

Project Cataloging

National level

The Challenge

Diversity among LCC approaches, styles, and technical capacity

The need to have one view across all the LCCs’ projects.

DD

Dual Approach1. Develop a unified project

metadata catalog via a data exchange and aggregation system,

2. Help LCCs with choices and adoption of diverse project tracking software that support national participation.

DD

Previous work:

Simple National Project CatalogSpring 2013

Gathered, standardized, and assembled project metadata from all 22 LCCs

Collaboratively developed a common standard for the data, got buy-in from all LCCs

Worked with each LCC to incorporate the standard with their current methods for managing this data

http://lccnetwork.org/DD

Existing National Level Project Database(as of Spring, 2013)

• 395 projects from 22 LCCs • Filter by LCC or Primary Category, full-text search• Links to detailed info on LCC website• Download the spreadsheet

Simple National Project Catalog Content Standard

Basic InformationLCC NameUnique ID for project (within LCC)Project TitleDescriptionProject StatusProject Start DateProject End DateProject URLProject Lead person name, emailProject Lead OrganizationPartner OrganizationsTarget Audience/End UsersFiscal Years of AllocationComments

Duplicate ReportingOther LCC co-funding this projectIs this project cataloged in ScienceBase?

FundingTotal Funding by Your LCCYour LCC Funding by Year and SourceTotal In-kind, matching, or other fundsMatching or In-Kind Funds by Year and SourceNotes about matching and in-kind contributions

Classifying and groupingProject CategoryDeliverableSubject KeywordsGeographic KeywordsCongressional Districts

DDRed = has a controlled vocabulary

Previous work:

Simple National Project Catalog

The SNPC Spreadsheet tool. Each LCC data manager was provided their data back in standardized form.

Previous work:

Simple National Project Catalog

DB or CMS

non-std spreadsheet

(or none)

LCCs with databases incorporated the standard and wrote CSV exports.

LCCs with spreadsheets or nothing yet were helped to adopt the standard spreadsheet.

At the end of this effort, everyone was theoretically on the same page and ready to easily update the National Project Catalog using the standard.

Excel, for analysis

Webinterface

(all LCCs)

Standardized metadata in

CSV

Standardized metadata in

CSV

Project Metadata Exchange and Aggregation

IDMN Objectives:

• Develop a design of a data exchange architecture

• Address the needs of the National LCC while allowing the LCCs to keep their individual data systems

• Implement prototypes to refine the protocols and provide reference implementations

Standardized metadata in

JSON

Generalized Architecture

National LCC Project Catalog

Spreadsheet with National Catalog Fields

LCC (master records)

Aggregation Site (derived records)

Standardized metadata in

CSV

Exchange Formats

Local DB/Application with National Catalog Fields Web

interface(all LCCs)Web

interfaceExports

Aggregation Site Decision: Prototype in ScienceBase

• Already has wide acceptance as a common platform for LCCs

• Willing to commit staff time to this effort

• Opportunity to integrate with other ScienceBase services

• Challenging to modify an existing system that serves multiple purposes, but worth it in the long run

ScienceBase

• Data catalog and collaborative data management platform with project cataloging capabilities

• Central search and discovery application• Data Repository• Used by many different communities for many different

purposes• Has a front end web application, DEPTH, to create project

metadata that complies with the national LCC standard• Web services for facilitating other applications which provides

added flexibility to display data

LW

ScienceBase as a master record source AND the aggregation site

LCC National Project Catalog

atScienceBase

Master LCC project records

Aggregation Site (derived records)

Import via standard

Exports

Spreadsheet with National Catalog Fields

Local DB/Application with National Catalog Fields

ScienceBase DEPTH

Direct entry

Status of ScienceBase Pilot Effort

• Edit in ScienceBase using DEPTH for master record project metadata entry and management (now being tested by core group)

• Batch upload of standard records using JSON-based REST Services (still in dev/testing with North Pacific)

• Export of data for use in LCC webpages (still in dev/testing with Desert LCC) LW

Prototype Architecture (interim arrangement)

Simple National Project Catalog

at lccnetwork.org

The Simple National Project Catalog continues to serve the purposes of aggregation and translation of LCC project data to the exchange standards while we develop and test things at ScienceBase.

Spreadsheet with National Catalog Fields

Local DB/Application with National Catalog Fields

CSVCSVCSV

JSON

Spreadsheet

IDMN Objective:• Investigate possible project cataloging or

project tracking systems which could be integrated into the overall framework, existing or in development.

Informing Selection and Adoption of Project Tracking Software in the LCCs

Supporting LCC Software Adoption• Reviewed the range of

viable options for managing project metadata and contributing to the national catalog

• Created information sheets to assist in informed choices

Option Comparison Summary

SNPC Spreadsheet

ScienceBase (DEPTH)

Arctic PTS

Great Northern PTS

*Customizing a CMS (Content management system)

Public Web Display

No Yes No No Yes

Web Data Entry

No Yes Yes No Possible

Technical Expertise Needed for Administration

No No High Low High

Project Tracking

No Some Yes Yes Possible

Customization Options (data, Reporting, etc.)

No Limited Yes Yes Possible

Automated National Data Integration

No Yes Planned Planned Possible

Ongoing Development

No Yes Yes Yes N/A

Contact Name Zhahai Stewart, Deanne DiPietro

Lei Ann Wilson

Josh Bradley

Matt Heller

N/A

● Five ways to accomplish project tracking and contribute to the national project catalog

● LCCs have diverse needs, no one solution is best for all of the LCCs

Project Tracking Option Information Sheets

See Confluence under IDMN Project Tracking Team

Simple National Project Catalog Spreadsheet

● Microsoft Excel● Low technical overhead● You can customize it (just keep

the standard fields intact)● You can upgrade later using

the CSV format

National LCC Project Catalog

Just send it

when requested

Standardized metadata in

CSV

Link: http://lccnetwork.org

Great Northern Project Tracking System (based on Arctic Project Tracker)

Manual entry

OR

ingest Simple National Project Catalog spreadsheet ● Microsoft Access

● Low/moderate technical overhead● Project tracking features (i.e. lcc strategic elements)● You can customize it (entry forms, tables, reports,

etc.)● Sciencebase linking/export options● Batch uploads of lccproject.org data

National LCC Project Catalog

Standardized metadata in

.CSV

Link :https://www.sciencebase.gov/catalog/folder/53324b07e4b08fca49cf2d9bWebinar: http://greatnorthernlcc.org/event/462

ScienceBase using DEPTH

Manual entry via DEPTH**

Batch upload*

Display of information on LCC website*

● Use via WWW● Low technical overhead● Integration with RFP and project

deliverable management applications

● limited customization

* = still in beta** = currently under evaluation

National LCC Project Catalog

Auto-majically

Standardized metadata inCSV or JSON

Arctic Tracker Project Tracking System

National LCC Project Catalog

Standardized metadata inCSV (JSON

export planned)Manual entry

Link http://arcticlcc.org/ptsVideo: https://www.youtube.com/playlist?list=PLDO3wp3QE78u_uBRoth4s7MBxMZi9pJSG

● Used by Arctic LCC and Western Alaska LCC● JS, PHP, and PostgreSQL● High technical overhead● Project tracking features (i.e. Tasks, Funding/Invoice Contracting

Workflows, etc)● You can customize it (free/open source)● Browser based multi-user edit support

Custom CMS-based System

Example above: California LCC’s Climate Commons website

Other LCCs doing their own systems include:South Atlantic, Gulf Coastal Plains and Ozarks, Upper Midwest and Great Lakes

● High technical overhead● You have to build it ● Incorporate the standard and

create an export to provide to the National Catalog

LCC National Project Catalog

Standardized metadata inCSV or, soon,

JSON*

* v1.0 is currently being tested

IDMN Project Tracking Accomplishments

• Established a metadata standard and formats for machine-readable exchange

• Described interim and final architectures

• Prototyped the import of project metadata into ScienceBase

• Developed a data entry form in ScienceBase DEPTH

• Evaluated compatible methods for LCC Project Tracking

• Recommendations for further work

Recommendations

● Need a follow-on effort to create a production system that supports LCC project cataloging and contribution to a National LCC Project Catalog

● List of technical specifications found in our “Roadmap” report, including:

○ use ScienceBase as the production site, add the functions that the Simple National Project Catalog is now serving,

○ refinements to the standard, ○ automate tasks now done manually.

See Confluence under IDMN Project Tracking Team for our Roadmap report

What’s Next

• Effort continuing as a subcommittee of the DMWG

• National Project Catalog Update: call for data in November

• Continuing to use lccprojects.org with ScienceBase projects integrated while we work toward the long-term goal of having it all in ScienceBase

Acknowledgments

Sean Finn, GN LCCJosh Bradley, Arctic LCCDeanne DiPietro, CA LCCEmily Fort, NCCWSC/USGSMatt Heller, GN LCCSally Holl, Desert LCCTim Kern, USGSZhahai Stewart, CA LCCLei Ann Wilson, USGS

Thanks to the National LCC for

funding the IDMN Project

[email protected][email protected]

[email protected]

Thank you!