Field Catalog 2 Development Mike Daniels for Greg Stossmeister Collaborative Tools and Metadata Group Computing, Data and Software Facility EAC Presentation.

Field Catalog 2 Development

Mike Daniels for Greg StossmeisterCollaborative Tools and Metadata GroupComputing, Data and Software Facility

EAC PresentationDecember 4, 2013

What: The field catalog is a web-based collaborative tool that provides critical information for real-time decision making, facilitates the communication of plans, status and scientific information and documents project operations.

Version 1.0 First Deployed: Nov 1995 1st Aerosol Characterization Experiments (ACE-1)

Most recent deployments: V 2.0 deployed for MPEX and SAS

History: Supported 70 Field Campaigns in a little over 17 years



Why develop the field catalog 2?• The field catalog is heavily used and the Perl cgi code was at the

end of its supportable lifecycle.• Changes in today’s field project requirements require new services

to be offered to support investigators in the field, and ensure project success.

Goals of new development:• Create single code base for all new catalogs in modern software

framework for improved sustainability, adaptability• Provide a tool for integrated display and sharing of all EOL field data

products• Ensure rapid, reliable delivery of products and information to field

project participants for real-time decision making• Provide high-demand services and provide for the addition of

new services as technology advances such as mobile computing

Field Catalog 2 Accomplishments

- A particular strategy of ours was to field-test components of field catalog 2 while we were still using the old field catalog version.

Data Ingest software

Old catalog latency: ~ 5 minNew ingest code ~20 sec

Also included the ability to mirror content between two catalogs: tested in DYNAMO

As the push has been for more real-time products at high frequencies, data volumeshave gone up dramatically:extensively tested in DC3 (> 1TB and > 8.5 Million products) and subsequent campaigns



Ingest Monitoring – Catalog Nagios

Old catalog: Manual process, difficult tocatch problems when they first begin

New code: Monitoring – automatic usingopen source tool called “Nagios”

Notification via e-mail (future: phone textmessages) and web

Monitors product flow as well as servers,disk space, and services such as chat,http, ftp, etc.



GIS – Catalog Maps

Old catalog: No serviceNew code: Started with Google Earth API transitioned to OpenStreetMaps and OpenLayers (Google - too many limitations – OS, devices, downloads)

Implemented playback mode in MPEX, inclusion of airborne cameras along with interactive dropsonde and other Field Catalog product layers


- This was the one piece we couldn’t test with the old field catalog. This marked the completion of our first new production version (and second)

User Interface – V2.0, 2.1

Old catalog: Multiple code bases,big Db tables, desktop/laptopDisplay only

New code: Single code base, Responsive Design, navigationrefactored based on feedback fromusers to handle high volume data.

Recent project users part of redesign process through user survey and partially functional mockups


Under the hood changes

• Agile development mode – scrum meetings, code reviews, sprints, • Responsive design – support range of devices and OS’s, tablets, phones, computers

• New workflows for Catalog Admins – use of repositories, prod & dev servers

• Vetting of metadata vocab (no more GOES-13, goes13, Goes-13)

• Integrated Database with Data Archive – reduce duplication of effort

• Ability to field remote catalogs – DYNAMO, HIWC and OFAP requests

• Collecting user feedback and incorporating users in design (Google surveys post campaign, building mockups and asking for feedback before the software is built)

Field Catalog 2 Lessons Learned - Summary of User Feedback

• Field Catalog navigation was not as user-friendly as in the past, products were hard to find, it was also hard to tell when new products came in.

• Report editing was no longer available and was missed. There was a workaround but it was somewhat cumbersome.

• Reports were hard to find in list view. Older way of seeing available reports was preferred by some.

• New color theming implemented had problems. Some pages were hard to read. We need more contrast between font and background colors in the core content areas.

• The mission table no longer provides quick links to all catalog products for a select IOP period. Only select products were listed.

• Lack of organization in preliminary data archive and metadata was a detriment. Others appreciated the ease of uploading products (FTP).

Field Catalog 2 Lessons Learned - Developer Perspective

• Our test environment did not exactly simulate the production catalog. In particular we did not accurately simulate the real-time data feeds and content. This made it hard to assess the functionality of the web interface before the campaigns and made for a lot of last-minute stress resolving problems as the field campaigns began.

• Usage Testing has to be done throughout the development process. It’s not just for the integration phase at the end. Code reviews were done throughout to satisfy the SEs but testing by Catalog Admins also needs to occur to ensure the usability of various features implemented.

Development Considerations

• Community Input – we work extensively with PIs in every field campaign to understand what is needed, where the problems are, what new functionality is required and what are the priorities. Includes planning meetings, in the field, project de-briefs

• Tool Issues – we test these tools heavily before and during the field campaign and track and resolve software and hardware issues as they come up. For issues that are not immediately resolvable we track these as usability issues and consider development solutions during or between campaigns as there is time.

• Technology – we try to keep an eye on new developments that might potentially benefit the community – Currently Web Mapping Service (WMS) and mobile computing

• Agile Development – we practice iterative development, taking time between projects to assess issues as well as collect new requirements for upcoming campaigns. We then develop new code based on priorities and resources.

• Resources – 2 SEs who are also heavily involved in Field Catalog 2 deployments + one-time funds support

Development plans 2014

• Reporting Tools – Complete single code base reporting tools backed by a Db and provide the ability to view facility status summaries for the entire project. Restore editing capability and implement multi-author forms.

• Complete the replacement of the Internet Relay Chat (IRC) Bot that is no longer supported and implement dedicated MySQL server to minimize load issues

• Develop on-line help documentation

• Transition COSMIC interactive soundings map to GIS tool

• Create form for Flight Plan generation from waypoints so users don’t have to create kmls on their own

Active Field Season expected


• Include model forecast fields in Catalog Maps for real-time decision- making as well as playback (CONTRAST, DEEPWAVE)

• Implement mapping, looping in Catalog Maps

• Display current aircraft position with icon that show heading and updates at 10-20 second intervals

• Develop a system for simulating aircraft flight tracks and redirection in real and accelerated time for more realistic project dry run exercises

(PECAN)

• Include thumbnail displays for more functional use of the ISFS/CentNet and other time series data products (LoSHA)



• Develop a low-bandwidth interface to Field Catalog products for users on ships or mobile instrument platforms (ReFLEX, PECAN)

• Develop a preliminary data access interface for the Field Catalog

• Incorporate time stamped WMS layers for standard use in Catalog Maps and possibly the Mission Coordinator Display

• Incorporate analysis tools into Metrics reporting to help better understand how the catalog is being used

• Investigate Cloud Services as an alternative in remote locations or where scalability is an issue


Field Catalog 2 Development Mike Daniels for Greg Stossmeister Collaborative Tools and Metadata Group Computing, Data and Software Facility EAC Presentation.

Documents

field catalog software

components of field

field catalog product

years field catalog

old field catalog version

gis catalog maps old

field campaigns

field project participants