A user assessment of decision support tools

A User Assessment of Decision Support Tools for Addressing Nonpoint Source Pollution in the Saginaw Basin Watershed:

Environmental Learning Using Computer Interactive Decisions (ELUCID) and the Great Lakes Watershed Management System (GLWMS)

Laura Young, Research AssistantSaichon Seedang, Research Specialist

Michigan State University Institute of Water Research

69th Soil Water Conservation Society International Annual ConferenceMaking Waves in ConservationJuly 29, Lombard, Illinois

CONTEXT & SYSTEMS OVERVIEW

Context

Awarded Great Lakes Restoration Initiative (GLRI) grant

Enhance technical assistance

Accelerate BMP implementation

Reduce non‐point source pollution

Trainings

• May 2014• Overview and walkthrough of systems• Tutorials• Evaluation

ELUCID

Environmental Learning Using Computer Interactive Decisions

Developed with input from various stakeholders

Flint River Watershed

ELUCID

Engage and inform various users groupsField staff, watershed groups, teachers, the public

ELUCIDSubwatershed Prioritization Field‐level Prioritization

ELUCID

Seamlessly linked with other systemsEPA databases, GLWMS

Great Lakes Watershed Management System

Evolved through various multi‐partner efforts

Four GLRI priority watersheds (Saginaw Bay)

Great Lakes Watershed Management System

Subwatershed prioritization for NPS pollutants

Great Lakes Watershed Management System

Provides baseline NPS pollution model estimates at field‐scales

Great Lakes Watershed Management System

Conducts field‐scale scenario evaluations of land cover changes or best management practices (BMPs)

Great Lakes Watershed Management System

FRAMEWORK FOR EVALUATION

Assessment PurposeTo evaluate:

– Perceived usefulness of system2

• Prioritizing high risk sites• Engaging producers

– Impacts of usage on efficiency at work– Integration into conservation work processes1

– Ease of use1– Transparency1– Anticipated Usage– Identify additional features needed to improve DSS’s– Participants’ understanding of material presented at the training

1. Inman, D., M. et. al. 2011. 

2. Ives., B., M. Olson, J. Baroudi. 1983.

Survey Instrument

• Types of Questions– ELUCID, GLWMS, Training– Five‐point Likert Scale (51)– Multiple Choice (7)– Short Answer (12)

• Administered online through Survey Gizmo

RESULTS

Survey Population

Sample size: 17 trainees

Response Rate: 82%– 14 completions– 1 partial completion– 2 did not take survey

Profession

Will help me work with producers to place BMPs on high risk areas

Will help me work with producers to place BMPs on high risk areas

Will increase my ability engage with a producer

Will increase my ability engage with a producer

A producer will more motivated to implement a BMP on a high risk area

after reviewing data in ELUCID

A producer will more motivated to implement a BMP on a high risk area

after reviewing data in GLWMS

ELUCID

GLWMS

Perceived Usefulness

ELUCID

GLWMS

Will help me work with producers to place BMPs on high risk areas

Will help me work with producers to place BMPs on high risk areas

Perceived Usefulness

A producer will be more motivated to implement a BMP on a high risk area

after reviewing data in ELUCID

A producer will be more motivated to implement a BMP on a high risk area

after reviewing data in GLWMS

ELUCID

GLWMS

Perceived Usefulness

Will increase my ability to engage with a producer

Will increase my ability to engage with a producer

ELUCID

GLWMS

Perceived Usefulness

Will make my work easier

Will make my work easier

Will reduce the amount of effort I spend on certain aspects of my work.

Will reduce the amount of effort I spend on certain aspects of my work.

Will reduce the amount of time I spend on certain aspects of my work.

Will reduce the amount of time I spend on certain aspects of my work.

Perceived Impacts on Efficiency

ELUCID

GLWMS

Will make my work easier

Will make my work easier

Perceived Impacts on Efficiency

ELUCID

GLWMS

Will reduce the amount of effort I spend on certain aspects of my work.

Will reduce the amount of effort I spend on certain aspects of my work.

Perceived Impacts on Efficiency

ELUCID

GLWMS

Will reduce the amount of time I spend on certain aspects of my work.

Will reduce the amount of time I spend on certain aspects of my work.

Perceived Impacts on Efficiency

ELUCID

GLWMS

ELUCID will easily integrate into my conservation work process.

The GLWMS will easily integrate into my conservation work process.

Perceived Potential for Integration

ELUCID fills a gap in the current set of tools I frequently use to do my job.

The GLWMS fills a gap in the current set of tools I frequently use to do my job.

ELUCID

GLWMS

ELUCID fills a gap in the current set of tools I frequently use to do my job.

The GLWMS fills a gap in the current set of tools I frequently use to do my job.

Perceived Potential for Integration

ELUCID

GLWMS

ELUCID will easily integrate into my conservation work process.

The GLWMS will easily integrate into my conservation work process.

Perceived Potential for Integration

ELUCID

GLWMS

Future Work

• Follow‐up survey– Barriers to targeted conservation– How respondents have used the system

• Focus groups• Examining Google Analytics data with evaluation results

ReferencesInman, D., M. Blind, I. Ribarova, A. Krause, O. Roosenschoon, A. Kassahun, H. Scholten, G. Arampatzis, G. 

Abrami, B. McIntosh, P. Jeffery. 2011. Perceived effectiveness of environmental decision support systems in participatory planning: Evidence from small groups of end‐users. Environmental Modelling & Software. 26, 302‐309.

Ives, B. M. Olson, and J. J. Baroudi. 1983. The Measurement of User Information Systems. Communications of the ACM. 26 (10): 785‐793.

Matthews, K.B., M. Rivington, K.L. Blackstock, G. McCrum, K. Buchan, and D.G. Miller. 2010. Raising the bar? The challenges of evaluating the outcomes of environmental modelling and software. Environmental Modelling & Software. 26: 247‐257.

Rhee, C. and H. Raghav Rao. 2008. Evaluation of Decision Support Systems. International Handbook on Information Systems. 

QUESTIONS?

www.iwr.msu.edu/flintriverelucid.iwr.msu.eduwww.iwr.msu.edu/glwms

Laura Youngyoungla9@msu.edu

517‐353‐8587