Strategies for Engaging End Users to Ensure Project Success William P Mahoney III

Strategies forEngaging End Users

to Ensure Project Success

William P Mahoney III

Strategies forEngaging End Users

to Ensure Project Success

William P Mahoney III

OutlineOutline

• Why is this topic important?• Defining a Decision Support System (DSS)• Assessing the client’s culture• Evolutionary software development process• Strategies for optimizing end-user buy-in• Case example – Road Weather

NCAR Research Applications Laboratory

NCAR Research Applications Laboratory

• SizeAbout 200 people, approximately half are

atmospheric scientists, and half are engineers

• Mission– Develop Solutions: Work closely with customers

to develop applications designed to solve specific problems

– Technology Transfer: Transfer knowledge and technology to US government agencies, the private sector, and foreign governments

Why is this topic important?Why is this topic important?

• The importance of connecting science & technology to society is increasing.

• Many new technologies are not being implemented.

• Developers have traditionally not been well connected to end users.

• Learning how to engage end users is critical for advancing technology.

What is being done?What is being done?

• American Meteorological Society has goal to reduce the gap between end users and the weather and climate enterprise.

• More companies and research organizations are engaging end users directly.

• Technology generators are starting to incorporate operations research concepts into their development cycle.

AMS Commission OrganizationAMS Commission Organization

AMS Commission on the Weather and Climate Enterprise

Commissioner: George Frederick

Board on EnterprisePlanning

Chair: Terry Tarbell

Board on EnterpriseCommunication

Chair: Matt Parker

Board on Enterprise Economic Development

Chair: Bill Mahoney

CommissionSteering Committee

Commission Executive Committee

CommercialWeatherServices

UniversitiesGovernment

WeatherServices

UserCommunity

Weather and Climate EnterpriseWeather and Climate Enterprise

AMS Board on Enterprise Economic Development

AMS Board on Enterprise Economic Development

Decision Support SystemsDecision Support Systems

a) An automated tool that makes decisions?

b) A semi-automated tool?c) A handbook of recommended

practices?d) Local newspaper or news program?e) Student assistant?

What is a decision support system?

Answer: All the above!Answer: All the above!

• It is imperative that you develop a relationship of trust with the client (sponsor and users) in any development process.

• Patiently listening to their needs is a critical first step.

• You need to understand their world.

Assessing the Client’s NeedsAssessing the Client’s Needs

Assessing the CultureAssessing the Culture

Before one can consider developing or

implementing a DSS, some important

questions need to be asked to assess the

client’s culture.

First Question:

1) What problem(s) are you trying to solve?

You need to understand the users’ decision space

Probing questions must be askedseveral different ways before apotential answer or solution mayemerge.

Assessing the CultureAssessing the Culture

2) What is the culture of the client’s

organization?

- Would a DSS be seen as threatening?

- Does automation pose problems?

- Are general support tools viewed positively?

Assessing the CultureAssessing the Culture

Assessing the CultureAssessing the Culture

3) What actual tasks and/or decisions could be supported?

- Routine or mundane tasks (e.g., data analysis)

- High-level analytical tasks (e.g., information assimilation)

- Critical decisions (e.g., impacting lives or costs)

Assessing the CultureAssessing the Culture

4) What job categories would benefit most?

- Lower skilled workers

- Supervisors

- Middle managers

- Senior managers

- Executives

Knowing the job category(s) that will utilize the decision support system will allow

intelligent judgments to be made about the design and

human factors issues.

5) What technical capabilities exist?

- Is there an in-house framework to support the new technology?

• Network system (external & internal)• Desktop computers for end-users• Remote field communications (PDAs)• Database of pertinent operations data

Assessing the CultureAssessing the Culture

TRS-80 Computer

6) What are the potential benefits?

- Safety

- Property Protection

- Productivity

- Risk Reduction

Assessing the CultureAssessing the Culture

Assessing the CultureAssessing the Culture

7) Who will champion the technology?

- Management vs. staff

- Technology push (by management) or pull (from staff)?

Management Staff

8) What are the users’ expectations?

This is a big deal!

It is critical that users expectations are at the appropriate level when new technologies are introduced. It is always better to error on the side of caution and let the users be creative on how they utilize new capabilities.

Assessing the CultureAssessing the Culture

Decision Support SystemsDecision Support Systems

Application Categories:

1) Strategic Planning (condition prediction)2) Tactical Planning (alert functions)3) Operations Management (productivity)4) Incident Management (notification function)5) Risk Management6) Evaluation of “What if?” scenarios7) Training Tool (off-line assessments)

Best Development ProcessBest Development Process

Software Concept

IncorporateCustomerFeedback

IncorporateCustomerFeedback

Elicit CustomerFeedback

Elicit CustomerFeedback

Deliverthe

Version

Deliverthe

Version

DeliverFinal

Version

DeliverFinal

Version

PreliminaryRequirementsAnalysis

Design ofArchitecture and System Core

Develop aVersion

Evolutionary Delivery Evolutionary Delivery

This method is consideredthe best software practice from Rapid Development, Taming WildSoftware Schedules by Steve McConnell, Microsoft Press,1996.

• There are no off-the-shelf plug and play DSSs that can fully address the needs of all end-users. There are no “one-size-fits-all” solutions!

• A “bottoms-up” rather than a “tops-down” approach should be used for DSS system development.

• Stakeholders need to determine the level of sophistication that is required for their specific DSS application.

Decision Support SystemsDecision Support Systems

Technical Risks & ChallengesFor Weather Related DSSs

Technical Risks & ChallengesFor Weather Related DSSs

Weather Diagnoses & ForecastsWeather Diagnoses & Forecasts

• The weather information requirements of each user community are now highly specialized.

• The weather research community has not traditionally been focused on the individual needs of specific user communities – change is occurring

Then

Now

What is usually required? - High resolution information (misoscale = 40 m to 4 km)

- Rapid updates (minutes to hours)

- Both short term and long lead time (seasonal) forecasts

- Diverse set of input data

Weather Diagnoses & ForecastsWeather Diagnoses & Forecasts

Users are now seeking:

- Probability metrics (or confidence)for key meteorological parameters

Example:

“What is the probability of the high temperature Reaching 95F between 4 and 6 pm tomorrow?”

“What is the probability of 3 inches of rain falling into my reservoir?”

Weather Diagnoses & ForecastsWeather Diagnoses & Forecasts

0

0.05

0.1

0.15

0.2

0.25

74

78

82

86

90

94

98

10

2

Tmax

• Weather DSS Testbeds should be established in end-user operational environments to test advanced “products” (methods, techniques, etc.) and to learn how end-users utilize the new information.

• Testbeds should be used to evaluate:• how to communicate uncertainty• new presentation options• how users interpret products• identify knowledge gaps in atmospheric science• societal aspects of new capabilities

Research NeedsResearch Needs

• Weather DSS Testbeds for a variety of economic sectors could be established. Industry partners could help offset costs (in-kind contributions, hosting of technology, etc.)

• Weather DSS Testbeds:Agriculture AviationSurface Transportation Human HealthEnergy ConstructionWater Resources Emergency

Management

Research NeedsResearch Needs

Winter Road MaintenanceWinter Road Maintenance

Case ExampleCase Example

MDSS User Needs AcquisitionMDSS User Needs Acquisition

1) Surface Transportation Weather Decision Support System Requirements (STWDSR)

2) Identified large Stakeholder Group3) Discussed overall goals and objectives with large

stakeholder group4) Created focus group from stakeholder group5) Assessed current practice6) Developed concept of operations7) Developed prototype design8) Developed prototype – first step that code was written!9) Demonstrated prototype10) Iterated11) Kept stakeholders in the loop throughout process

Current CapabilitiesCurrent Capabilities

The highway maintenance managers currently usenewspapers, Weather Channel, USA Today, NWS, and private meteorologists to make decisions.

Road weather information systems are not integrated with road management systems. The winter road maintenance decision process is generally reactive in nature.

On a plow route by plow route basis, users want:

• Treatment Type (chemical, plow, sand, etc.)• Treatment Amount (lbs per lane mile, etc.)• Treatment Location (plow routes)• Treatment Timing (start/end)

Expressed User NeedsExpressed User Needs

MDSS ProductsMDSS Products

• Weather parameters Air temperature Relative humidity Wind speed and direction Precipitation type, rate,

accumulation

• Road Parameters Road temperature Bridge temperature Bridge frost potential Blowing snow potential Road contamination & chemical

concentration

• Treatment Recommendations Treatment Type Treatment Location Treatment Amount

Thank You