Is There a “Fast-track” Into the Black Box?: The Cognitive Models Procedure Robert R. Hoffman, Ph.D. John W. Coffey, Ed.D. Mary Jo Carnot, M.A. Institute.

Is There a “Fast-track” Intothe Black Box?:

The Cognitive Models Procedure

Robert R. Hoffman, Ph.D.

John W. Coffey, Ed.D.

Mary Jo Carnot, M.A.

Institute for Human & Machine Cognition

University of West Florida

Poster presented at the 41st Annual Meeting

Psychonomics SocietyNew Orleans16 Nov 2000

Abstract

The Cognitive Models Procedure supports experts in crafting a model of their own reasoning, by relying on the presentation of "bogus" models. The goal is to have the expert re-construct a better model, which ideally will converge on the “modal model” of expert reasoning which includes the Duncker refinement cycle, recognition-priming, and situation awareness. The models are then cross-validated by: (1) field study of the observable behaviors that are entailed by models (e.g., the expert asserts that he always first inspects a certain data type), and (2). having a group of experts attempt to determine which of the models characterizes each of the experts in the team (or organization). This poster reports an application of this interview technique in a project on the reasoning and knowledge of expert weather

forecasters.

Application Domain

• Meteorology and Oceanography Training Facility, Pensacola Naval Air Station

• Pool of forecasters spanning Apprentice-Journeyman-Expert-Senior Expert levels of proficiency

Determination Standard Range Observed Range

LOW HIGHSenior Expert 50,000 - up 54,136 55,412Expert 40,000 - 49,999 47,064Junior Expert 30,000 - 39,999 31,032 36,898Senior Journeyman 25,000 - 29,999 25,720 29,127Journeyman 15,000 - 24,999 19,758 20,940Junior Journeyman 10,000 - 14,999 11,067 14,464SeniorApprentice 5,000 - 9, 999 6,384 8,448Junior Apprentice - 4,999 1, 848 2,016

Domain-appropriate Proficiency Scale

Knowledge Modeling

Utilized the following methods:– Structured Interviews– Protocol Analysis– Knowledge Audit– Critical Decision Method– Concept-Mapping

The Procedure

• Phase 1 - Choice among “bogus” models

• Phase 2 - Refinement

• Phase 3 - Guessing Game

• Phase 4 - Verification via direct observations

Base Model of Expertise

DataExamination

Action Queue

Course of action

Mental simulation,trend analysis, courses of action

Judgments,Predictions

Hypotheses,Questions

RecognitionPriming

Problem of the Day

Focus, leverage points,gaps, barriers

Mental Model

Refinement

Cycle

Action Plan

Refinement

Cycle

Situation

Awareness

Cycle

“Bogus” Models used in the CMP

MENTAL MODEL

COMPAREWITH MOS DISAGREE

INSPECT OTHERDATA

OUTPUTFORECAST

AGREE

REVISE

SURFACE DATA

UPPER AIR DATA, RADAR, GOES

MENTAL MODEL

COMPARE WITH MOS

REVISE

OUTPUTFORECAST

STYLE 1 STYLE 2

SURFACE DATA

Phase 1-2 Results

The strategy of providing the Participant with the "Bogus model" guidance in crafting a representation of their own reasoning strategy seems to have been successful. Each Participant balked at the bogus models, but then went on to craft a model that they felt comfortable with.

SURFACE OBSERVATIONS,SATELLITE IMAGERY

MENTALMODEL

OUTPUTFORECAST

EXAMINE FOR PERSISTENCE(past 24 hours ofobservations)

COMPARE WITHNUMERICALMODELS

ARE THERE ANY SALIENTDISAGREEMENTS IN THE DATA??

REVISE

DO ALL THE DATAAGREE?

A Journeyman's ModelExample Models

from Phase 1, 2

WATCHWEATHER CHANNEL

SKY WATCH

WALL OF THUNDER: SATELLITE IMAGES OBSERVATIONS (ASOS) LIGHTNING RADAR

MENTALMODEL

FORM

REFINE

NUMERICAL MODELS: NGM

COMPLETE DD-175-1 FORMS

NUMERICAL MODELS: ETA NOGAPS MM5 (4-DAY OUTLOOK)

PERSISTENCE( IN SUMMER)

LOCALKNOWLEDGE

PRODUCE FORECAST

EXAMINE WHEN TIMEPERMITS

EXAMINE

REFINE

EXAMINE

BEFORE ARRIVAL

AGREE

DISAGREE

REFINE

A Journeyman's Model

EXAMINE UPPER AIR DATA

COMPARE TO RADAR& SATELLITE DATA

FORM MENTAL MODEL

NGM

ETA

MM5

MOS

ACCOMMODATELOCAL EFFECTS

OUTPUT FORECAST

COMPARE TO NUMERICAL MODELS

AGREE

DISAGREE

DISAGREE

DISAGREEAGREE

AGREE

AGREE

REVISE

FORM REVISE

REFINEMENT CYCLE

REFINEMENT CYCLE

A SeniorExpert's Model

MENTALMODEL

SKY WATCHING

WALL OF THUNDER: ORDER IS OPPORTUNISTIC

FORM

REFINE

INCLUDES: HEURISTICS, CLIMATOLOGY, PERSISTENCE, PATTERN RECOGNITION

NUMERICAL MODELS:

REFINE IF EVIDENCE DISCONFIRMS

OUTPUT FORECAST

A Senior Expert's Model

Phase 3 Results

• Most participants adopted a "divide-and-conquer" strategy of first trying to identify models of senior experts or forecasters with whom they were more familiar, and identifying last the models that they thought were bogus and the models they thought were those of Apprentices.

• Of all of the identification judgments (N = 60), 13 or 22 percent were correct identifications.

Expert models were sometimes incorrectly identified as being models of Apprentices and bogus models.

Participants found the task to be an interesting challenge. Participants' comments during Phase 3 were revealing of the extent to which

they have opportunities to become familiar with one another's strategies. Participants may have opportunities to see what one another does, but do not

share much information about their actual strategies for data search, mental model formation and hypothesis testing.

Phase 4

• Observation of forecaster behavior when they first came on watch after a period of days when they had not been on the watchbill.

• Allowed probe questions: Understanding of the current weather situation? (e.g.,

"Is what you're seeing fit with persistence?" "Are the models agreeing?")

Skywatching (e.g. " Did you look at the Weather Channel before you came in?")

What are you going to do now? (e.g., "Are you going to look at the models?)

WALL OF THUNDER: SATELLITE IMAGES OBSERVATIONS (ASOS) LIGHTNING RADAR

MENTALMODEL

FORMREFINE

NUMERICAL MODELS: NGM

COMPLETE DD-175-1 FORMS

NUMERICAL MODELS: ETA NOGAPS MM5 (4-DAY OUTLOOK)

PERSISTENCE( IN SUMMER)

LOCALKNOWLEDGE

PRODUCE FORECAST

EXAMINE

REFINE

EXAMINE

WATCHWEATHER CHANNEL

SKY WATCH

BEFORE ARRIVAL

AGREE

DISAGREE

REFINE

DONE AT FDOWORKSTATION

AFFIRMED

AFFIRMED

AFFIRMED

NO NEED IN THISSITUATION(DONE LATER IN THEWATCH)

AFFIRMED

THIS WAS A PERSISTENCESITUATION

AFFIRMED

NOT NEEDED ON THISWATCH IN THISTIME PERIOD, BYHAPPENSTANCE

AFFIRMED

EXAMINE WHEN TIMEPERMITS

Example Phase 4 Results

MENTALMODEL

WATCHWEATHER CHANNEL

SKY WATCH

WALL OF THUNDER: ORDER IS OPPORTUNISTIC

NUMERICAL MODELS:

COMPLETE DD-175-1 FORMSAND OUTPUT FORECAST

LOCAL KNOWLEDGEHEURISTICS

FORM

MODIFY FORECASTS

UNFOLDING WEATHERESPECIALLY SEVERE

SITUATION AWARENESS CYCLE

REFINE

AFFIRMED

AFFIRMED

AFFIRMED

NOT AFFIRMEDIN THIS CASE (PERSISTENCE)

NOT NEEDED DURINGTHIS TIME PERIODON THIS WATCH

AFFIRMED

SURFACE OBSERVATIONS,SATELLITE IMAGERY

MENTALMODEL

OUTPUTFORECAST

EXAMINE FOR PERSISTENCE(past 24 hours ofobservations)

COMPARE WITHNUMERICALMODELS

ARE THERE ANY SALIENTDISAGREEMENTS IN THE DATA??

REVISE

DO ALL THE DATAAGREE?

GP DID NOT INSPECTSURFACE DATA

MENTAL MODELREVISION BASED ONSATELLITE IMAGERY

SKYWATCHING

THIS WAS NOT APERSISTENCESITUATION

RESOLVING HIS OWN MENTALMODEL WAS MORE IMPORTANT THANGETTING THE DATA INTO AGREEMENT

AFFIRMED

AFFIRMED

AFFIRMED

AFFIRMED

LEGEND SOLID BLACK = REVISED PATHSOUTLINE = COMMENTARIES

EVENT,QUERY, REPLY,ANALYSIS

TIME

E 4:40 P#2 arrived at METOC

E P#2 looked at the latest TAF

Q RH: "Did you skywatch on your way in? Did you look at theweather channel?"

R P#2: "Skywatch, yes. But we do not have a TV."

Q RH: "What was your understanding of the current weathersituation based on what you saw?"

R P#2: Thundering was somewhere.There was a cirrus mid-deck out to the north.There were cumulus towers--meaning instability--though theywere small.Must be decent capping as there was no thunder nearby.

E 4:40 P#2 looked at the Wall of Thunder--SIGMETS

E P#1 to P#2: "I don't know what's going on."P#2 to P#2: "You always know what's going on."P#1: "Everything's to the north of us."

E 4:46 P#2 inspects GOES water vapor loop on the FDO workstation.

E 4:51 P#1 told P#2 about the pilots that are out and en route back toNASP.

Example Results

Five elements of P#2's Model were affirmed:• Examination of satelli te imagery• Mental model formation• Mental model refinement• Comparison to computer model guidance• Reliance on local knowledge

Three elements of P#2's Model were qualif ied:• He did not begin by examining surface data.• This was not a persistence situation• Refinement focused on developing a coherent mental model

Four elements of the Base Model were affi rmed--• Data examination• Mental model formation• The refinement cycle• Reliance on local knowledge (i.e., this was not a persistence situation)

Participant Phase 1 TaskTime

Phase 3 TaskTime

Phase 4 TaskTime

P#1 23 minutes 20 minutes 21 minutesP#2 na 26 minutes 76 minutesP#3 15 minutes 10 minutes 30 minutesP#4 15 minutes 11 minutes 11 minutesP#6 9 minutes 17 minutes 42 minutesP#13 15 minutes naP#20 10 minutes 13 minutes

Total total 364 minutes

Conclusions

• Results from the procedure included models of the reasoning of seven forecasters, affirmed in a second phase, and then (for five of them), affirmed with modifications based on observations of actual forecasting behavior.

• For the total task time of 364 minutes, the procedure would average out to 52 minutes task time to develop a model. This figure is without doubt considerably less that than time takes in traditional experiments (i.e., think aloud problem solving with protocol analysis) to reveal and

verify reasoning models. • The CMP holds promise as a "fast track into the black box," allowing

the development of reasoning models and the testing of hypotheses concerning reasoning models in less time than taken by traditional experimentation.

References• Hoffman, R. R., & Markman, A. M. (Eds.) (in press). Human factors

in the interpretation of remote sensing imagery. NY: Lewis.• Hoffman, R. R., Crandall, B., & Shadbolt, N. (1998). A case study in

cognitive task analysis methodology: The Critical Decision Method for the elicitation of expert knowledge. Human Factors , 40 , 254-276.

• Hoffman, R. R., Shadbolt, N., Burton, A. M., & Klein, G. A. (1995). Eliciting knowledge from experts: A methodological analysis. Organizational Behavior and Human Decision Processes, 62, 129-158.

• Hoffman, R. R., Detweiler, M. A., Lipton, K., & Conway, J. A. (1993). Considerations in the use of color in meteorological displays. Weather and Forecasting, 8, 505-518.

• Hoffman, R. R. (1991). Human factors psychology in the support of forecasting: The design of advanced meteorological workstations. Weather and Forecasting, 6, 98-110.