Toward a Large-Scale Model of Language Comprehension in ACT-R 6 July 2007 Jerry Ball 1, Andrea Heiberg 2, and Ronnie Silber 3 Air Force Research Laboratory.

Toward a Large-Scale Model of Language Comprehension in ACT-R 6

July 2007

Jerry Ball1, Andrea Heiberg2,

and Ronnie Silber3

Air Force Research Laboratory1

L3 Communications2

Lockheed Martin3

2

The VERBOSE Project

• Verbalization Between Operators and Synthetic Entities

• Synthetic Teammate

–Air Vehicle Operator (AVO)

• Flies Simulated Unmanned Aerial Vehicle (UAV)

• Communicates with other team members

–Language Capable

• Comprehension

• Generation

• Dialog Management

3

Chat

Output

Language

Comprehension

Language

Generation

Dialog

Manager

ACT-R Architecture

Motor ModelMotor

Actions

Situation ModelVisual

Input

Chat

Input

VERBOSE System Overview

4

• Unmanned Aerial Vehicle (UAV) Simulation

• Three-Person Team Performing Reconnaissance Missions

– Air Vehicle Operator (AVO)

• Flies UAV

– Payload Operator (PLO)

• Takes pictures of targets

– Data Exploitation Mission Planning & Communications (DEMPC) Officer

• Plans mission

Cognitive Engineering Research on Team Tasks (CERTT) Lab

5

AVODEMPCPLO

Experimenter Station

Cognitive Engineering Research on Team Tasks (CERTT) Lab

6

NextWaypoints

AltitudeAir Speed

To GoalGear and

Flaps Request Flight Plan

Artificial horizon and

course

AVO Screen 1

7

FuelTime

Autopilot

Moving Map

Alarms

AVO Screen 2

8

AVO Chat Screen

9

Theoretical FoundationsLanguage Representation and Processing

• Double R Grammar

– Cognitive Linguistic theory of the grammatical encoding of referential and relational meaning

• Double R Process

– Psycholinguistic theory of the processing of English text into Double R Grammar based representations

• Double R Model

– Computational implementation using the ACT-R cognitive architecture and modeling environment

www.DoubleRTheory.com

10

Empirical Validation of Linguistic Representations

• Empirical study to determine the kinds of linguistic representations that humans construct during language comprehension

– What is/are the basic structure(s) of the clause?• Subject Predicate

• Subject Verb Object

• Other

– Methodology• Ask subjects to group words in expressions into

meaningful units

• Assumes subjects have explicit knowledge of the meaningful groups that occur in expressions

the man bit the dog

11

Empirical Study – Preliminary Results

• A pilot study was conducted

• Results were compared to an early study by Levelt which used multidimensional scaling:

– Levelt, W. (1970). A scaling approach to the study of syntactic relations. In G. B. Flores d’Arcais and W. Levelt (eds.) Advances in psycholinguistics 109-121. NY: Elsevier.

12

Levelt’s Early Study

the boy has lost a dollar

60

80

100

Relative frequency of “more related”

in triadic comparison

%

Connectedness

method

13

Pilot Study vs. Levelt Study

boy has lost a dollarthe

boy has lost a dollarthe

Levelt Study

Pilot Study

14

Double R Model

15

Levelt’s Early Study

Jan

(John)

eet

(eats)

Piet

(Peter) eet

(eats)

en

(and) peren

(pears)

3.0

5.0

7.0

Mean scale value

(7-point rating scale)

Connectedness

method

appels

(apples)

16

Pilot Study vs. Levelt Study

apples and Peter eats pearsJohn eats

Levelt Study

Pilot Study

appels en Piet eet perenJan eet

17

Double R Model

18

Start with Scripted Communication

• AVO: DEMPC this is AVO, please let me know the first waypoint!

• DEMPC: The first waypoint is LVN. It’s an entry point. There are no airspeed or altitude restrictions. The effective radius is 2.5 miles.

• AVO: PLO, I’m heading towards LVN.

• DEMPC: We’re within the effective radius so go to the second waypoint.

• AVO: Are there any altitude or airspeed restrictions for the second waypoint?

• DEMPC: The second waypoint is H-AREA. It’s a target. The airspeed restriction is between 50 and 200 knots. There is no altitude restriction. The effective radius is 5 miles.

• PLO: AVO, please keep the altitude over 3000 feet for the photo!

• PLO: I have a good photo of H-AREA.

• AVO: Moving on to the next waypoint.

19

Double R Model

20

Double R Model

21

Double R Model

22

Double R Model

23

Double R Model