Collaborative Ethnography, An Approach to the Elicitation of Cognitive Requirements of Teams

7/26/2019 Collaborative Ethnography, An Approach to the Elicitation of Cognitive Requirements of Teams

1/6

Proceedings the 2009 13th International Conference on Computer Supported Cooperative Work in Design

Collaborative Ethnography: An approach to the elicitation of cognitive

requirements teams

Luiz Carlos L Silva Junior , Marcos

R

S Borges , Paulo Victor

R

de Carvalho

Programa de P6s-Graduac;ao em Informatica Universidade Federal do Rio de Janeiro Brasil.

{luizcarlos mborges}@nce. ufrj.r

[email protected]

Abstract

The paper presents a collaborative ethnography

approach

or

cognitive requirements elicitation

o

work

teams in complex environments. It discusses the

concepts

o

cognitive systems and their requirements

andpresents a review

o

methods commonly used in the

elicitation

o

requirements both in the case

o

traditional systems and complex systems. hen it

points to some advantages

o

a collaborative approach

in comparison to other approaches. An evaluation plan

o

the approach based on experimentation and the

development

o

a groupware

to

support the proposed

methodology is also presented. he groupware aims to

stimulate collaboration and an organization in the

elicitation process

o

cognitive requirements.

Keywords: Ethnography, CSCW, Requirements,

Cognitive systems.

1. Introduction

The human interactions in group work activities are

important objects in the study human problem

solving and decision-making processes, and give

important clues to investigate the tacit knowledge that

teams uses during their work activity. Behind these

interactions we can find important mechanisms

reasoning that will drive people, according to their

experience, to select and manage their actions in

context-dependent work situations. The analysis

these interactions can lead to the discovery basic

requirements for the construction artifacts that can

efficiently support the process

team members

decision-making.

The aim

this paper is to show how collaborative

ethnography (i.e. an ethnography performed by many

agents who can interact with each other) can be used to

elicit the knowledge that shape team members

interactions. We also show in which way the

knowledge obtained using the collaborative

ethnography can be very important to elicit useful

cognitive requirements for the implementation

information support systems that will give adequate

support to people tasks.

We begin the paper with cognItIve systems

definitions and its requirements through some examples

978-1-4244-3535-7/09/ 25.00 2009 IEEE

these systems in real life. Then we present some

cognitive task analysis (CTA) methods used in

knowledge elicitation process describing their strong

and weak points. Later, the collaborative ethnography

approach for cognitive requirements elicitation will be

presented and compared with the other methods

normally used. Finally, we show the requirements a

groupware to support the collaborative ethnography

approach to improve some weak points

the method,

enabling its use in an effective way.

Cognitive systems design requirements

A cognitive system is a self regulated and adaptable

system that functions using knowledge about itself and

the environment to planning and modify their actions

[9]. adequate design a cognitive system depends

basically on: The existence a common vocabulary

between their parts; the study

the agents' cognition

during their actual activity, analyzing the man

technology interactions in real environments [10] rather

than studies that analyze each system component apart

(i.e. man, machine and interface) in more controlled

settings. Our research focus on cognitive systems

design because their ever growing sociotechnical

complexity due new technologies (automation,

communication, information) and new system

structures (more regulation levels, consumer feedback

and control) claims to a change the traditional

(linear) models and old paradigms

human cognition

in system design to they take in consideration the actual

human-system interactions.

Nowadays we can observe the technology

facilitating the execution

tasks and operations with

high degree automation. Cognitive systems are

present in energy power plants, air traffic management

systems, vehicles, control

emergencies, and in work

environments that demand high degree coordination.

To produce effective cognitive systems, i.e. cognitive

systems that can be adequately handled by the human

agents, there is a need to consider actual cognitive

requirements in the system design phase.

However, the increase use

technology results in

an increase the complexity cognitive systems. The

digital control rooms with large video display units,

computers with ever growing processing capacity, and

other technological devices to support cognitive tasks

167


2/6

generate another problems, as the search for high

efficiency, making that the systems operate closer to

their maximum capacity and safety boundaries, an

increasing performance dependence among the various

tasks, and a significant growth o the amount o data

that have to be processed by the human being [10].

Such situations can generate a considerable

degradation in these systems. To cope with complexity,

human agents adapt their activities generating

variability in the tasks execution, which may lead to

systems malfunction the drift to failure mechanism). It

is important to note that in the last few decades serious

accidents have happened in sociotechnical complex

systems, indicating the need to change the paradigm on

how human cognition is treated in complex systems

design [18].

Many o these system failures and accidents emerge

in situations, in which operators did not understand the

system actual situation, i.e., the information systems do

not provide the right information in the right moment

to the operators who were not able to adapt their

behaviors according to the actual situation demands and

make their decisions in a safe way. To deal with this

problem is necessary to emphasize design processes

that facilitate the adaptation o the human cognition to

the system functionality.

The cognitive requirements, in the scope

o

this

work, can be defined as the functional requirements o

the system that will give critical support to the cognitive

activities o the operator throughout the execution o

his/her work. This means that these requirements will

be paramount for the construction

o

information

systems capable to enable people to achieve adequate

situation awareness enhance perception possibilities,

decision-making and action planning support).

These cognitive requirements must be incorporated

in the design o displays, man-machine interfaces, and

are present in the collaborative characteristics o many

systems. We can observe some examples

in

control

rooms [5], [19]; military decision-making systems [3];

air traffic control [2], [8].

To elicit knowledge for complex system design, we

argue that is necessary to carry out cognitive task

analysis in actual work situations, understanding how

and why operators make their actions, and the intrinsic

human-system relations human-organization, human

technology, and human-human actions). Therefore, we

claim that collaborative ethnography is one o the most

adequate approach to elicit cognitive requirements. In

the next section, we will describe some methods

traditionally used to elicit knowledge for information

systems design, in order to compare them to the

collaborative ethnography approach we propose in this

paper.

3. Knowledge elicitation approaches for

cognitive

t sk

analysis

In this section, some CTA methods already used for

in information system design in several environments

will be described. Some o these methods, like

interviews and direct observation are often used in the

elicitation o requirements for information system

design and business-oriented processes. Ethnographic

approaches are not so used in information systems

design, but they are widely used in the elicitation

o

experts knowledge and social factors.

3.1. Interviews

Interview is one

o

the most common methods for

requirements elicitation, however nor always does this

technique capture all the information needed to a

complete requirements definition. To a large extent, the

data generated from interviews are field notes, meeting

reports, which are difficult to analyze later on, and can

be complemented with the existing documentation in

the organization.

In many situations, the information elicited from

interviews was incomplete, due the existence o

difficult questions to be answered, i.e., questions related

to activities where tacit knowledge is used. Bell [ ]

noted that is not rare the situation in which people tell

the idealized information, i.e., people say how they

should do their tasks rather than how they actually do

them.

There are research to refine the interview technique

[4], helping the organization and integration o the data

using argument structures or domain models, as the

Joint Application Development JAD) approach that

provide a way to requirement elicitation allowing the

identification o different points o views, and means to

solve conflicts and achieve consensuses.

3.2. Observation

The observation o people actual work activities in

the organizations is a useful way to understand the

interactions, practical skills, and tacit knowledge

developed by the work teams, as well as the

organization culture. According to Bell [1], we do not

have to ask people what they do. Rather, we must

observe how people carry out their activities, because

from observation findings it is possible to discover the

reasons and motivations behind systems use, as well as

the problems or difficulties related with system

usability.

The use o observation appears in some domains for

knowledge elicitation and system improvement. In

ergonomics, direct observations are the most used

method to understand the actual work conditions, the

operators activities and their influence on the people

health and in the system production [6].

168


3/6

The observation can be participative or non

participative according to the behavior

o

the observers

in the observation process [17]. In the participative

observation, observers interact with the workers

observed to achieve a better understanding o the work

activities. The non-participative observations can be

direct or indirect. In the direct observations the people

are observed individually, the behaviors are written

down and registered field notes , whereas in the

indirect observations there is the use o technology to

support the register process audio and video

equipment .

However, observations techniques are not efficient

for the capture

o

cognitive skills in many cases. In

emergence situations where the observer cannot

intervene directly or even be present during the activity,

it is easy to catch the tacit knowledge and the cognitive

skills using only non-participative observations.

Therefore, the experiments, simulations, interviews or

tell histories techniques may support the capture tacit

knowledge in these situations [3].

3.3. Ethnography

The ethnography is a methodology that comes from

the Social Anthropology, which consists o studying an

object living the same reality o this object. This

approach aims to understand and to describe, a nation,

its people or culture, using natural or participative

observation for long periods.

Using ethnography, the observed group or culture

issues are understood by living in the same

environment, being present, and making the things that

the people make and as they make [1]. It is possible to

understand how and, mainly, why the activities are

done in one determined way, because the phenomenon

is studied inside the social, cultural and organizational

context [16].

It is important to note that the ethnography contains

traces o the two previous techniques, because the

information collected are complemented with data from

observations and from informal interviews. However,

the difference in the ethnography is that all individuals

observers and observed are situated in the same

context. Therefore, the method has many possible

advantages [13]: Bigger familiarization with the domain

o the organization; discovery o the informal activities,

issues related to cognitive and collaborative

mechanisms; better understanding

o

the actions,

decisions, strategies, behaviors and interactions

o

the

agents in the context where they occur; perception on

how the interaction with the resources or devices

occurs; and more visibility about usability problems o

the existing technologies.

Moreover, the ethnography shows how the physical

disposal

o

the work environment and the layout

o

the

devices and equipment have influence, positive or

negative, on the efficiency o the activities as well as in

the difficulties, and the respective adaptations to face

work problems.

Using the ethnography for the elicitation

o

requirements, we are able to get more details and a

more complete description o these requirements,

because the activities are observed instead o described

by the users. These factors are especially important for

the analysis and design

o

complex systems.

One

o

the ethnographic techniques we will use

ahead in the Collaborative Ethnography we propose in

this research is the Fast Ethnography [11] see figure

that recognizes the time and resource limitations to

carry out detailed and intensive ethnographic studies,

and uses short term observations together with the

debriefmeetings feedback.

Figure Processo de etnografia rapida

The ethnographic approach presents also some

problems. Normally, its execution is complex, due the

impossibility o one person alone to capture and to

analyze all the information

o

the social environment.

The method demands extensive planning and

coordination issues

the ethnography will be carried

out by teams, because people may have different

perceptions and points o view about the observed

activities. Another limitation is the need o a long time

to capture and analyze what occurs in the field, to get

rich and detailed information [15]. In some cases, is

necessary a familiarization with the domain to be

studied. Moreover, it is difficult the use o the

technique in large work settings, complex and

distributed, due the huge number o variables to be

observed.

The technique also can present risks for the

researchers, or to be impracticable the presence o the

researches/observers jeopardize the work activities

observed. In some cases, there is the need o getting the

permission for entrance in the work settings and for

registering the information, and also the acceptance and

the assent from those who will be observed [6].

In the scope o software development process the

ethnographic technique also presents some difficulties.

One o them is absence o formalism and objectivity

required for information systems designers, because it

presents the results in a descriptive and textual format

[20], making difficult their transformation in software

requirements.

69


4/6

4. The collaborative ethnography approach

for cognitive requirements elicitation

In this section we will discuss how the

collaborative approach can address the elicitation

of

cognitive requirements elicitation

of

wor k teams. We

first stress the importance

of

the combination

of

collaboration and ethnography particularly how this

combination can help the study

of

complex activities.

Then we present a succinct review

of

previous work

which have used this combined approach to elicitate

of

interactions and the social dimension

of

teamwork.

groups and their interaction being performed by a team

of

ethnographers is a very challenging goal. The huge

amount

of

information collected that needs to be

organized and processed requires not only an organized

process bu t also an appropriate supporting tool. The

complexity

of

this alternative is a consequence

of

combined circumstances originated by the multiple and

perhaps conflicting perspectives from the observers

p ar t an d the potentially high nu mb er

of

interactions

among the members

of

the observed group. We will try

to avoid the loss

of

relevant information intrinsic to

scenario B.

4.1. hycollaborative ethnography?

4.2. Applications of collaborative ethnography

The collaborative ethnography is a technique that

has been used extensively in other domains for the

social analysis

of

work activities. May and Pattillo

McCoy [14] reported a work reproducing the way the

ethnographers discuss their findings in a collaborative

way generating a single document with the information

collected and their different viewpoints. They reported

how different the ethnographers perceptions about the

environment are. They claim that variety is due to

several cognitive factors that also help the reach

of

consensus in the discussion.

It is also important to note the value of

the

collaborative ethnography to complement the details

of

the object

of

study. The inconsistencies and ambiguities

of

observations are more easily identified and solved. In

other words the ethnographers

playa

positive influence

on the results

of

the process.

On the other hand it becomes clear that the

collaborative ethnography has some drawbacks. First it

shares or perhaps increases the time spent to collect

analyze and conclude about the collected information.

The need to join to compare and to discuss the

information collected requires time. The collaboration

required needs extra effort and is time consuming. The

field notes needs to be combined in order to reflect the

different perspectives

of

observers.

According to May and Patillo-McCoy [14] the

observation itself influences the results

of

the

ethnography. Thus the use

of

several observers opens

the possibility

of

analyzing this impact in the process

and at the same time enriching the information

collected. We expect that the approach contributes to

the expose

of

tacit information specialized knowledge

and the decision rationale mostly due to the interaction

between the ethnographer with the task the

environment and especially with other ethnographers.

Machado [13] also carried out a collaborative

ethnography study aimed to elicitate software

requirements. The study reports an experience with

field observations and with the use

of

a tool that

facilitates the consolidation

of

data from multiple

sources into a shared repository. In comparison with

traditional techniques based on interviews the

h t

the om Ina o e

nograp Iy s U les

Observer/ Observed Individual Teams

Individual A B

Teams C

T bl 1 C r

First it is pertinent to clarify why to add the

collaboration to the ethnography method. A

straightforward combination between observers and

observed subjects is presented in Table

We ca n have

a single pe rso n observing another pers on o r a group

and a team observing a single person or a group.

Scenario A is the most common case: An

individual observing another individual carrying out

some task. This situation also refers to a number

of

observer/observed pairs being done in parallel.

f

the

observation refers to the same task some variation may

occur between different observers/observed. The results

are quite dependent on the background and on the

previous experience

of

the observer. There may also be

some information loss during the observation. In this

scenario the aggregation

of

information from different

sources is not an easy task and may show some

inconsistencies.

Scenario B is possible but not common. In this

scenario the loss

of

information is potentially high. It is

very difficult to a single observer to capture and to

understand all tasks and interactions particularly if the

activities are complex. A way to overcome these

difficulties is to do the observation in several sessions.

The scenario C is normally used when the

observation requires multiple perspectives over the

same observer o r task. In this scenario it is expected

that the information gathered and processed would be

richer that that collect by a single observer. This can be

considered a particular case

of

scenario D assuming

that the observers would meet and discuss about their

findings before during and after the ethnography. This

is not easy to realize without the support

of

an

appropriate process and a supporting tool.

Finally the scenario D is the circumstance that

will be dealt with in this work. The observation

of

170


5/6

observation approach have shown more efficient and

generated more complete information.

The experiments carried out in that study has

shown that a groupware plays an important role in the

coordinating of activities and on the organization of the

data videos, photos, comments, notes, etc. reducing

the burden generated by the collaborative ethnography.

5. roupw re requirements to support

collaborative ethnography

According to Guerlain [7], the evaluation

of

activities from a coordinated team involves the

independent observation of multiples individuals. The

analysis of these independent observations requires they

should be coordinated, codified and correlated before a

subjective evaluation can be performed.

Thus, similar to other works [7]

[ 2]

[13], the

support

of

a groupware tool aimed to assist the

information exchange and the discussion among

ethnographers is desired. The groupware tool that

supports the collaborative ethnography should facilitate

collaboration and interaction after the gathering

of

field

data in order to identify the cognitive requirements with

less effort.

The basic functions for the groupware to support

the collaborative ethnography are:

register, update and closing

of

ethnography

projects;

record profiles of observer and observed persons;

assign of observers to observed activities and

persons;

record of notes and maintenance ofhistorical data;

document creation and its association with

problems, comments, ideas and hypotheses raised

by ethnographers;

support the discussion and negotiation about the

documents and the requirements generated;

support awareness about the level of participation

and contribution;

support multimedia elements;

support awareness mechanisms for both

synchronopus and asynchronous interaction,

informing them about contributions new elements,

comments, etc. added by other members.

The groupware tool will be used after the

information capture in the field. The team members will

be able to individually insert all type of information

they believe are important to the requirements

elicitation. Then, the tool will support the discussion,

the consensus information, and also how the former led

to the latter. It will also allow members to group their

notes using categories actor, setting, process, etc. . The

notes can be associated with problems, cognitive

requirements and possible solutions to these

requirements. See Figure 2 for a schematic view of the

data managed by the groupware.

Figure Products and their relationships

Collaborative Ethnography

The tool will provide means to coordinate the

ethnography activities in both asynchronous and

distributed way. It should help analysts to reach

consensus, besides inducing the reflexive thinking. It

will also help the planning

of

further studies and

observations on specific parts

of

the work, which

demand a deeper analysis. The tool should be designed

using usability principles that should be evaluated

during the study. However, the main goal of the study

and the tool, which is to contribute to the efficiency and

efficacy

of

the process, will be pursued. A set

of

questionnaires will try to verify the conformity

of

the

approach and the tool to these goals.

6

Conclusions nd future work

In this work we reviewed some important concepts

in the area of cognitive systems. We also revised the

main approaches for the knowledge elicitation:

interviews, observation and ethnography. We

confirmed that these approaches, though extensively

adopted, still show some inadequacies and inefficiency

in the requirements elicitation process, particularly in

relation to cognitive requirements in complex

environments.

To deal with these problems, we proposed a

collaborative ethnography approach as a more

appropriate alternative for the cognitive requirements

elicitation. We claim that the ethnography approach has

many advantages, such as a detailed analysis of the

environment where the operator performs his/her task.

When ethnography is combined with a collaborative

approach that supports different viewpoints as well as

interaction and discussion among the observers, we

believe we have an approach that produces a more

efficient and consequential results.

Part of the tuning of our approach is dependent on

the application domain. Thus the groupware tool has to

be highly parameterized. The goal is to facilitate the

identification

of

cognitive requirements from the mass

of information gathered in the process and the reaching

of consensus over the main points to address.

7


6/6

For the experiments we will conduct two types

of

techniques for the elicitation

of

cognitive requirements:

the first based on interviews and the second based on

the approach proposed in this paper. We intend to carry

out a qualitative and quantitative comparison of the

results. We aim to evaluate not only the method itself

but also the support

of

the groupware tool. Our purpose

is to check

if

our hypothesis about the efficiency and

efficacy

of

our approach hold true in a variety

of

situations.

eferen es

[1]

Bell, G., (2004). Insights into Asia: 9 Cities, 7

Countries, 2 Years - What people really want from

technology. Technology@lntel Magazine.

[2] Bentley, R., Hughes, J., Randall, D., Rodden, T.,

Sawyer, P., Shapiro, D., Sommerville, I., (1992).

Ethnographically-informed systems design for air traffic

control, In Proc. Computer-Supported Coooperative

Work Conference, ACM Press, New York, 123-129.

[3]

Crandall, B., Klein, G Hoffman R.R., (2006). Working

minds: a practitioner s guide to cognitive task analysis

Cambridge, Mass.: MIT Press.

[4] Christel,

M

G.; Kang,

K

C., (1992). Issues in

Requirements Elicitation, Technical Report, CMU/SEI

92-TR-012, ESC-TR-92-012, September.

[5] dos Santos, I.1.A.L., Teixeira, D.V., Ferraz, F.T.,

Carvalho, P.V.R. (2007). The use of a simulator to

include human factors issues in the interface design

of

a

nuclear power plant control room. Journal of Loss

Prevention in the Process Industries 21(3), 227-238.

[6] Guerin, F., Laville, A., Daniellou, F., Duraffourg, 1.,

Kerguelen, A., (2004). Compreender 0 trabalho para

transforma-Io: a pratica da ergonomia. Edgard Blucher:

u n d ~ o

Vanzolini, Sao Paulo.

[7] Guerlain S., Shin T., Guo H., Calland 1.F., (2002). A

team performance data collection and analysis system.

In: Proc. of the human factors and ergonomics society

46th annual meeting, Baltimore, MD, 30 September-4

October.

[8] Harper, R., Hughes, J.A. and Shapiro, D., (1990).

Harmonious Working and CSCW: Computer

Technology and Air Traffic Control, In: Studies in

Computer Supported Cooperative Work: Theory,

Practice and Design, ed. Bowers and

S

Benford,

Amsterdam, North- Holland.

[9]

Hollnagel, E Woods, D D., (1999). Cognitive

Systems Engineering: New wine in new bottles.

International Journal of Human-Computer Studies,

Volume 51, Number 2, August 1999 p p 339-356(18)

[10] Hollnagel,

E

Woods,

D

D., (2005). Joint cognitive

systems: Foundations of cognitive systems engineering.

CRC Press, Inc., Boca Raton, FL, 2005.

[11] Hughes, J., King, V., Rodden, T., and Andersen, H.,

(1994). Moving out from the control room: ethnography

in system design. Proc. of the ACM conference on

Computer supported cooperative work, Chapel Hill,

North Carolina, October 1994, p 429-439.

[12] Hughes, 1., O Brien, 1., Rodden, T., Rouncefield, M

(1997). Ethnography, Communication and Support for

Design. Technical Report Ref. CSEG/24/1997, Lancaster

University, Lancaster, UK, 1997. Available at:

http://www.comp.lancs.ac.uk/computing/research/cseg/9

7_rep.html

[13]

Machado, R G., (2008). Developing of a Collaborative

Ethnography Method for Software Requirements

Elicitation. Rio de Janeiro, 2008. M.Sc. Dissertation,

Graduate Program in Informatics, Federal University of

Rio de Janeiro, Brazil (In Portuguese).

[14]

May, R A B., Pattillo-McCoy, M., (2000). Do You See

What I See? Examining a Collaborative Ethnography.

Qualitative Inquiry, Vol. 6,No 1 65-87.

[15] Millen, D., (2000). Rapid ethnography: time deepening

strategies for HCI field. Proc. ACM Symposium on

Designing Interactive Systems, ACM Press, New York,

280-286.

[16] Myers, M D., (1999). Investigating information systems

with etnographic research. Communications

of

the

Association for Information Systems, Vol. 2, Article 23,

December.

[17] Nardi, B., (1997). The use of ethnographic methods in

design and evaluation. In: Helander, M.G., Landauer,

T.K., Prabhu, P (Eds), Handbook of Human-Computer

Interaction. Elsevier Science, North-Holland, Holanda,

361-366.

[18] Reason,

T., (1990). Human Error. Cambridge, U.K.:

Cambridg University Press.

[19] Roth, E M., Christian, C.K., Gustafson, M., Sheridan,

T.B., Dwyer, K., Gandhi, T.K., Zinner, M.1., Dierks,

M.M., (2004). Using field observations as a tool for

discovery: analysing cognitive and collaborative

demands in the operating room. Cognition, Technology

and Work 6, 148-157.

[20] Sommerville, I., Rodden, T., Sawyer, P., Bentley, R.,

Twidale, M., (1993). Integrating ethnography into the

requirements engineering process. Proc. of IEEE

International Symposium on Requirements Engineering,

San Diego, CA, 165-173.

172

Collaborative Ethnography, An Approach to the Elicitation of Cognitive Requirements of Teams

Documents