TER Workshop J P San Diego

Examining Learner-computer Interactions: Advanced Lab-based Research Methods

Slides before 1st Section

Divider

Motivation of the research

Strategy as a unit of

analysis

Illustrative Analyses and

Some Findings

Unused Section Space 2

Technology Enhanced Research

Unused Section Space 1

Advanced lab-based methods

Illustrative study

J.P. San Diego and J.C. Aczel

Outline• Examining learner-computer interactions

– Focus on detecting learning, even when nothing is being explicitly “taught”

– Trying to understand why and how learning is occurring– Within the learning context– Largely visual

• Advanced lab-based research methods– Not just pointing a camera at a screen– Or asking “what are you learning?”– Advanced data collection: eye-tracking, sketches, gestures,

physiological measures– Advanced data analysis: handling multiple video streams,

software analytics and strategy as unit of analysis

Outline• Based on a specific study

– PhD research– Will broaden this out to reflections of other methods

and applications

Acknowledgements• Dr. James Aczel, Dr. Barbara Hodgson and Prof.

Eileen Scanlon• Prof. Josie Taylor and Dr. Richard Cox• Prof. Marian Petre, Prof. John Mason, Dr. Ann

Jones, Dr. Patrick McAndrew and Dr. Denise Whitelock

• Dr. Ekaterini Tzanidou, Dr. Geke vanDijk, Dr. Miquel Prats and Ms. Anesa Hosein

• The participants of the study

AcknowledgementsProf. Diana Laurillard and Prof. Margaret CoxPascal Mangold of Mangold Software & Consulting

GbHM Microsoft Research in Cambridge through Dr. Fabien

PetitcolasIET-IT (Will Woods, P. Downs, D. Perry & S.

Hammond) and CALRG ColleaguesOU-LTS (Mr. Collin Thomas) Prof. Jeff Johnson, Prof. Chris Earl, Dr. Peter Lloyd

and Dr. Georgy Holden,

The 2003 MSc Study

Methodological challenges

“You need times ‘cause you need it to that (points on the screen) times twenty”

“Oh OK I can see what it is doing (the graph) It is going towards there”

Traditional approaches to analysing video data

• Methodological– Reflexivity (e.g. Camera effect), selectivity (transcript

as data versus video as data)

• Technical– Selecting, setting up, and operating video equipment

• Practical– Data storage, transcription and coding

• Ethical– Anonymity and privacy

Digital Video and digital data

• Advantages– Consistent record than observation notes, capture

difficult-to-record events, multi-perspective, multi-observers, offers flexibility, stimulus for discussion

• Recent developments– Variety of media, logs in video forms, video search

technologies, processing power of computers, sensors, eye-tracking, haptics, sketch recognition, etc.

The data capture setup

19 February 2007 LKL Seminar ([email protected]) 12

Data capture and analysis tools

INTERACT™

Protocols• Think-aloud

– Ericsson and Simon (1984)• Eye-tracking

– Yoon and Narayanan (2003), Hansen et al . (2001)• Sketching

– Pirrie (1996, 1997), Cox (1996)

The study design

• Data collection– 18 students with A-level Maths or higher

• 3 comparable tasks– External mathematical representations

• Each task presented in either static, dynamic and interactive forms

Standard external representations and instantiations

Instantiations

• Static: Non-moving, non-changing, non-interactive

• Dynamic: Capable of animation through alpha-numeric inputs

• Interactive: Directly manipulable graphs

The data

JSD.mov

Main research question• How do representations instantiated in

different ways influence learners’ cognitive processes?

Strategies• strategic theories

– strategies are attempts to solve problems– theories are conjectured expectations, dispositions, or assumptions

(articulated or not), of some sort of reality in a particular context– a strategy can be considered as theoretical, in a sense, in that it incorporates

expectations about some state of affairs– theory can be considered strategic, in a sense, in that some are

instrumentally better adapted to reality than others• Donald T. Campbell: Blind-Variation-and-Selective-Retention

– a mechanism for introducing variation [thought trials];– a consistent selection pressure [concerns]– a mechanism for preserving and reproducing the selected variations

• Learning– processes of discontinuous trial-and-improvement of strategic theories under

the selection pressures provided by concerns

Hypotheses• Strategies with each standard external representation can be

characterised at different levels of granularity.• Learners’ choice of strategies depends not just on the standard

external representations given but also on the instantiation.• Mental constructions of images with graphical representations

vary between instantiations.• Attention paid with each standard external representation

varies between instantiations. • Expression of inferences varies depending on the instantiation• Analyses of strategies based on gazes, actions, utterances and

sketches can identify factors contributing to strategy choice in a way that is not possible with traditional observation techniques.

Strategies identified

• Representation-specific– Algebraic, graphic and numeric

• Imagining– Pen, mouse, mental, gaze, gesture

• Re-representing– Visual, textual, symbolic

Representations-specific strategies by task X instantiation

Frequency of participants for each strategy

The chart shows the participants’ imagining strategies graphed by instantiation across the

three tasks

Areas Of Interest

Root task - Area of Interest Analysis (AOI)

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

COUNT 57.4% 10.9% 31.6% 49.6% 23.4% 27.1% 57.2% 11.3% 31.5%

DURATION 60.6% 11.4% 28.0% 45.8% 29.1% 25.1% 65.9% 9.7% 24.4%

Graphic Equations Numbers Graphic Equations Numbers Graphic Equations Numbers

Static Dynamic Interactive

“Aha! moments”

Participant’s talk: Aha! They are the same distance away.

“Invisible path”

Participant’s talk: This is going from minus two…

“Invisible region”

Participant’s talk: I’m trying to imagine what happens as the line tends to infinity…

Re-representation

Participant's talk: I don’t know what to call it… Err… I’ll just draw it

‘Freeze frames’

Attention paid to representations

Focus of attention

Findings relating to difficulties

00:14:13:22P4: It will never ever comes cross… Something... it never comes across

Bringing the evidence together

19 February 2007 LKL Seminar ([email protected]) 39

Other examples of evidence

Other examples of evidence

19 February 2007 LKL Seminar ([email protected]) 41

Participant's talk: I don’t know what to call it… Err… I’ll just draw it

• Current project– hapTEL (Haptic Technology Enhanced Learning)– PhD Student (Arash Shahriari-rad)

• TER and Formative feedback

The Future

19 February 2007 LKL Seminar ([email protected]) 43

The Future…

• Current focus on attention– Mobiles– Windows– Books

• Jo Iacovides & games– jaw tension (EMG)– skin conductance (GSR)– heart-rate (EKG)– brainwaves (EEG)

From Marvin Minsky (The Society Of Mind)

It often does more harm than good to force definitions on things we don't understand. Besides, only in logic and mathematics do definitions ever capture concepts perfectly.

The things we deal with in practical life are usually too complicated to be represented by neat, compact expressions.

Especially when it comes to understanding minds, we still know so little that we can't be sure our ideas about psychology are even aimed in the right directions.

In any case, one must not mistake defining things for knowing what they are.