1 Cognitive Analysis of Student Learning Using LearnLab Brett van de Sande, Kurt VanLehn, & Tim Nokes University of Pittsburgh.

1

Cognitive Analysis of Student Learning Using LearnLab

Brett van de Sande, Kurt VanLehn, & Tim Nokes

University of Pittsburgh

2

Agenda

I. LearnLab methodology

II. Demonstration of Andes,

an intelligent homework tutor

III. Log File Analysis

3

Goal: To understand physics learning

• Policy level– e.g., Physics for high school freshman?

• Instructional level– e.g., How much assistance to give?– e.g., How much practice per topic?– e.g., How to handle errors?

• Neurocognitive level– e.g., Can neuroimaging distinguish deep from

shallow studying of a text?

Our focus

4

Traditional methodsfor studying learning

• Design experiment– Modify text, classroom activities, tests…– e.g., Project Scale-up

• Lab experiment– Modify just one factor– Brief; money instead of grades, …

5

PSLC methods

• Educational data mining– Logs from instrumented courses– Some analysis is automated

• In vivo experiments– Control of variables– Instrumented courses

Next

6

Instrumented courses (Called LearnLab courses)

• Existing class + data collection– Homework done on a tutoring system or

photocopied and analyzed– Photocopies of quizzes, exams– FCI given before and after the course– Demographics, GPAs, Majors…– Handouts, slides, clicker data,…

• Instructor, student & IRB cooperation– Anonymity

7

Existing Physics LearnLab Course(s)

• US Naval Academy– Course take by all 2nd year students– LearnLab is in 4 of about 20 sections– Profs. Wintersgill, McClanahan

• Your course here

8

Basic data mining question

• What features of students’ histories are statistically associated with learning gains?

• e.g., What are the differences between histories of Student A and Student F?

Student A: 25% on pretest

Semester-long history 85% on post-test

Student F: 25% on pretest

Semester-long history 20% on post-test

9

Knowledge decomposition hypothesis

• Decompose knowledge to be learned into a set of knowledge components– e.g., Newton’s third law– e.g., Centripetal acceleration

• Assume each knowledge component is learned independently– An approximation/idealization

10

Data mining with knowledge components (KCs)

Student KC Pre-test History Post-test

A 1 35% … 85%

A 2 15% … 10%

A 3 25% … 20%

B 1 50% … 20%

B 2 10% … 10%

B 3 25% … 80%

For each KC, find statistical associations between histories and gains.

11

History decomposition hypothesis

• Decompose the student’s history into events such that each event addresses only one (or a few) knowledge components.– Reading a paragraph about Newton’s 3rd law– Drawing a reaction force vector– Seeing the instructor draw a reaction force– Drawing a centripetal acceleration vector

• Assume that a KC’s learning gain depends only on that KC’s events

12

Events for 1 student on 1 KC(e.g., Newton’s 3rd law)

Time Context Behavior8/27/07 9:05 FCI item 3 Incorrect

8/27/07 9:12 FCI item 10 Incorrect

9/13/07 18:06 Textbook, pg. 111 Highlighted

9/13/07 21:11 Problem 5-11, drawing FBD

Omitted force on the hand due to block

9/14/07 9:12 Lecture, slide 20 Taking notes

9/15/07 22:05 Problem 5-11, drawing FBD

Draws force on the hand due to block

etc.

13

Some events are not currently available

Time Context Behavior8/27/07 9:05 FCI item 3 Incorrect

8/27/07 9:12 FCI item 10 Incorrect

9/13/07 18:06 Textbook, pg. 111 Highlighted

9/13/07 21:11 Problem 5-11, drawing FBD

Omitted force on the hand due to block

9/14/07 9:12 Lecture, slide 20 Taking notes

9/15/07 22:05 Problem 5-11, drawing FBD

Draws force on the hand due to block

etc.

14

More feasible data mining

• Predict learning gains of a KC given the sequence of events relevant to that KC

• On an event that assesses mastery of a KC, predict the student’s performance during that event given the sequence of preceding events relevant to that KC

15

Predicting correctness ofevents that assess mastery

Context Event type P(Correct)FCI item 3 Assessment 0.10

FCI item 10 Assessment 0.12

Reading textbook, pg. 111, paragraph 3

Instruction Not applicable

Problem 5-11, drawing force on hand due to block

Assessment 0.30

Lecture, slide 20 Instruction Not applicable

Problem 5-11, drawing force on hand … with remedial feedback if needed

Assessment then instruction

0.55

16

Learning curves

• Plot assessment events on x-axis– Ordered chronologically

• Plot measure of mastery on y-axis– Usually aggregated across subjects

e.g., proportion of 100 subjects who performed correctly on this event

17

An expected learning curveF

requ

ency

of

corr

ect

0

0.5

1.0

1 2 3 4 5 6 7 8

Assessment events

18

Summary of PSLC educational data mining

• Given knowledge to be learned– Decompose into knowledge components

• Given students’ histories from an instrumented course– Divide into assessment/instruction events– such that one KC (or a few) per event

• For each KC, find a function on a sequence of events that predicts the KC’s– learning gain during the course– learning curve

19

PSLC methods

• Educational data mining– Logs from instrumented courses– Some analysis is automated

• Andes produces logs with KCs • DataShop draws learning curves, etc.

• Correlation ≠ Causation• In vivo experiments

– Control of variables– Instrumented courses

Next

20

Two major typesof in vivo experiments

• Short & fat– During one lesson or one unit

• Long & skinny– During whole course– “invisibly”

21

Example of a short, fat, in vivo experiment (Hausmann 07)

• During a 2-hour period (usually used for lab work)

• ~25 students in the room, each with a laptop and a headset mike

• Repeat 3 times: – Study a video while explaining it into the mike– Solve a problem

• 4 experimental conditions, varying the content of the video and the instructions for explaining it

• Random assignment of students to conditions• Dependent measures include learning curves• Result: Instructions to self-explain worked best regardless

of content of the video

22

Example of a long, skinny in vivo experiment (Katz 07)

• During 8 weeks of a 13-week course• Random assignment to 2 conditions:

– Experimental group: After solving certain homework problems, the student discussed the solution with a natural language tutoring system

– Control group: Extra homework problems

• Result: Experiment > Control on some conceptual measures

23

Robust Learning

• Immediate learning– During an immediate post-test– Similar content to training (near transfer)

• Robust learning– Far transfer– Retention– Acceleration of future learning

• Does manipulation of instruction on topic A affect rate of learning of a later topic, B?

24

Summary of PSLC methodology

• Data mining– Instrumented (LearnLab) courses– Knowledge components– Instructional and assessment events– Learning curves

• In vivo experiments– Short & fat vs. long & skinny– Robust learning

25

Agenda

I. LearnLab methodology

II. Demonstration of Andes,

an intelligent homework tutor

III. Log File Analysis Next

26

27

Define variables

Draw free body diagram (3 vectors and body)

Define coordinates (3 choices for this problem)

Upon request, Andes gives hints for what to do next

28

Principle-based help for incorrect entry

Red/green gives immediate feedback for student actions

29

30

# Log of Andes session begun Tuesday, July 17, 2007 12:12:28 by [User] on [Computer]

...05:03 DDE (read-problem-info "S2E" 0 0) ...02:35 Axes Axes-671 64 335 143 29602:35 Axes-dlg Axes-671 || …02:38 C dir 4002:42 BTN-CLICK 1 OK02:42 DDE (assert-x-axis NIL 40 Axes-671 "x" "y" "z")02:42 DDE-COMMAND assoc step (DRAW-AXES 40)02:42 DDE-COMMAND assoc op DRAW-VECTOR-ALIGNED-AXES02:42 DDE-COMMAND set-score 3902:42 DDE-RESULT |T| ...10:02 E 0 F1_y+F2_y=010:02 EQ-SUBMIT 010:02 DDE (lookup-eqn-string "F1_y+F2_y=0" 0)10:47 DDE-COMMAND assoc parse (= (+ Yc_Fn_BALL_WALL1_1_40

Yc_Fn_BALL_WALL2_1_40) 0)10:47 DDE-COMMAND assoc error MISSING-FORCES-IN-Y-AXIS-SUM10:47 DDE-COMMAND assoc step (EQN (= (+ Yc_Fw_BALL_EARTH_1_40

Yc_Fn_BALL_WALL2_1_40 Yc_Fn_BALL_WALL1_1_40) 0))10:47 DDE-COMMAND assoc op WRITE-NFL-COMPO10:47 DDE-RESULT |NIL| ...10:50 DDE-RESULT |!show-hint There is a force acting on the ball at T0 that you

have not yet drawn.~e| ...16:38 END-LOG

problem name

session time

student action (equation)

error analysis:intended action

student action (draw axes)

interpretation:compare to model

green

red

31

Demonstration by Tim Nokes# Log of Andes session begun Wednesday, April 18, 2007 21:08:07 by [user] on [computer]...0:02 DDE (read-problem-info "FARA9" 0 0)...0:13 Help-Hint 0:13 DDE (Get-Proc-Help)0:13 DDE-COMMAND assoc (NSH NEW-START-AXIS 0)0:13 DDE-RESULT |!show-hint It is a good idea to begin most problems by drawing an

axis. This helps to ground your work and will be useful later on in the process.~e|…0:17 Begin-draw 50001 Axes-1 185 331...0:30 New-Variable resistance...0:39 DDE (define-variable "R" |NIL| |resistance| |R| |NIL| |NIL| Var-2 "20 ohm")0:39 DDE-COMMAND assoc step (DEFINE-VAR (RESISTANCE R))0:39 DDE-COMMAND assoc op DEFINE-RESISTANCE-VAR0:39 DDE-COMMAND assoc parse (= R_R (DNUM 20 ohm))0:39 DDE-COMMAND set-score 30:39 DDE-RESULT |T|....0:50 DDE (lookup-vector "B" Unspecified B-field |s| NIL 0 |NIL| Vector-3)0:50 DDE-COMMAND assoc entry (VECTOR (FIELD S MAGNETIC UNSPECIFIED

TIME NIL) ZERO)0:50 DDE-COMMAND assoc error DEFAULT-SHOULD-BE-NON-ZERO0:50 DDE-COMMAND assoc step (VECTOR (FIELD S MAGNETIC UNSPECIFIED TIME

NIL) OUT-OF)0:50 DDE-COMMAND assoc op DRAW-FIELD-GIVEN-DIR0:50 DDE-COMMAND set-score 20:50 DDE-RESULT |NIL|...9:51 DDE-RESULT |T|9:55 END-LOG

32

Agenda

I. LearnLab methodology

II. Demonstration of Andes,

an intelligent homework tutor

III. Log File Analysis Next

33

Model Solution Set

Solution 0

Principle A

Op1

Op3

Op6

Op7

Principle B

Op2

Op3

Op5

Op8

Op10

Solution 1

Principle C

Op10

Op11

Op12

Principle A

Op1

Op3

Op6

Op7

Principle D

Assumption: Opi = KC

34

# Log of Andes session begun Friday, July 27, 2007 14:29:38 by bobh on BOBH

…

0:02 DDE (read-problem-info "S2E" 0 0)

…

11:45 Vector-dlg Vector-673 ||

…

11:48 CLOSE type instantaneous

11:48 SEL type 1 instantaneous

11:51 BTN-CLICK 1 OK

11:51 DDE (lookup-vector "a" instantaneous Acceleration |ball| NIL 0 |T0| Vector-673)

11:51 DDE-COMMAND assoc step (VECTOR (ACCEL BALL :TIME 1) ZERO)

11:51 DDE-COMMAND assoc op ACCEL-AT-REST

11:51 DDE-RESULT |T|

…

problem name

student actions

match model solution:assoc step = entryAssoc op = operator

green

35

14:03 E 8 Fearth_y = m*g

14:11 EQ-SUBMIT 8

14:11 DDE (lookup-eqn-string "Fearth_y = m*g" 8)

14:11 DDE-COMMAND assoc parse (= Yc_Fw_BALL_EARTH_1_0 (* m_BALL g_EARTH))

14:11 DDE-COMMAND assoc error MISSING-NEGATION-ON-VECTOR-COMPONENT

14:11 DDE-COMMAND assoc step (EQN (= Fw_BALL_EARTH_1 (* m_BALL g_EARTH))),(EQN (= Yc_Fw_BALL_EARTH_1_0 (- Fw_BALL_EARTH_1)))

14:11 DDE-COMMAND assoc op WT-LAW,COMPO-PARALLEL-AXIS

14:11 DDE-COMMAND set-score 74

14:11 EQ-F 8

14:11 DDE-RESULT |NIL|

student actions

red

guessintended

errorinterpretation

36

Review Video

Match steps in video to log file

37

Researchable questions

Timing Sequencing(order of steps)

Hint Usage

Problem solving skills

Errors as window to mental state

Self-correction of errors

38

DataShophttps://learnlab.web.cmu.edu/

> Launch DataShop> New user? Sign up now!> (Create account)