Livenotes A System for Cooperative and Augmented Note-Taking in Lectures Matthew Kam, Jingtao Wang, Alastair Iles, Eric Tse, Jane Chiu, Daniel Glaser,

LivenotesA System for Cooperative and

Augmented Note-Taking in Lectures

Matthew Kam, Jingtao Wang, Alastair Iles, Eric Tse, Jane Chiu, Daniel Glaser, Orna Tarshish and John Canny

University of California, Berkeley, USA

Video

Outline

Motivation Solution Experiment Results Implications Conclusion

Motivation: Problem Statement Constructivism

Learners are not blank slates that teachers write on Learners need to actively construct their own

understanding and knowledge

But large lecture classes are not conducive for active learning Passive mode of oral dissemination Lack of interactivity among students Lack of interactivity with instructor

Motivation: Precedents Face-to-Face Tutored Video Instruction (TVI),

Distributed TVI (Gibbons, Stanford) Group review of pre-recorded lectures Regular pauses for small-group discussion Students using DTVI received grades 0.5 std dev higher

than non-TVI students (Smith et al. 1999)

Peer Instruction (Mazur, Harvard) Lecture pauses for small-group discussion with neighbors Improvements in conceptual understanding and problem-

solving (Crouch and Mazur 2001)

Motivation: Small-Group, Cooperative Learning

More than 375 research studies since 1898 (Johnson and Johnson 1989)

Cooperative group learning results in greater Efforts to achieve Higher-level reasoning Transfer from original context to new situations Generation of new ideas and solutions

Motivation:Background Lecture Notes

(Hartley 1978, Kiewra et al. 1988) Experiments on note-taking that compared students annotating over:1) Complete lecture notes provided by instructor, vs.

2) Skeletal (i.e. partial) notes, vs.

3) No background notes

Results: students were found to achieve maximum retention with skeletal notes

Livenotes Recap

Both a technology and educational practice Large lecture classes Small-group discussions in ongoing lecture

Cooperative note-taking: Combines real-time note-taking with discussion

Augmented note-taking: Skeletal slides for students to annotate over

Related Systems No interaction between students

Classroom Presenter (Washington) StuPad, eClass (Georgia Tech)

No real-time interaction between students, i.e. sharing of notes takes place after lecture NotePals (Berkeley)

Limited real-time interaction between students OneNote (Microsoft)

Outline

Motivation Solution Experiment Results Implications Conclusion

Livenotes Evolution

2000: Implemented in Java, for WinCE Clios

Late 2000 to early 2003: 5 small-scaledeployments using Clios, laptops and Tablet PCs

Spring 2003: Medium-scale experiment in undergraduate class using Tablet PCs

Since 2003: Ported to Microsoft .NET

Livenotes User Interface

Pen and keyboard input

Group awareness (e.g. each user’s page number)

Import background slides

Unique user colors

Client-Server Topology

802.11b networking

Large class broken down into many small groups (3-7 students)

One Tablet per group is set to server mode

Other members’ Tablets connect wirelessly to group’s server

Server

Clients

Server

Clients

Group n:

Group 1:

…

Outline

Motivation Solution Experiment Results Implications Conclusion

Hypotheses

Cooperative note-taking:Shared whiteboard interface enhances learning through cooperative note-taking and discussion

Augmented note-taking:Background slides enhances learning by augmenting student note-taking

Experiment

Spring 2003 undergraduate HCI class

21 volunteers, randomly partitioned into Cooperative note-takers Individual note-takers

(control group) 4 weeks (7 lectures)

Preloaded skeletal PowerPoint slides

Previous Observation

From 5 previous deployments, we learned that Graduate students engaged

spontaneously in group discussions

Undergraduates were not used to discussing lecture material with one another

For this experiment (with undergraduates), we held short, live group discussions in the classroom

Data Collection

Short quizzes (4 lectures) Survey questionnaires

First week of deployment (~38% response rate) End of semester (~29%)

Qualitative interviews (3 users) Transcripts of students’ notes (~1581 pages)

Quantitative Analysis Unit of analysis: mark

Spatio-temporally contiguous segment of user input

E.g.: “This lecture is very interesting”

Quantitative hand-coding of ~1581 pages

Taxonomy of Marks

Note-taking: someone taking notes on lecture

Commentary: someone making a statement

Question: someone soliciting a response

Answer: response to a question, clarification

Reinforcement: contribution to an existing thread

Outline

Motivation Solution ExperimentResults Implications Conclusion

Cooperative Note-Taking:Richer Notes Cooperative note-taking group engaged in more

than twice as much activity as individual note-taker

Cooperative Note-Taking:Richer Notes Almost one quarter of marks made by cooperative

note-takers were attributed to group interaction

Student Learning

Survey question: “How did Livenotes, if at all, assist your learning in lecture?”

Early survey after 2 sessions with Livenotes:75% of respondents self-reported affirmatively

Survey after semester (i.e. 7 lecture sessions): 83% of respondents self-reported affirmatively

Cooperative Note-Taking: Taking Turns to Take Notes

66% of survey respondents agreed that cooperative note-taking is more useful

“Someone else might note something that I missed or hadn’t realized.”

“I liked how note-taking became a cooperative effort … someone can take over if another user is still inputing some notes, but the prof [had] moved on already.”

Cooperative Note-Taking: Paying Greater Attention

36% of students who self-reported learning benefit explicitly attributed that to social aspect of cooperative note-taking:

“Helped me to focus more in lecture. Often I fall asleep/lose attention in lecture. Having group members to respond to kept me better on track.”

Cooperative Note-Taking: Dual Conversations

Need to keep up with both lecture and on-tablet conversation:

“It is helpful to be able to discuss questions. However, this does take attention away from the lecture if you are focusing on answering/asking a question.”

Cooperative Note-Taking: Decreasing Distraction

Is “running Livenotes during class distracting?” (1 = extremely distracting, 5 = not distracting at all) Survey after two lectures: 2.6 out of 5 Survey after deployment ended: 3.83 out of 5

From student notes, “playful” behavior were observed to disappear almost completely after 2 lectures

Cooperative Note-Taking:Unanswered Group Questions

Students did not have time to answer some questions because they needed to keep up with lecturer

Some questions were unanswered because no group member knew the answer

Cooperative Note-Taking:Interaction During Pauses Group interaction during pauses in lecture accounted

for over half of group activity

Redeeming PowerPoint

Criticisms leveled at Microsoft PowerPoint The Cognitive Style of PowerPoint (Edward Tufte) Death by PowerPoint webpage I Powerpoint (David Byrne, Talking Heads)

Too boring, passive, does not promote active engagement with material

But students commented that augmented note-taking is “like having a conversation with the professor”

Augmented Note-Taking: Observed Behaviors

Elaborated on bullets Appended bullets to list Concurred and disagreed with bullet Noted gist of HCI principles Noted advantage and disadvantage of HCI

technique Answered questions in bullets

Augmented Note-Taking:Elaborated on Bullets

Augmented Note-Taking:Appended Bullets to List

Augmented Note-Taking:Answered Questions in Bullet

Augmented Note-Taking: Answered Questions in Bullet

Students responded to questions in bullets even when when they were not cooperative note-takers

Each group responded to 35% of the questions

Each question received a response from 36% of the groups

Augmented Note-Taking: Student Learning

Several high-quality notes in both individual and cooperative note-taking groups resulted from students “working off” bullets Possibly due to bullets focusing student attention to

relevant portions of lecture

A larger proportion (55%) of students who self-reported learning benefit attributed it to augmented note-taking, compared to cooperative note-taking. Half of this sub-group attributed that to having slides

at hand to annotate over

Quiz Scores

Quiz 1 Quiz 2 Quiz 3 Quiz 4

C [1] I [2] C I C I C I

Mean Score [3]

63.9 66.0 71.2 75.0 58.5 67.4 53.2 55.7

Std. Dev. 9.2 7.3 12.1 n/a 9.5 11.5 8.6 24.4

Sample size 8 4 9 1 9 4 7 4

P-value 0.702 0.771 0.168 0.810

No statistical significance But sample size was too small due to poor

attendance at end of semester

[1] Cooperative note-takers.

[2] Individual note-takers.

[3] Quiz scores presented in this table are normalized on a scale of 100.

Outline

Motivation Solution Experiment Results Implications Conclusion

Student-Instructor Interaction

To help instructor assess student learning, we deployed feedback feature in last two sessions

Students provide instructor with real-time, anonymous lecture feedback

Recently allowed students to alert instructor that they have questions

Recommendation:Background Slides as Scaffold

Bullets are a lightweight means for lecturer to engage actively with students during class Posing questions Counter-intuitive bullets Provocative statements

Direct student attention to critical parts of lecture E.g. prompts such as “Pros?” and “Cons?” with

blank spaces for students to fill in

Outline

Motivation Solution Experiment Results ImplicationsConclusion

Conclusion Cooperative note-taking

Richer variety of notes, higher-order thinking More than twice as much notes as individuals Members took turns to take notes Students kept awake to interact with group

Augmented note-taking Observed dialogue with bullets Reflected higher-order thinking High-quality notes resulted from “working off” bullets

Acknowledgement Corporation for Education Network Initiatives in

California Microsoft Research National Science Foundation Qualcomm

Volunteers from Computer Science 160, Spring 2003

Public domain source code by James R. Weeks

Questions?

Livenotes can be downloaded from:http://www.cs.berkeley.edu/~mattkam/livenotes

Matthew Kam, Ph.D. studentDepartment of Electrical Engineering and Computer Sciences,

and Berkeley Institute of Design

University of California at Berkeley, USA

[email protected]