The Role of Learning Analytics: A personal Journey… 5/22/2013

The Role of Learning Analytics: A personal Journey…

5/22/2013

Oded Meyer Department of Mathematics and StatisticsGeorgetown University

Some background about myself…..

Learning Analytics

Data driven approach for the purpose

of understanding and improvinglearning and the and the

environmentin which it occurs.

Carnegie Mellon Open Learning Initiative (OLI)

Scientifically-based online learning environments based on the integration of technology and the science of learning with teaching. OLI is designed to simultaneously improve learning and facilitate learning research.

The OLI Statistics Course

Educational Mission of Funder (2002) (The William and Flora Hewlett Foundation) Provide open access to high quality post-secondary

education and educational materials to those who otherwise would be excluded due to:– Geographical constraints– Financial difficulties– Social barriers

To meet this goal:– A complete stand-alone web-based introductory

statistics course.– openly and freely available to individual learners online.

Important form of analytics: The Science of Learning

General:• Make the structure and “big picture” salient

Learning Science Principles - continued

• Immediate and targeted feedback students achieve desired level of performance faster.

Science of Learning - continued

Discipline specific principles:• Hands-on activities• Use real data• De-emphasize calculations.

Start Analytics Early in the Design process

After one module :

• Qualitative feedback• Observe students• Talk-allowed protocols

Instructor feedback loop analytics

• Feedback to the instructor about students’ learning Learning Dashboard

- Presents the instructor with a measure of student learning for each learning objective.

- More detailed information:• Class’s learning of sub-objectives• Learning of individual students• Common misconceptions

Learning Dashboard Team led by Dr. Marsha Lovett

Instructor feedback loop analytics

Key:• Clearly defined learning objectives.• Tying each activity to a learning objective.

Benefits:• Can reveal misconceptions• Impacts how your spend class time• Can reveal “expert blind spots”

Students who achieved proficiency in finding median did poorly in the following assessment item tied to the same sub-skill:

Larger Scale Analytics:

Assessing the Effectiveness of the Course When used in the Blended (Hybrid) Mode

Larger Scale Analytics: Assessing the Effectiveness of the Course

The Hewlett Foundation’s “AccelerateLearning Challenge”:

Can students using the OLI course in the blended mode learn the same material as

they would in the traditional course in shorter time and still have equal or better

learning gains.

Three accelerated Studies#1 Small class, expert instructor (2007)

#2 Replication with larger class (2009)With retention follow-up 4+ months later

#3 Replication with new instructor (2010)

Experienced statistics instructorNew to OLI Statistics course and hybrid

mode

Study 1: Method~180

studentsenrolled

68 volunteers for special

section

44 students,traditional

control condition

24 students,adaptive/

accelerated condition

Adaptive/Accelerated vs. Traditional

Two 50-minute classes/wkEight weeks of instructionHomework: complete OLI activities on a scheduleTests: Three in-class exams, final exam, and CAOS test

Four 50-minute classes/wkFifteen weeks of instructionHomework: read textbook & complete problem setsTests: Three in-class exams, final exam, and CAOS test

<<?=

Same content but different kind of instruction

Dependent Measure

CAOS = Comprehensive Assessment of Outcomes in a First Statistics course (delMas, Garfield, Ooms, Chance, 2006)

– Forty multiple-choice items measuring students’ “conceptual understanding of important statistical ideas”

– Content validity – positive evaluation by 18 content experts

– Reliability – high internal consistency– Aligned with content of course (both sections)– Administered as a pre/posttest

Study 1: CAOS Test Results

Adaptive/Accelerated group gained more (18% vs 3%)pre/post on CAOS than did Traditional Control, p < .01.

Chance

These analytics got the attention of education leaders in the U.S. who are facing the “cost disease” in higher ed.

William Bowen (former President of Princeton)replicated our study in the “Interactive Learning Online at Public Universities” study.

The study further indicated that blended learning offers the potential of more economical and rapid pathways to mastery.

Analytics on Students’ Learning Habits

• Students in both groups recruited to complete time-logs

• Self-report for both groups• Analogous point in the course (2/3 through)• Six consecutive days: Wednesday - Monday

Study 1: Time Spent Outside of Class

No significant difference between groups in the time students spent on Statistics outside of class

Study 2: Replication & Extension

• Same method, same procedure, same instructor

• Larger class (52 students in Adaptive / Accelerated)

• Follow-up retention study conducted 4+ months later

Study 2: CAOS Test Results

Adaptive/Accelerated group gained more pre/post on CAOS than did Traditional Control, p < .01.

Chance

Using Analytics to Assess Retention

Jan Feb Mar Apr May Jun Jul Aug Sep Oct

Follow-up Begins

Adapt/Acc EndsTrad’l Ends

Adapt/Acc Delay (13 Students) Trad Delay (14 Students)

Study 2, Retention: Re-taking CAOS

At 6-month delay, Adaptive/Accelerated group scored higher on CAOS than Traditional Control, p < .01.

Chance

Study 3: Further Replication & Extension

• Same method, same procedure

• New instructor• Not involved in development of OLI course• New to OLI statistics and hybrid teaching mode• Instructor held constant for both Adapt/Acc and

Control conditions

• Larger class (40 students in Adaptive / Accelerated)

Study 3: CAOS Test Results

0

10

20

30

40

50

60

70

Adapt/Acc Trad Control

Pretest

Posttest

Adaptive/Accelerated group gained more pre/post on CAOS than did Traditional Control, p < .01.

Chance

Current and Future Analytics

• Continued “gap analysis”.

• Better alignment between learning objectives/sub-objectives and activities.

• Student Dashboard provide learners with insight into their own learning habits and can give recommendations for improvement.

• Learning-facing analytics allows learners to compare their own performance against an anonymous summary if their course peers

Lessons Learned…

1. Pedagogy must drive technology and not the other way.

2. Developing online materials is a collaborative effort.

3. Developing online materials is an iterative process.

4. Steep learning curve.5. Growth as a teacher.

“Improvement in post-secondary education will require converting teaching from a ‘solo sport’ to a community-based research activity”

Herbert Simon, Last Lecture Series, Carnegie Mellon, 1998