Using Analytics to Increase Student Success

Getting Students To Do the WorkUsing Learner Analytics Data to Inform Instructional StrategiesNew Directions in Online Learning San Diego, 2014 David Lindrum

Study Questions vs. Assignments

WEBTEXT

Study Questions vs. Assignments

Study Questions vs. Assignments

WEBTEXT

WEBTEXT

Study Questions vs. Assignments

380 Study Questions

WEBTEXT

Study Questions vs. Assignments

• 33 Audio / Video Selections

• 34 Articles and Studies

• 8 Infographics, Websites, etc.

WEBTEXT

Assignments in each chapter

Study questions on each page

Study Questions vs. Assignments

Data Collected From Student Activities

Log-In

- first enrollment - each log-in time - most recent date

Active Time

Time spent on each page typing, clicking, or scrolling.

Participation

- pages viewed - questions answered - number of resets

Scoring

- on multiple choice - first attempt - most recent attempt

Higher Order

- assignments - short answer - response boards

About Our DataIn two studies, assignment scores (non-resettable, application-type questions) are significant predictors of midterm and final scores. Other webtext activities are also correlated with course success, including minutes per question and question completion.

In multiple online courses (over 8,000 students), webtext success pass closely matches course pass rate.

A Tale of Two Courses

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Webtext Completion*PSY Pilot Instructor 1

*N=35 students; excludes 5 students with 0 time.

Percentage Complete

median✦ 21% webtext

completion ✦ 102 questions

answered ✦ 8.8 hours

active time ✦ 6.2 minutes

per correct question

PSYSummer 2012

median

!

Just 20-30% of students do assigned reading. (Hobson, 2004)

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Webtext Completion*

*N=46 students; excludes 1 student with 0 time.

median✦ 99% webtext

completion ✦ 475 questions

answered ✦ 16.4 hours

active time ✦ 2.9 minutes

per correct question

PSYFall 2012

PSY Pilot Instructor 2

median

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Webtext Completion*

*N=46 students; excludes 1 student with 0 time.

median✦ 99% webtext

completion ✦ 475 questions

answered ✦ 16.4 hours

active time ✦ 2.9 minutes

per correct question

PSYFall 2012

PSY Pilot Instructor 2

median

30% Course Credit for Webtext Assignments and Study Questions

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Webtext Completion*PSY Pilot Instructor 1

*N=35 students; excludes 5 students with 0 time.

Percentage Complete

median✦ 21% webtext

completion ✦ 102 questions

answered ✦ 8.8 hours

active time ✦ 6.2 minutes

per correct question

PSYSummer 2012

median

0% Course Credit for Webtext Assignments and Study Questions

Students don’t do optional

Assign credit for formative activities that will prepare students to do well on final.

Credit should generally be between 20-30% of course credit.

What We Learned

Students don’t do optional

Assign credit for formative activities that will prepare students to do well on final.

Credit should generally be between 20-30% of course credit.

Students respond to personal outreach (nagging, parenting)

Instructor 2 emailed students at the end of every week

Go get your learning resource. You can’t pass this class without it.

I see that you’re falling behind. If you don’t catch up, you won’t pass this class.

What We Learned

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Webtext Completion*PSY Pilot Instructor 1

*N=35 students; excludes 5 students with 0 time.

Percentage Complete

median✦ 21% webtext

completion ✦ 102 questions

answered ✦ 8.8 hours

active time ✦ 6.2 minutes

per correct question

PSYSummer 2012

median

What exactly are successful-student behaviors?

Successful Student Behaviors

1. Successful students do optional activities like study questions.

Assignments

Study Questions

Polls

Your Turns

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

A’s B’s C’s D’s F’s

Median Webtext Completion PercentagesBy Assignment Scores

Fall 2012POL

No credit

n=48 at R1, f2f

“[L]earning techniques such as taking practice tests and spreading study sessions out over time — known as distributed practice — were found to be of high utility because they benefited students of many different ages and ability levels and enhanced performance in many different areas.” Improving Students’ Learning with Effective Learning Techniques (Dunlosky, et al. 2013)

Credit buys time. !(and time is fixed.)

2. Successful students complete more work.

Assignments

Study Questions

0 10 20 30 40 50 60 70 80 90 100

A’s B’s C’s D’s F’s

Average Webtext CompletionBy Final Grade POL

n=57 at R1, f2f

0

10

20

30

40

50

60

70

80

90

100

Ch 1 Ch 2 Ch 3 Ch 4 Ch 5 Ch 6

Median Webtext Scores (Lines)

Fs

Bars represent completion. !Lines represent percentage of points earned per module. !

PSYAs of October 28, 2013

Ds

Cs

Bs

As

n=539

0

10

20

30

40

50

60

70

80

90

100

Ch 1 Ch 2 Ch 3 Ch 4 Ch 5 Ch 6

Median Webtext Scores (Lines) and Completion (Bars)

Fs

Bars represent completion. !Lines represent percentage of points earned per module. !

PSYAs of October 28, 2013

Ds

Cs

Bs

As

n=539

3. Successful students invest more time.

Webtext Hours

0 4.5 9 13.5 18

A’s B’s C’s D’s F’s

Median Active Time in Webtext - HoursBy Webtext ScoresAs of October 28, 2013

ORG

n=1,040 at online

0

0.8

1.6

2.4

3.2

4

0%

20%

40%

60%

80%

100%

As Bs Cs Ds Fs

Assignment Score Assignment Completion Hours

Median Assignment TIme and Scores

As of October 28, 2013HISBy Webtext Scores

n=560 at online

0

50

100

150

200

250

300

350

400

450

500

0 500 1000 1500 2000 2500 3000 3500 4000 4500 5000

Median Time

Median Correct

Active Time in Minutes

Co

rrec

t R

esp

on

ses

Active Time and Correct Responses All Students, All Questions (large, 2-year)

4. Successful students start working before the

first day of class.

0

1.25

2.5

3.75

5

0%

20%

40%

60%

80%

100%

Week 1 Week 2 Week 3 Week 4 Week 5 Week 6

Pre-Term Score Pre-Term Hours Week 1 Score Week 1 Hours Week 2+ Score Week 2+ Hours

Median Webtext Scores By Week

As of October 28, 2013PSYWith Median Hours Per Week

n=539 at online

Successful Instructor Behaviors

Follow up. (Which is why instructors need actionable analytics.)

WHAT IF WE KNEW STUDENTS WERE

DOING THE WORK?

Questions?

[email protected]