Learning Analytics: Realizing their Promise in the California State University

John Whitmer, CSU Office of the Chancellor & CSU ChicoKate Berggren, CSU Northridge

Hillary Kaplowitz, CSU NorthridgeTom Norman, CSU DH

Learner Analytics Realizing their Promise in the CSU

Download slides at: http://bit.ly/HqaHBF

Outline

1. Promise of Learner Analytics

2. Tools & Systems in Practice

3. CSU Case Studies:• Analytics at Work in the Classroom (Hillary)• GISMO & SQL Query Tools (Kate)• Vista in RELS 180 (John)

4. Q & A

1. PROMISE OF LEARNER ANALYTICS

Steve Lohr, NY Times, August 5, 2009

http://1.usa.gov/GDFpnI

Draft DOE Report released April 12

Economist. (2010, 11/4/2010). Augmented business: Smart systems will disrupt lots of industries, and perhaps the entire economy. The Economist.

7

Source: jisc_infonet @ Flickr.com

Source: jisc_infonet @ Flickr.com

8

What’s different with Big Data? 4 V’s:1. Volume2. Variety3. Velocity4. Variability

(IBM & Brian Hopkins, Forrester)

Academic Analytics

“Academic Analytics marries large data sets with statistical techniques and predictive modeling to

improve decision making”

(Campbell and Oblinger 2007, p. 3)

10

Academic Analytics1. Term adopted in 2005 ELI research report

(Goldstein & Katz, 2005)

– Response to widespread adoption ERP systems, desire to use data collected for improved decision making

– 380 respondents; 65% planned to increase capacity in near future

2. Call to move from transactional/operational reporting to what-if analysis, predictive modeling, and alerts

3. LMS identified as potential domain for future growth

DD Screenshot

Learner Analytics:

“ ... measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs.” (Siemens, 2011)

or said plainly:

What are students doing?

Does it matter?

Learner Analytics1. Analyze combinations of data including:

– Frequency of ed tech usage (e.g. clickstream analysis)– Student learning “outputs” (e.g. quiz scores, text answers)– Student background characteristics (e.g. race/ethnicity)– Academic achievement (e.g. grades, retention, graduation)

2. Current rsch: mostly data mining, not hypothesis-driven

3. More complex than Academic Analytics, considering: – Immaturity of ed tech reporting functionality– Translation of usage into meaningful activity– No significant difference: not what technology used, it’s how it’s

used, who uses it, and for what purpose

A few promises of analytics for faculty and students …

1. Provide behavioral data to investigate student performance

2. Inform faculty about students succeeding or at risk of failing a course

3. Warn students that they are likely to fail a course – before it’s too late

4. Help faculty evaluate the effectiveness of practices and course designs

5. Customize content and learning activities (e.g. adaptive learning materials)

What’s the promise of analytics for academic technologists?

1. Decision-making based on actual practices (not just perceptions) and student outcomes

2. Support movement of A.T. into strategic role re: teaching and learning by:– demonstrating the link between technology

and learning– distinguishing our role from a technology

infrastructure provider

Our 2 biggest barriers

Image Source: http://bit.ly/Hq9Cdg

Image Source: Utopian Inc http://bit.ly/Hq9sCq

Image Source: Privacy in the Cloud: http://bit.ly/HrF6zk

2. TOOLS & SYSTEMS IN PRACTICE

SIGNALS

http://www.itap.purdue.edu/studio/signals/Purdue Signals Project

SNAPP

http://www.snappvis.org/SNAPP (Social Networks Adapting Pedagogical Practice)

KHAN

http://www.khanacademy.org/Khan Academy

OLI

http://oli.web.cmu.edu/openlearning/initiative/processCM Open Learning Initiative

PARCHMENT

http://www.parchment.com/c/my-chances/Parchment

3. CSU CASE STUDIES

27

ANALYTICS AT WORK IN THE CLASSROOM (HILLARY)

28

How can data help teachers and students?

Two stories about how data helped students and teachers work better together

29

30

31

“Hey Professor,

I just looked at my assignments and realized that my Chapter 11 summary did not get submitted, which I'm having trouble believing that I didn't submit it... especially because I see that I did it, and I always submit my assignments as soon as I finish them.”

32

Now the hard part….

Do I believe him? If I only I could check…

33

34

35

And it was all his idea…

The student suggested that I check Moodle and if that didn’t work told me how to check the Revision History in GoogleDocs with step-by-step directions!

36

37

Hybrid Course Weekly Structure

1. Watch lectures

2. Read textbook

3. Online chat and tutoring

4. Post questions and take practice

quiz

4. Class meets

5. Aplia quiz

38

“The quiz is unfair”

39

But the story was not that simple…

»Reports on Moodle painted a different picture»Student was watching the lectures at 10:00 p.m.»Then immediately taking quiz

40

Enabled constructive feedback…

1. Advised the student how the structure of the course was designed to enhance learning

2. Student revised their study habits3. Improved grades and thanked the instructor!

41

What we can do with data now

1. Use Reports in Moodle to verify student claims2. Review participant list to see last access time3. Empower students to review their own reports4. Analyze usage and advise students how to study better5. Review quiz results to find common misconceptions

42

And if we had better tools that are easier to use…

1. Let our students see more details about how their habits affect their grades and encourage them to use them

2. Give instructors access to more information and better tools to organize data so they can see patterns of access and time on task and how they relate to outcomes

3. Have tools that red flag students with teacher set criteria 4. Help streamline workflow for instructors by organizing

student information– View all ungraded assignments

43

Could we help improve student learning outcomes if we knew the effect of…

Coffee

Time

Amount

Textbook

LMS AccessLMS Activities

Mobile

Attendance

Friends

44

GISMO & SQL QUERY TOOL (KATE)

45

GISMO – Course Block

46

GISMO – Access Overview

47

GISMO – Access by Student

48

GISMO – Quiz Overview

49

SQL Query Tool

50

List of Contributed Queries

51

Query Example

52

Query Results: Most Active Courses

53

Query: Most Popular Activities

54

Query: Systemwide use of

Activities and Resources

55

Query:Forum Use Count by Type

Learner Analytics

Thomas J. NormanCalifornia State University

Dominguez Hills

eBook

A New Way of Reading

From Textbooks to Apps

Assignments

Grading To Do List

Real Time Metrics

Warnings

LearnSmart Progress

Analysis by AACSB Categories

Bloom’s Taxonomy

Performance by Learning Objective/Difficulty

Ideas? Questions?

• [email protected]

• [email protected]

• 310-243-2146

72

CSU CHICO VISTA ANALYTICS

73

LMS Learner Analytics @ Chico StateCampus-wide

– How are faculty & students using the LMS? – What meaningful activities are being conducted? – How does that usage vary by student background, by college, by

department?

Course level– What is the relationship between LMS actions, student background

characteristics and student academic achievement? (6 million dollar question)

– Intro to Religious Studies: redesigned in Academy eLearning, increased enrollment from 80 to 327 students first semester

Ultimate goal: provide faculty and administrators with what-if modeling tools to identify promising practices and early alerts

74

Chart from Scott Kodai, Chico State

CSU Practice

INTRODUCTION TO RELIGIOUS STUDIES (RELS 180)

80

CLOSING THOUGHTS

Call to Action

1. Metrics reporting is the foundation for Analytics

2. Don’t need to wait for student performance data; good metrics can inspire access to performance data

3. You’re *not* behind the curve, this is a rapidly emerging area that we can (should) lead ...

4. If there’s any ed tech software folks in the audience, please help us with better reporting!

Want more? Resources on Analytics

Googledoc: http://bit.ly/HrG6Dm

83

Q&A and Contact Info

Resources Googledoc: http://bit.ly/HrG6Dm

Contact Info: • John Whitmer ([email protected])• Hillary C Kaplowitz ([email protected])• Berggren, Kate E ([email protected])

Download presentation at: http://bit.ly/HqaHBF