Learning Analytics 101

June 10-15, 2012

Growing Community; Growing Possibilities

Steve Lonn, University of MichiganJosh Baron, Marist College

2012 Jasig Sakai Conference 2

1. What is Learning Analytics (LA)?

2. Current LA work in Higher Education

3. Data available in Sakai CLE & OAE

4. Big Questions to Ponder

5. Q & A

Slides Available: slideshare.net/stevelonn/

Agenda

BIG Data

“...datasets whose size is beyond the ability of typical database software tools to capture, store, manage, and analyze.”

Manyika et al. (2011)

Big Data in Higher Education

Analytics:

An overarching concept that is defined as data-driven

decision making

van Barneveld, Arnold, & Campbell, 2012adapted from Ravishanker

Analytics at Your Institution RIGHT NOW

Business / Academic Analytics:

A process for providing higher education institutions with

the data necessary to support operational and financial

decision making

van Barneveld, Arnold, & Campbell, 2012adapted from Goldstein and Katz

evidenceframework.org/big-data/

Educational Data Mining

Learning Analytics

Bienkowski, Feng, & Means, 2012◦ SRI International

2012 Jasig Sakai Conference

Dept. of Education Issue Brief

Generally emphasizes reduction into small, easily analyzable components◦ Can be then adapted to student by software

◦ Siemens and Baker, 2012

Predicting future learning behavior

Domain models for content / sequences

Software-provided pedagogical supports

Computational models that incorporate student, domain, and pedagogy

Example: Cognitive Tutors Pittsburgh Advanced Cognitive Tutor Center Carnegie

MellonUniversity

http://ctat.pact.cs.cmu.edu

Educational Data Mining:

A process for analyzing data collected during

teaching and learning to test learning theories and

inform educational practice

Bienkowski, Feng, & Means, 2012

Understand entire systems and support human decision making

Applies known methods & models◦ answer questions about learning and

organizational learning systems

Tailored responses◦ adapted instructional content, specific

interventions, providing specific feedback

Learning Analytics

Learning Analytics:

The use of analytic techniques to help target instructional, curricular, and support resources to support

the achievement of specific learning goals through

applications that directly influence educational practice

van Barneveld, Arnold, & Campbell, 2012adapted from Bach

Predictive Analytics◦ uncover relationships and patterns◦ can be used to predict behavior and events

Visual Data Analytics◦ discovering and understanding patterns in large

datasets via visual interpretation

Additional Terms

Terms are not ExclusiveTerm Definition Level of

Analytics An overarching concept that is defined as data-driven decision making

All levels

Academic Analytics

A process for providing higher education institutions with the data necessary to support operational and financial decision making

Institution

A process for analyzing data collected during teaching and learning to test learning theories and inform educational practice

Department / Instructor / Learner

Learning Analytics

The use of analytic techniques to help target instructional, curricular, and support resources to support the achievement of specific learning goals through applications that directly influence educational practice

Department / Instructor / Learner

Examples of LA ProjectsWho’s been working in this space in Higher Education?

Purdue University’s Course Signals

◦ College-wide learning analytics approach

University of Michigan’s E2Coach

◦ Course-specific learning analytics approach

UMBC’s “Check My Activity” Tool

◦ Student-centered learning analytics approach

Three Different Approaches…

Built predictive model using data from…◦ LMS – Events (login, content, discuss.) & gradebook

◦ SIS – Aptitude (SAT/ACT, GPA) & demographic data

Leverage model to create Early-alert

system◦ Identify students at risk to not complete the course

◦ Deploy intervention to increase chances of success

Systems automates intervention process◦ Students get “traffic light” alert in LMS

◦ Messages are posted to student that

suggest corrective action (practice tests)

Impact on course grades and retention◦ Students in courses using Course Signals…

scored up to 26% more A or B grades up to 12% fewer C's; up to 17% fewer D's and F‘s

Ellucian product that integrates w/Blackboard

Open Academic Analytics Initiative (OAAI)◦ Creating a similar Sakai-based OS solution2012 Jasig Sakai Conference

Arnold & Pistilli, 2012 - LAK

Focused specifically on introductory Physics Uses data from…

◦ Pre-course survey: academic info,

learner’s goals, psycho-social factors

◦ Performance: Exams, Web HW, Sakai

Michigan Tailoring System (MTS)◦ OS tool designed for highly customized messaging

◦ Used in health sciences for behavior change

◦ Messaging based on input from many sources

University of Michigan’s E2Coach

“…to say to each what we would say if we could sit down with them for a

personal chat.”

Example MTS Message

UMBC found that students earning D/F’s use Bb 39% lessthen higher grade achievers◦ Not suggesting cause and effect◦ Goal is to model higher achiever

behavior Provides data directly student

◦ Compare LMS use to class averages◦ Can also compare averages usage

data to grade outcomes Feedback has been positive

UMBC’s Check My Activity Tool

Student Success Plan – Sinclair CC◦ Holistic case-management system◦ Connects faculty, advisors, counselors, & students◦ Jasig Incubation Project

STAR Academic Journey – U of Hawaii◦ Online advising and degree attainment system

SNAPP – UBC/Wollongong◦ Visualize networks of

interaction resulting fromdiscussion forum posts andreplies

A Few Others Projects…

Papers and Articles on Purdue’s Course Signals http://www.itap.purdue.edu/learning/research/

Michigan’s Expert Electronic Coaching http://

sitemaker.umich.edu/ecoach/home

UMBC’s Check My Activity Tool http://

www.educause.edu/EDUCAUSE+Quarterly/EDUCAUSEQuarterlyMagazineVolum/Vi

deoDemoofUMBCsCheckMyActivit/219113

Student Success Plan http://

www.educause.edu/EDUCAUSE+Quarterly/EDUCAUSEQuarterlyMagazineVolu

m/TheStudentSuccessPlanCaseManag/242785

STAR Academy Journey http://

net.educause.edu/ir/library/pdf/pub7203cs7.pdf

SNAPP http://research.uow.edu.au/learningnetworks/seeing/snapp 2012 Jasig Sakai Conference

More information at…

Data Available in SakaiWhat can we know in CLE and OAE products?

User-level data stored as “events”

List of events available on Confluence◦ Search for “event table description”

CLE Data Overview

sakai_session SESSION_ID SESSION_USER SESSION_IP SESSION_USER_AGENT SESSION_START SESSION_END SESSION_SERVER SESSION_ACTIVE SESSION_HOSTNAME

sakai_eventEVENT_IDEVENT_DATEEVENTREFSESSION_ID

EVENT_CODECONTEXT

Site-level data stored in separate tables

CLE Data Overview

sakai_siteCUSTOM_PAGE_ORDEREDSITE_IDTITLETYPESHORT_DESCDESCRIPTIONICON_URLINFO_URLSKINPUBLISHEDJOINABLEPUBVIEWJOIN_ROLECREATEDBYMODIFIEDBYCREATEDONMODIFIEDONIS_SPECIALIS_USER

sakai_realmREALM_KEYREALM_IDPROVIDER_IDMAINTAIN_ROLECREATEDBYMODIFIEDBYCREATEDONMODIFIEDON

realm_id like '/site/' || site_id

W05 F0

06 F06

W07 F0

08 F08

W09 F0

10 F10

W11 F1

10,000

12,000

14,000

16,000

Project Sites Logarithmic (Project Sites)Course Sites Logarithmic (Course Sites)Max Users Logarithmic (Max Users)Logarithmic (Max Users)

Thanks to John Leasia

Site Creation & Concurrent Users

Overall Tool Usage

%1% 1% 1%

Presence Web Content ResourcesAttachments Test Center AssignmentsSyllabus Forums GradebookDrop Box Evaluations MessagesPreferences Basic LTI Site Info / SetupRealms Announcements ChatCalendar Digest ChecklistiTunes U Help ModulesEmail Archive Course Eval Help PodcastsWiki Polls NewsSign Up Library Materials OSPDiscussion Engin Honor Code UbookLibrary Help Super User DB HelpSearch Page Order Global Alert

Instructors Using / Not Using Sakai

Dental School

Subjects Using the Wiki Tool

Sum of RevisionsBITENGRNURSIOESIENGLISHRCHUMSAAASEECSRCLANGCOMPPSYCH

Count of Course Sites

SIENGLISHBITEECSPSYCHCOMPMODGREEKNURSNRELING

Summary information about site visits, tool activity, and resource activity

Site Stats Tool

Site Stats: Activity Detail

User-level data available via “activity feeds”◦ follows a “push and publication” model rather than

a “store and query” model (CLE is store & query)

◦ Activity is both highly specific: individual interactions between users, content, contexts…

◦ …and more general: user interaction everywhere rather than only within a single course context.

What new questions will we ask?◦ Interesting activity can happen with external

capabilities: CLE tools, LTI tools, widgets. How will we ensure this data is captured?

Analytics in OAE

Many thanks to Nate Angell for OAE slides

OAE activity stream design: feeds

FYI: Designs are still in draft form.

OAE activity stream design: user

OAE activity stream design: content

Activity (OAE) & Grades (CLE): Week 1

Student success in OAE & CLE

Developed by the Kaleidoscope Project in collaboration with rSmart.

Activity (OAE) & Grades (CLE): Week 7

Activity (OAE) & Grades (CLE): Animation

Tools / services to support analytics initiatives◦ Ways to connect different silos of data◦ Methods to connect back to CLE / OAE

LTI? Web services? Others?

OAE improvements over CLE approach to user data◦ What data is most relevant for analytics?◦ What displays and/or data are most useful to help

learners?

Developer Questions to Ponder

Some Big LA Questions: Research & Ethics

Josh Baron, Marist College

Data Mining vs. Learning Science Approaches◦ Do we build predictive models from large data sets or

from our understanding of learning sciences?◦ Is both the right answer? How does that work?

Challenges of Scaling LA Across Higher Ed◦ Does each institution have to build its own model?

How “portable” are predictive models?

◦ Do we need an open standard for LA? Could LIS and LTI play a role?

How can LA be used to assist ALL students?◦ Michigan’s E2Coach system is a good example

Big LA Research Questions

“The obligation of knowing” – John Campbell◦ If we have the data and tools to improve student

success, are we obligated to use them? Consider This > If a student has a 13% chance of

passing a course, should they be dropped? 3%? Who owns the data, the student?

Institution?◦ Should students be allowed to “opt out”?

Consider This > Is it fair to the other students if by opting out the predictive model’s power drops?

What do we reveal to students? Instructors?

Consider This > If we tell a student in week three they have a 9% chance of passing, what will they do? Will instructors begin to “profile” students?

Big LA Ethical Questions

Resources & Conference Sessions

Connect with Learning Analytics communities

http://www.solaresearch.org/

Learning Analytics & Knowledge Conferences (LAK)

STORM – initiative to help fund research projects

FLARE – regional practitioner conference ◦ Purdue University, Oct 1-3, 2012

Symposium on Learning Analytics at Michigan

http://sitemaker.umich.edu/slam/

15 speakers (12 UM, 3 external)

Videos & slides available from all speakers

Analytics in Higher Education: Establishing a Common Language◦ Van Barneveld, Arnold, Campbell, 2012◦ http://www.educause.edu/Resources/AnalyticsinHigherEducationEsta/245405

Analytics to Literacies: Emergent Learning Analytics to evaluate new literacies◦ Dawson, 2011- http://blogs.ubc.ca/newliteracies/files/2011/12/Dawson.pdf

Learning Analytics: Definitions, Process Potential◦ Elias, 2011◦ http://learninganalytics.net/LearningAnalyticsDefinitionsProcessesPotential.pdf

The State of Learning Analytics in 2012: A Review and Future Challenges◦ Ferguson, 2012 - http://kmi.open.ac.uk/publications/pdf/kmi-12-01.pdf

Academic analytics: A new tool for a new era. ◦ Campbell, Deblois, & Oblinger (2007). Educause Review, 42(4), 40-57. ◦ http://net.educause.edu/ir/library/pdf/ERM0742.pdf

Mining LMS data to develop an "early warning system" for educators: A proof of concept. ◦ Macfadyen & Dawson (2010) - Computers & Education, 54(2), 588-599.

Classroom walls that talk: Using online course activity data of successful students to raise self- awareness of underperforming peers. ◦ Fritz, 2011 - Internet and Higher Education, 14(2), 89-97.

Publications

Wednesday, 13 June◦ Learning Analytics: A Panel Debate on the Merits,

Methodologies, and Related Issues (1:15pm)

◦ Learning Analytics at Michigan: Designing Displays for Advisors, Instructors, and Students (2:30pm)

◦ BOF for Learning Analytics: Current and Planned Projects and Tools (3:45pm)

Thursday, 14 June◦ Creating an Open Ecosystem for Learner Analytics

(10:15am) Open Academic Analytics Initiative (OAAI) https://confluence.sakaiproject.org/x/8aWCB

Related Conference Sessions

Steve Lonn◦ slonn@umich.edu @stevelonn

Josh Baron◦ Josh.baron@marist.edu @joshbaron

Questions?

Learning Analytics 101

data available

sakai cle oae4

events visual data analytics

analytics la

predictive analytics

educational practicebienkowski

business academic analytics

supporthuman decision

Education

Analytics 101 (2016 Edition)

Website Analytics 101

Employer Brand Analytics 101

Learning Analytics: Shifting from theory to practice. ·...

Learning Analytics Architecture to Scaffold Learning...

Analytics 101 - Getting Started

Big Data in Learning Analytics - Analytics for Everyday...

Text Mining Analytics 101

Google Analytics 101 Overview

Google Analytics 101

Analytics 101

[GDC2015] Analytics 101

Analytics 101 Presentation (1)

Kursus: Google Analytics - 101

Web analytics 101

Marketing Analytics 101