The Learning Analytics Landscape Surveying Where we are & Where we are Going NERLA 2013 - Second North East Regional Learning Analytics Symposium Post questions/thoughts using hash tag #NERLA13 Josh Baron Senior Academic Technology Officer Marist College
38
Embed
The Learning Analytics Landscape - NERCOMP › uploadFiles › 3898000001E8.filename.Learning... · College-wide learning analytics approach • University of Michigan’s E. 2. Coach
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
The Learning Analytics Landscape Surveying Where we are &
Where we are Going
NERLA 2013 - Second North East Regional Learning Analytics Symposium
Post questions/thoughts using hash tag #NERLA13
Josh Baron Senior Academic Technology Officer
Marist College
POP QUIZ
36%
Reference: Integrated Postsecondary Education Data System (IPEDS)
13% Four Graduation Rate for
Savannah State University (2010)
Presentation Outline
High-Level Overview of Learning Analytics (LA)
Open Academic Analytics Initiative (OAAI)
Peaks on the Learning Analytics Landscape
The Big Questions about Big Data
Group Discussion/Q&A
Post questions/thoughts using hash tag #NERLA13
Analytics in Higher Ed from 50k Feet
Academic Analytics Learning Analytics A process for providing higher education institutions with the data necessary to
support operational and financial decision making*
The use of analytic techniques to help target instructional, curricular, and support resources to support the achievement of
specific learning goals*
Focused on the business of the institution Focused on the student and their learning behaviors
Management/executives are the primary audience
Learners and instructors are the primary audience
* - Analytics in Higher Education: Establishing a Common Language
Different Routes on the LA Landscape
Target Audience Results from analytics
presented to…
Instructor – Decides what, if any, action to take with the learner.
Student – Interprets results & decides on
action to take
Analytics Approach Predictions are based on…
Learning Sciences – Theories and understandings of
student behavior
“Big Data”– Analysis of large data sets of student behavior
Institution
Program
Course
The Open Academic Learning Analytics Initiative (OAAI)
Open Academic Analytics Initiative
EDUCAUSE Next Generation Learning Challenges (NGLC)
Funded by Bill and Melinda Gates Foundations
$250,000 over a 15 month period Goal: Leverage Big Data concepts to create
an open-source academic early alert system.
Student Attitude Data (SATs, current GPA, etc.)
Student Demographic Data (Age, gender, etc.)
Sakai Event Log Data
Sakai Gradebook Data
Predictive Model Scoring
Identifies students “at risk” to not complete
course
SIS
Data
LM
S Da
ta
OAAI Early Alert System Overview
Intervention Deployed “Awareness” or Online
Academic Support Environment (OASE)
Developed model using historical data
“Creating an Open Ecosystem for Learning Analytics”
ONLINE ACADEMIC SUPPORT ENVIRONMENT (OASE)
• OER Content • Self-Assessments • Learning Skills -
Flat World Knowledge
• Learning Support Facilitation & Mentoring
OAAI GOALS AND MILESTONES • Build “open ecosystem” for Learning analytics
• Sakai Collaboration and Learning Environment • Secure data capture process for extracting LMS data
• Pentaho Business Intelligence Suite • Open-source data mining, integration, analysis and reporting tools
• OAAI Predictive Model released under open license • Predictive Modeling Markup Language (PMML)
• Researching learning analytics scaling factors • How “portable” are predictive models?
• What intervention strategies are most effective?
How important is “openness” to the Learning Analytics field? Use hash tag #NERLA13
OAAI RESEARCH METHODOLOGY
• Compared Marist model to Purdue University • Share some common academic characteristics
• Conducting real world pilots • Pilots run in Spring and Fall 2012
• Introductory-level F2F courses
• ~35 courses at community colleges
• ~30 courses at HBCUs
• Conducting student surveys and focus groups
INITIAL FINDINGS - PORTABILITY
• Marist found similar predictive elements and correlations as Purdue
Model accuracy remained in the 60-80% range when Marist-derived model was deployed at community colleges and HBCUs.
INITIAL FINDINGS – INTERVENTIONS
• Found statistically significant differences between treatment and control groups • ~7% increase in course grades
• More students “mastered content” (C or better)
• Similar trend among low income students
• Students were more likely to withdraw • Similar to Purdue findings, not necessarily negative
INSTRUCTOR FEEDBACK
"Not only did this project directly assist my students by guiding students to resources to help them succeed, but as an instructor, it changed my pedagogy; I became more vigilant about reaching out to individual students and providing them with outlets to master necessary skills. P.S. I have to say that this semester, I received the highest volume of unsolicited positive feedback from students, who reported that they felt I provided them exceptional individual attention!
FUTURE RESEARCH INTERESTS
• Factors that impact on intervention effectiveness • Intervention Immunity – Students who do not respond
to first intervention tend to never respond
• Student Engagement – How can we increase the level of engagement between students and help resources?
• Can predictive models be customized for specific delivery methods and programs/subjects?
• Can Learning Analytics identify “at risk” students who would otherwise not be identified?
What do you think should be the major focus of future LA research funding? Use hash tag #NERLA13
• SNAPP - Social Networks Adapting Pedagogical Practice • Leveraging “big data” visualization
22
TOUR OF PEAKS ON THE LA LANDSCAPE
• Built predictive model using data from… • LMS – Events (login, content, discuss.) & gradebook • SIS – Aptitude (SAT/ACT, GPA) & demographic data
• Leverage model to create Early-alert system • Identify students at risk to not complete the course • Deploy intervention to increase chances of success
• Systems automates intervention process • Students get “traffic light” alert in LMS • Messages are posted to student that
suggest corrective action (practice tests)
23
PURDUE UNIVERSITY’S COURSE SIGNALS
• Impact on course grades and retention • Students in courses using Course Signals…
• scored up to 26% more A or B grades • up to 12% fewer C's; up to 17% fewer D's and F‘s
• Ellucian product that integrates w/Blackboard
24
PURDUE UNIVERSITY’S COURSE SIGNALS
• Focused specifically on introductory Physics
25
UNIVERSITY OF MICHIGAN’S E2COACH
“…to say to each what we would say if we could sit down with them for a personal chat.”
26
EXAMPLE MTS MESSAGE
• UMBC found that students earning D/F’s use Bb 39% less then higher grade achievers • Not suggesting cause and effect • Goal is to model higher achiever
behavior • Provides data directly to student
• Compare LMS use to class averages • Can also compare averages usage
data to grade outcomes • Feedback has been positive
of interaction resulting from discussion forum posts and replies
28
SOCIAL NETWORK ANALYSIS & VISUALIZATION
THE BIG QUESTIONS…
• “The obligation of knowing” – John Campbell • If we have the data and tools to improve student
success, are we obligated to use them? • Consider This > If a student has a 13% chance of passing a
course, should they be dropped? 3%?
• Who owns the data, the student? Institution? • Should students be allowed to “opt out”?
• Consider This > Is it fair to the other students if by opting out the predictive model’s power drops?
• What do we reveal to students? Instructors? • Consider This > If we tell a student in week three they have a
9% chance of passing, what will they do? • Will instructors begin to “profile” students?
30
LEARNING ANALYTICS ETHICS
What are some of the big ethical questions surrounding LA? Use hash tag #NERLA13
Big Questions about Big Data Could Learning Analytics end up driving the
wrong learning outcomes? What do you see as the biggest ethical
issues surrounding Learning Analytics? How important is “openness” in LA? Open
standards? Open licensing? If you were Bill and Melinda Gates what
would your funding priorities be around LA? Where is LA in the “hype cycle”? Where will
it end up?
LEARNING ANALYTICS AND KNOWLEDGE CONFERENCE (LAK) THE INTERNATIONAL JOURNAL OF THE SOCIETY FOR LEARNING ANALYTICS RESEARCH SOLAR FLARES – REGIONAL CONFERENCES SOLAR STORMS – DISTRIBUTED RESEARCH LAB MOOCS ON LEARNING ANALYTICS http://www.solaresearch.org/