Learner Analytics: from Buzz to Strategic Role Academic Technologists

DETCHE Conference 2011Kathy Fernandes

Scott KodaiJohn Whitmer

Learner Analytics Beyond the Buzz

Download presentation at: http://slidesha.re/sFKjcm

“But everything we know about cognition suggests that a small group of people, no matter how intellingent, simply will not be smarter than the larger group. ... Centralization is not the answer. But aggregation is.”

- J. Surowiecki, The Wisdom of Crowds, 2004

Ambitous Outline

1. Situating Analytics

2. Academic Analytics– Case Study: CSU Data Dashboard

3. Learner Analytics – Case Study: CSU Chico

4. Promising Efforts & Resources

5. Q & A

SITUATING ANALYTICS

Steve Lohr, NY Times, August 5, 2009

Economist. (2010, 11/4/2010). Augmented business: Smart systems will disrupt lots of industries, and perhaps the entire economy. The Economist.

7

Source: jisc_infonet @ Flickr.com

Source: jisc_infonet @ Flickr.com

What’s the promise of Analytics for Academic Technologists?

1. Decision-making (and service-evaluating) based on practices (not just perceptions) and performance outcomes

2. If we’re moving into a strategic role re: teaching and learning, analytics can:– demonstrate the link between technology and learning– distinguish our role from a technology service provider

(PS - anyone else concerned about the validity of student evaluations and self-reported data?) – “Rate your level of technology expertise (novice,

intermediate, expert)”

Academic Analytics

“Academic Analytics marries large data sets with statistical techniques and predictive modeling to

improve decision making”

(Campbell and Oblinger 2007, p. 3)

10

Academic Analytics1. Term adopted in 2005 ELI research report

(Goldstein & Katz, 2005)

– Response to widespread adoption ERP systems, desire to use data collected for improved decision making

– 380 respondents; 65% planned to increase capacity in near future

2. Call to move from transactional/operational reporting to what-if analysis, predictive modeling, and alerts

3. LMS identified as potential domain for future growth

CSU GRADUATION INITIATIVE DATA DASHBOARD

CSU Graduation initiative

1. System Commitment to raise freshman graduation rate 8% by 2015-2016

2. Cut achievement gap for under-represented minority students by 50%

3. Each CSU campus created own plan & activities to meet goals

More info: http://graduate.csuprojects.org/

DD Screenshot

Learner Analytics:

“ ... measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs.” (Siemens, 2011)

Learner Analytics

1. Assess relationship between learning context (aka educational technology usage) and student learning and/or achievement

2. Most research to date: LMS for fullly online courses

3. More complex than Academic Analytics, considering: – Variation in LMS usage by course– LMS learning actions are patterns, not clicks– No significant difference literature: not what technology

used, it’s how it’s used, who uses it, and for what purpose

16

Academic technologists have unique knowledge to design and conduct learner analytics (it’s our magic, a la Richard Katz!)

17

CSU CHICO VISTA ANALYTICS

18

19

20

21

22

23

24

Learner Analytics on Chico Vista Usage

1. What is the relationship between LMS usage and student achievement?

2. What is the relationship between the number of LMS tools used (aka ‘breadth of faculty LMS adoption’) and student achievement?

3. Perform analysis within courses

4. Ultimate goal: provide administrators and faculty with what-if modeling tools, building on reports in data warehouse

CSU Practice

Call to Action

1. Metrics reporting is the foundation for Analytics

2. Don’t need to wait for student performance data; good metrics can inspire access to performance data

3. You’re *not* behind the curve, this is a rapidly emerging area that we can (should) lead ...

Promising Efforts & Directions

1. WCET “Predictive Analytics Framework” (http://bit.ly/tMYFNF)– Participants: American Public University System, Colorado CCS, University

of Hawaii System, University of Illinois at Springfield, Rio Salado College, University of Phoenix

2. Building Organizational Capacity for Analytics Survey (http://bit.ly/vPxKnw)

3. Educause Analytics “Capacity Building” initiative (http://bit.ly/rLux6x)

Note: each of these efforts is supported by Linda Baer, Gates Foundation

Resources to move forward with Analytics at your campus

Learner Analytics bibliography: http://bit.ly/rC0l5T Visualizing Data: Essential Collection of Resources:

http://bit.ly/sNriMe Moodle Custom SQL queries report:

http://bit.ly/toPWWD Bb Stats: http://bit.ly/w0L6th Bb Project Astro: http://bit.ly/w0L6th

30

Q&A and Contact Info

• Kathy Fernandes (kfernandes@csuchico.edu)

• Scott Kodai (skodai@csuchico.edu)

• John Whitmer (jwhitmer@csuchico.edu)

Download presentation at: http://slidesha.re/sFKjcm

31

Works CitedArnold, K. E. (2010). Signals: Applying Academic Analytics. Educause Quarterly, 33(1).California State University Office of the Chancellor. (2010). CSU Graduation Initiative

Retrieved 10/18, 2010, from http://graduate.csuprojects.org/Campbell, J. P. (2007). Utilizing student data within the course management system to

determine undergraduate student academic success: An exploratory study. Unpublished Ph.D., Educational Studies, United States -- Indiana.

Campbell, J. P., DeBlois, P. B., & Oblinger, D. G. (2007). Academic Analytics: A New Tool for a New Era. EDUCAUSE Review, 42(4), 17.

Goldstein, P. J., & Katz, R. N. (2005). Academic analytics: The uses of management information and technology in higher education. . Washington, DC.

Macfadyen, L. P., & Dawson, S. (2010). Mining LMS data to develop an "early warning system" for educators: A Proof of Concept. Computers & Education(54), 11.

Offenstein, J., Moore, C., & Shulock, N. (2011). Advancing by Degrees: A Framework for Increasing College Completion.

Siemens, G. (2011, 8/5). Learning and Academic Analytics. http://www.learninganalytics.net/

Surowiecki, J. (2004). The Wisdom of Crowds. New York: Anchor Books.

32

BONUS SLIDES!

33

Academic Analytics Levels & Frequency

Analytics Level Respondents

Level 1: Extraction and reporting of transaction-level data 263Level 2: Analysis and monitoring of operational performance 51

Level 3: What-if decision support 6Level 4: Predictive Modeling/Simulation 7Level 5: Automated triggers/alerts 17

N/A 32

263

51

67 17

32Level 1: Extraction and reporting of transac-tion-level dataLevel 2: Analysis and monitoring of opera-tional performanceLevel 3: What-if de-cision supportLevel 4: Predictive Modeling/SimulationLevel 5: Automated triggers/alertsN/A

Table and Chart adapted from Goldstein & Katz, 2005

34

Research Findings

1. There is not a relationship between sophistication of technology and sophistication of application/deployment– Largest raw number of advanced users had simple

transactional reporting tools

2. Factors leading to higher levels application:– Leadership commitment to evidence-based decision

making– Staff skills– Effective end user training

35

CSU GRADUATION INITIATIVE DATA DASHBOARD

36

Advancing by Degrees: A Framework for Increasing College Completion

-Institute for Higher Education Leadership and

Policy and The Education Trust

Data Dashboard Theoretical Framework & Guiding Questions

1. What percentage of students reach each of the leading indicators?

2. What is the impact of reaching each of the leading indicators on success rate?

3. Does meeting any of the indicators reduce or eliminate gaps between student groups?

DD Screenshot

DD Screenshot

39

EXAMPLES OF LEARNER ANALYTICS RESEARCH

40

JP Campbell Dissertation Study (2007)

Utilizing student data within the course management system to determine undergraduate student academic success: An exploratory study

1. LMS usage for entire university for 1 semester (70,000 records, 27,000 students)

2. 15 demographic variables, 20 Vista variables 3. Outcome variable: student grade4. Multivariate regression to create predictive model

for significant variables

41

How much do Vista usage variables increase predictive accuracy compared to predictions based on student characteristics only?

a) 0.3%b) 5%c) 12%d) 25%e) 54%

42

How much do Vista usage variables increase predictive accuracy compared to predictions based on student characteristics only?

a) 0.3%b) 5%c) 12%d) 25%e) 54%

Prediction rate: 62.4%

43

Why such a small increase?

1. Variation in usage creates “missing data” for tools not used in other courses

2. Lesson Learned: perform analysis relative to students within the same course

3. Next Generation implementation: Purdue Biology course using “Signals” early warning system with students (Arnold, 2010)– D/F grades reduced 14%– B/C grades increased 12%

44

Macfadyen and Dawson (2010)In a fully online biology course at the University of British Columbia (n=118, 5

sections, 3 semesters), found that:

1. 33% of student grade variability could be explained by 3 variables (discussion messages posted, mail messages sent, and assessments completed)

2. 13 variables (out of 22 studied) had significant correlations with final student grade (R2 values from .05 to .27)– Significant variables included number online sessions, total time only, and activities

within content, mail, assessment, and discussion areas– Variables not significant included some predictable items, such as visits to

MyGrades, uses of search, ‘who is online’, and the ‘compile’ tool. They also included surprising items, such as the number of assignments read, the time spent on assignments, and announcement views

3. 73.7% of the students correctly classified as at-risk (i.e. final grade of D or F) through predictions based on these three variables