John Whitmer, Ed.D. Academic Technology Services California State University, Office of the Chancellor WASC ARC Conference April 11, 2013 Improving Student Achievement with New Approaches to Data: Learning Analytics & the CSU Data Dashboard slides @ slideshare.net/JohnWhitm
58
Embed
Improving Student Achievement with New Approaches to Data
Presentation delivered at WASC ARC conference on April 11, 2013 on the CSU Data Dashboard and Chico State Learning Analytics case study.
Chico State Case Study: Academic technologies collect highly detailed student usage data. How can this data be used to understand and predict student performance, especially of at-risk students? This presentation will discuss research on a high-enrollment undergraduate course exploring the relationship between LMS activity, student background characteristics, current enrollment information, and student achievement.
CSU Data Dashboard: By monitoring on-track indicators institutional leaders can better understand not only which milestones students are failing to reach, but why they are not reaching them. It can also help campuses to design interventions or policy changes to increase student success and to gauge the impact of interventions.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
John Whitmer, Ed.D.Academic Technology Services
California State University, Office of the Chancellor
WASC ARC ConferenceApril 11, 2013
Improving Student Achievement with New Approaches to Data:
Learning Analytics & the CSU Data Dashboard
slides @ slideshare.net/JohnWhitmer/
slides @ slideshare.net/JohnWhitmer/
Outline
1. Context: California State University & Graduation Initiative
2. Chico State Learning Analytics Case Study
3. CSU Data Dashboard Project
4. Next Steps
5. Discussion
slides @ slideshare.net/JohnWhitmer/
1. CONTEXT
slides @ slideshare.net/JohnWhitmer/
California State University http://calstate.edu
23 campuses 437,000 FTE students 44,000 faculty and staff Largest, most diverse, &
one of the most affordable university systems in the country
Play a vital role in the growth & development of California's communities and economy
By 2015, the CSU will improve graduation rates by 8 percentage points systemwide and halve the achievement gap. – Baseline 6-Year Graduation Rate: 46%– Target 6-Year Graduation Rate: 54%
• Bimodal outcomes compared to traditional course • 10% increase on final exam• 7% & 11% increase in DWF
• Why? Can’t tell with aggregated data
54 F’s
slides @ slideshare.net/JohnWhitmer/
slides @ slideshare.net/JohnWhitmer/
Learner Analytics
“ ... measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs.” (Siemens, 2011)
slides @ slideshare.net/JohnWhitmer/
Pervasive Adoption of Learning Management Systems
Institution-Supported IT Resources and Tools. Reprinted from “The ECAR Study of Undergraduate Students and Information Technology,” Eden Dahlstrom, 2012 by the EDUCAUSE Center for Applied Research.
slides @ slideshare.net/JohnWhitmer/
Guiding Questions
1. How is student LMS use related to academic achievement in a single course section?
2. How does that finding compare to the relationship of achievement with traditional student characteristic variables?
3. How are these relationships different for “at-risk” students (URM & Pell-eligible)?
4. What data sources, variables and methods are most useful to answer these questions?
Excel Early data exploration; simple sorting; tables for print/publication
Tableau Complex data summaries and explorations; complex charts; presentation charts
Final/formal descriptive data; statistical analysis; some charts (scatterplots)
Statistical analysis (factor analysis)
Statistical analysis (charts)
slides @ slideshare.net/JohnWhitmer/
Correlation: Student Char. w/Final Grade
Scatterplot of HS GPA vs. Course
Grade
slides @ slideshare.net/JohnWhitmer/
Predict the trend
LMS use and final grade is _______ compared to student characteristics and final grade:
a) 50% smaller
b) 25% smaller
c) the same
d) 200% larger
e) 400% larger
slides @ slideshare.net/JohnWhitmer/
Predict the trend
LMS use and final grade is _______ compared to student characteristics and final grade:
a) 50% smaller
b) 25% smaller
c) the same
d) 200% larger
e) 400% larger
slides @ slideshare.net/JohnWhitmer/
Correlation LMS Use w/Final Grade
Scatterplot of Assessment Activity
Hits vs. Course Grade
slides @ slideshare.net/JohnWhitmer/
Chart: LMS & Student Characteristics
slides @ slideshare.net/JohnWhitmer/
Combined Variables Regression Final Grade by LMS Use & Student Characteristic Variables
LMS Use
Variables
25% (r2=0.25)
Explanation of change in final grade
Student Characteristic
Variables
+10%(r2=0.35)
Explanation of change in final grade
>
slides @ slideshare.net/JohnWhitmer/
Predict the trend
LMS use and final grade is ______ for “at-risk”* students compared to not at-risk students?
a) 50% smaller
b) 20% smaller
c) No difference
d) 20% larger
e) 100% larger
Relationship indicates how strongly LMS use is correlated with final grade; lower value equals less impact
*at-risk = BOTH under-represented minority and Pell-eligible
slides @ slideshare.net/JohnWhitmer/
Predict the trend
LMS use and final grade is ______ for “at-risk”* students compared to not at-risk students?
a) 50% smaller
b) 20% smaller
c) No difference
d) 20% larger
e) 100% larger
*at-risk = BOTH under-represented minority and Pell-eligible
slides @ slideshare.net/JohnWhitmer/
Question 3 Results:Regression by “At Risk” Population Subsamples
slides @ slideshare.net/JohnWhitmer/
At-Risk Students: “Over-Working Gap”
27
slides @ slideshare.net/JohnWhitmer/
Activities by Pell and Gradegrade / pelleligible
A B+ C C-
Pell-Eligible Not Pell-Eligible Pell-Eligible Not Pell-Eligible Pell-Eligible Not Pell-Eligible Pell-Eligible Not Pell-Eligible
0K
5K
10K
15K
20K
25K
30K
35K
Value
Content
Content
Engage
Engage
Assess
Assess
Admin
Admin
Content
Content
Engage
Engage
Assess
Assess
Admin
Content
Content
Engage
Engage
Assess
Assess
Content
Content Engage
Engage
Assess
Assess
Admin
Admin
Measure Names
Admin
Assess
Engage
Content
Extra effort in content-related activities
slides @ slideshare.net/JohnWhitmer/
Conclusions
1. LMS use is a better predictor of academic achievement than student characteristics.– LMS use frequency is a proxy for effort.
2. LMS data requires extensive filtering to be useful; student variables need pre-screening for missing data.
3. LMS effectiveness for at-risk students may be caused by non-technical barriers.
4. Small strength magnitude suggests that better methods could produce stronger results.
slides @ slideshare.net/JohnWhitmer/
Next Generation Learning Analytics
Graphic Courtesy Sasha Dietrichson, X-Ray Research SRL
slides @ slideshare.net/JohnWhitmer/
Next Steps
Potential for improved LMS analysis methods: time series analysis social learning activity patterns discourse content analysis
Group students by broader identity, with unique variables: Continuing student (Current college GPA, URM, etc.) First-time freshman (HS GPA, SAT/Act, etc)
slides @ slideshare.net/JohnWhitmer/
3. DATA DASHBOARD PROJECT
slides @ slideshare.net/JohnWhitmer/
THE FRAMEWORK
Advancing by Degrees: A Framework for Increasing College Completion by Offenstein, Moore & Schulock
Institute for Higher Education Leadership and Policy and The Education Trust (http://bit.ly/10QtMXC)
This research describes academic patterns (or leading indicators) that occur early in the pipeline that can be tracked and monitored in real time against milestones on the graduation route.
slides @ slideshare.net/JohnWhitmer/
Leading indicators statisticallyimprove predicted probabilities of completion over just the use of student background characteristics
slides @ slideshare.net/JohnWhitmer/
Milestones are measurable educational achievements that students reach along the path to degree completion.
slides @ slideshare.net/JohnWhitmer/
Milestones Leading Indicators Year-to-year Retention Transition to college level coursework
(English and Math) Earn one year of college level credits Complete General Education Complete degree
Remediation Begin remedial coursework in the first term, if
needed. Complete needed remediation
Gateway Courses Complete college-level math and/or English in
the first or second year Complete a college-success course or other
first-year experience program
Credit Accumulation and Related Academic Behaviors Complete high percentage of courses
attempted (low rate of course dropping and/or failure)
Complete 20-30 credits in the first year Earn summer credits Enroll full time Enroll continuously, without stop-outs Register on-time for courses Maintain adequate academic progress
slides @ slideshare.net/JohnWhitmer/
Driving Questions for Dashboard
1. What percentage of students reach each of the leading indicators?
2. What is the impact of reaching each of the leading indicators on success rate?
3. Does meeting any of the indicators reduce or eliminate gaps between student demographic groups?
slides @ slideshare.net/JohnWhitmer/
PROOF OF CONCEPT
slides @ slideshare.net/JohnWhitmer/
Purpose
Demonstrate potential value of combined reporting and statistics
Evaluate availability and integration of data
Pilot potential tools in real-world scenario
NOTE: production system may be dramatically different from POC, given lessons learned and scalability
slides @ slideshare.net/JohnWhitmer/
slides @ slideshare.net/JohnWhitmer/
slides @ slideshare.net/JohnWhitmer/
1. Report Parameters
slides @ slideshare.net/JohnWhitmer/
2. Retention Rates
slides @ slideshare.net/JohnWhitmer/
3. Retention Rates by URM Status
slides @ slideshare.net/JohnWhitmer/4. Data Export Options