Using Analytics to Inform “Tactical” Design Decisions Hae Okimoto, University of Hawaii Casey Sacks, Colorado Community College System Karen Swan, University of Illinois Springfield Karen Vignare, University of Maryland University College July 8, 2014
Using Analytics to Inform "Tactical" Design Decisions
Dec 18, 2014
Presented at Blended Learning Conference and Workshop on July 8, 2014. Presenters included:
Hae Okimoto (University of Hawaii System, USA)
Casey Sacks (Colorado Community College System, USA)
Karen Swan (University of Illinois Springfield, USA)
Karen Vignare (UMUC, USA)
Presentation abstract:
Engage in a discussion with representatives of community college systems in Colorado and Hawaii, University of Maryland University College, and the University of Illinois Springfield about how they are using the Predictive Analytics Reporting (PAR) Framework to inform their tactical decision making around course and program design, decision making which has led them to blended solutions.
The Predictive Analytics Reporting (PAR) Framework is a non-profit multi-institutional data mining collaborative that brings together 2 year, 4 year, public, proprietary, traditional, and progressive institutions to collaborate on identifying points of student loss and to find effective practices that improve student retention in U.S. higher education. With sixteen WCET member institutions, over 1,700,000 anonymized student records and 8,100,000 institutionally de-identified course level records, the PAR Framework offers educational stakeholders a unique multi-institutional lens for examining dimensions of student success from both unified and contextual perspectives.
The PAR Framework collaborators are also working on developing tools to support decision making at the local institutional level. These include: tools for identifying local predictors of retention and progression, tools for exploring institutional data, tools for benchmarking local retention and progression outcomes against other similar institutions along a variety of dimensions, and a tool for identifying and classifying interventions aimed at improving student success along two dimensions - predictors of retention and progression and time in students' academic life cycle.
What does all this data mean and how can it help with decisions around redesign? This session is for those interested in "tactical" redesign focused on blended solutions. We will discuss how data and analytics can be used in both a broad sweep at the program and course structural levels, as well as in the more detailed exploration of student behaviors and their impact within courses and programs. Audience participation will be encouraged.
Hae Okimoto (University of Hawaii System, USA)
Casey Sacks (Colorado Community College System, USA)
Karen Swan (University of Illinois Springfield, USA)
Karen Vignare (UMUC, USA)
Presentation abstract:
Engage in a discussion with representatives of community college systems in Colorado and Hawaii, University of Maryland University College, and the University of Illinois Springfield about how they are using the Predictive Analytics Reporting (PAR) Framework to inform their tactical decision making around course and program design, decision making which has led them to blended solutions.
The Predictive Analytics Reporting (PAR) Framework is a non-profit multi-institutional data mining collaborative that brings together 2 year, 4 year, public, proprietary, traditional, and progressive institutions to collaborate on identifying points of student loss and to find effective practices that improve student retention in U.S. higher education. With sixteen WCET member institutions, over 1,700,000 anonymized student records and 8,100,000 institutionally de-identified course level records, the PAR Framework offers educational stakeholders a unique multi-institutional lens for examining dimensions of student success from both unified and contextual perspectives.
The PAR Framework collaborators are also working on developing tools to support decision making at the local institutional level. These include: tools for identifying local predictors of retention and progression, tools for exploring institutional data, tools for benchmarking local retention and progression outcomes against other similar institutions along a variety of dimensions, and a tool for identifying and classifying interventions aimed at improving student success along two dimensions - predictors of retention and progression and time in students' academic life cycle.
What does all this data mean and how can it help with decisions around redesign? This session is for those interested in "tactical" redesign focused on blended solutions. We will discuss how data and analytics can be used in both a broad sweep at the program and course structural levels, as well as in the more detailed exploration of student behaviors and their impact within courses and programs. Audience participation will be encouraged.
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Using Analytics to Inform “Tactical” Design Decisions
Hae Okimoto, University of HawaiiCasey Sacks, Colorado Community College System
Karen Swan, University of Illinois SpringfieldKaren Vignare, University of Maryland University College
July 8, 2014
Predictive Analytics Reporting (PAR)
• National• Non-Profit• Multi-Institutional• Collaborative
Student Success & Institutional Effectiveness
About the PAR Framework
• established, growing non-profit collaborative focused on using existing institutional data to improve institutional effectiveness & student outcomes
• funded by Bill & Melinda Gates Foundation 2011-2014• managed at the Western Interstate Commission for
Higher Education (WICHE)• engagement with > 39 forward thinking US institutions• small, high functioning team with partner, subject and
domain expertise
• a “big data” analysis effort to identify drivers related to retention & progression, and to inform student loss prevention
• PAR member institutions voluntarily contribute de-identified student records to create a single federated database
• identifying scalable approaches to student success through common data definitions, common outcome measures, common definitions of interventions
About the PAR Framework
reusable predictive
models
common definitions of terms
student level watch lists for targeted
interventions
multi-Institutional collaboration
measurable Intervention results
PREDICT
ACT
RESULTS
common definitions of interventions
scalable cross-institutional improvements
About the PAR Framework
Institutional Partners Founding Partners (since 2011): American Public University SystemColorado Community College SystemRio Salado CollegeUniversity of Hawaii SystemUniversity of Illinois SpringfieldUniversity of Phoenix
New Members (as of Oct 2013): Northern Arizona University Kaplan University Excelsior College University of North Dakota
Implementation Partners (since 2012):Ashford UniversityBroward CollegeCapella UniversityLone Star College SystemPenn State World CampusSinclair Community CollegeTroy UniversityUniversity of Central FloridaUniversity of Maryland University CollegeWestern Governors University
predictives help focus on the highest need; but they don’t solve the problem
• Benchmarks understand how institutions’ student outcomes compare to their peers by scaling a multi-institutional database for benchmarking and research purposes.
• Models identify which students need assistance, by using in-depth, institutional specific predictive models which are unique to the needs and priorities of member institutions based on their specific data.
determine the best way to address areas of weakness identified in benchmarks and models by scaling and leveraging a member and literature validated framework for examining interventions within and across institutions (SSMx)
• Interventions
CCCS redesign of Reading, English & Math
• Loss points for students in the traditional sequence
• Charge committee to redesign developmental education
• Workgroup, policy, grant support, implementation
CCCS College Composition and Reading (CCR)
• From up to 6 courses to preparation after a single semester
• Integrated reading and English• Most students in the ALP type model• Extra support and structure for students with
lowest placement profiles
CCCS Math
• From 4 courses to selecting a pathway for preparation– STEM (transfer path), non-STEM (transfer path),
non-transfer math• Structure for corequisite instruction of college
and developmental math
University of Hawaii – Community College Developmental Math
• 75% of students from public school start at a community college
• 85% of entering community college students not prepared for college level math
• 61% enroll in remedial/developmental math• 56% successfully complete a course• 3-4 courses in developmental course sequence
The Issue - LCC
Students earning C or better in developmental math
Acceleration
Emporium Redesign
• Learning & practice time are individualized & optimized
• Software-based assistance is always available
• In-person assistance is available during scheduled class times & open lab
• High mastery standards are balanced with repeatable quizzes & tests
Success Rates
Redesign
Legacy
Theoretical Support
• Institutional Departure– Academic difficulties– Inability to integrate educational & occupational
goals– Failure to engage in social & intellectual life
• Tinto & Bean’s Model of Student Integration– Social– Academic
Using Data to Make Tactical Decision
University of Maryland University College• Investment in Office of Analytics (& IR)• Outside partnerships (PAR, Civitas Learning) &
grants (Kresge transfer data repository)• Executive level steering committee & Cross-
Functional Interventions team• Commencing a Rapid Action Team (yes, RAT!)
on student success
The Synergy of Student Analytics Projects
KresgeTransfer PAR
Civitas Learning
18
Student Success
Interventions Team
Advising
Academic Departments
Provost Office
Analytics
Inst. Research
Student Services
19
Benchmarking with PAR
• UMUC Denise Nadasen Co-chairing (with UIS) • Gain a better understanding of how to
integrate multi-institutional data• Learned how to manage a large-scale data
project• Developed data-sharing partnerships with
nationally known institutions
Interventions Accounting 220 & Accounting 221
Test ControlSuccessful Course Completion 72%* 58%*Term GPA 2.52* 2.10*Reenrollment 78% 72%Change in GPA 0.31 0.07
Key Findings
• Students participating in tutoring had a significantly higher rate of successful course completion.
• Students participating in tutoring had a significantly higher term GPA.
• The test group reenrollment rate was six percentage points higher but not statistically significant.
Interventions
Messaging• Designed by Cross-functional intervention
team• Using Civitas risk score for students in gateway
courses• Results are mixed—Fall term saw test
reenrollment rate increase but all other terms mixed
Jumpstart—New Onboarding
• Four Week, one credit—diagnostics, mission statement, career planning, academic plan
Key Findings (Fall & Spring, awaiting summer)• There is a significant difference between JumpStart
Completers and the Control Group for Semester GPA, Successful Course Completion, and Reenrollment.
• There is not a significant difference between All JumpStart Students and the Control Group for Semester GPA and Successful Course Completion.
• There is a significant difference between All JumpStart Students and the Control Group for Reenrollment.
Student Success Matrix (SSMX)
• an organizational structure that helps institutions inventory, organize & conceptualize interventions aimed at improving student outcomes
What is the Student Success Matrix?
• categorizes interventions along two major dimensions – known predictors of retention/progression & timing of intervention in context of its delivery
• other aspects addressed include functionality, return on investment (ROI), focus, and institutional level
©PAR Framework 2014
learner characteristics
learner behaviors
fit/feelings of belonging
other learner support
course/program characteristicsinstructor
behaviors
time connection entry progress completionpredictors
Why Use the Student Success Matrix?Provides a common structure for categorizing and quantifying student supports across an institution
• alignment of interventions to predictors• review of timing of interventions• Identifying gaps & redundancies in student support• prioritizing interventions• illuminating opportunities for impact measurement
Comparable categorization across institutions allows for intervention benchmarking and sharing what works with specific categories of ‘at risk students.
©PAR Framework 2014
Automated SSMX Tool
Automated SSMX Tool
Automated SSMX Tool
Related Documents
