Enhancing ODL Students' Success through Risk Profiling and Prediction: The Case of Unisa

Enhancing ODL Students' Success through Risk Profiling and Prediction: The Case of

Unisa HANLIE LIEBENBERGSenior Specialist: Institutional Research, Unisa

PROF GEORGE SUBOTZKYExecutive Director: Information & Strategic Analysis,

Unisa

DION VAN ZYLManager: Information Services, Unisa

Presented at:NADEOSA Conference, Johannesburg, 30 August 2011

Acknowledgements • The efforts of numerous DISA staff members in

gathering and preparing information is acknowledged

• In particular, the help and support of Robert Lightbody, admin Asst/caregiver to Prof Subotzky, was invaluable in preparing this presentation

Overview

1. Background, Key Challenges & Research Problem

2. Unisa Student Success Framework3. Unisa Conceptual & Predictive

Models of Success4. Unisa Tracking System5. Data Analysis Challenges6. Segmented Profiling: Categorising

Student Risk

Background• Whilst various theoretical models

contribute towards understanding the various dimensions impacting on student success, utilising actionable intelligence to inform effective interventions remains daunting

• This challenge is particularly formidable at Unisa, which now has +340 000 mainly non-traditional, older, part-time, underprepared students

• They face challenging socio-economic circumstances, particular work-related and domestic responsibilities, which impede on student success

Research Problem

• To address this, Unisa recently developed a student support & success framework, comprising 4 elements:

- Conceptual model- Predictive model- Student support interventions- Evaluating impact

• Critical challenge: moving from conceptual model of student success to profiling, tracking, assessing and predicting risks to success

Key Challenges

• Key concerns and critical questions that arose in developing an integrated Student Support and Success Framework in the Unisa context

• More particularly, the process of moving from the conceptual modelling of student success – a necessary first step – to the detailed student profiling, tracking and predictive modelling of risks upon which effective interventions are based

• Key challenge: translating and operationalising relevant constructs of the high-level conceptual model to create a comprehensive student profile, tracking system and predictive model which retains sufficient complexity but remains practicable 

The Challenge of Translating Theory into Practice

A theory that could fully explain every aspect of the attrition process would contain so many constructs that it would become unwieldy if not unmanageable. Such situations call for the use of theoretical models which are simplified versions of reality that strip away the minute details to concentrate on factors that are assumed or deduced to be important. ... Models can be judged by their usefulness. A model of the attrition process should contain sufficient constructs to explain what is undoubtedly a complex process and yet sufficiently simple to be understandable and useable. It should be able to explain collected descriptive data, and it should provide a framework against which predictions can be hazarded and judgements made about potential interventions.

Kember (1989: 279-280)

Operationalising the Conceptual ModelThis implies:

• Identifying and defining all academic and non-academic variables needed for construct measurement, segmentation, profiling and predictive modelling;

• Utilising suitable data gathering methods that yield consistent, complete and unbiased data; and

• Applying appropriate advanced statistical analysis that can identify complex underlying multivariate dynamic relationships between variables and constructs

Elements of the Unisa Student Success Framework

Evaluating impact over

time

Incrementally implementing an institution-wide Student

Support Framework

Comprehensive profiling,

tracking and intelligence gathering

culminating in predictive model of student

risks/success

Extensive literature review &

conceptual modeling of all

factors affecting

success in Unisa context

Processes:• Informed responsibility & ‘choice’• Ontological/epistemological dev.• Managing risks/opportunities/

uncertainty: Integration, adaptation, socialisation & negotiation

Domains:• Intra-personal• Inter-personal

Modalities:• Attribution• Locus of

control• Self-

efficacy

Processes:• Informed responsibility &

choice• Managing

risks/opportunities: Transformation, change management, org. learning, integration & adaptation

Modalities:•

Attribution

• Locus of control

• Self-efficacy

Domains:• Academic• Operational• Social

TRANSFORMED INSTITUTIONAL IDENTITY & ATTRIBUTES:

STUDENTIDENTITY & ATTRIBUTES:

• Situated agent: SES, demographics• Capital: cultural, intellectual, emotional,

attitudinal• Habitus: perceptions, dispositions,

discourse, expectations

Success

INSTITUTIONALIDENTITY & ATTRIBUTES:

• Situated organisation: history, location, strategic identity, culture, demographics• Capital: cultural, intellectual, attitudinal• Habitus: perceptions, dispositions,

discourse, expectations

SHAPING CONDITIONS: (predictable as well as uncertain)• Social structure, macro & meso shifts: globalisation, political economy, policy; National/local culture & climate

• Personal /biographical micro shifts

SHAPING CONDITIONS: (predictable as well as he uncertain)• Social structure, macro & meso shifts: globalisation, internationalisation, political economy, technology, social demand

• HE/ODL trends, policy• Institutional biography & shifts; Strategy, business model & architecture, culture & climate, politics & power relations

Choice, Admission

Learning activities

Coursesuccess

Gradua-tion

THE STUDENT WALK: Multiple, mutually constitutive

interactions between student, institution & networks

• Managing complexity/ uncertainty/ unpredictability/risks/opportunities• Institutional requirements known &

mastered by student• Student known by institution through

tracking, profiling & prediction

FIT

FIT

FIT

FIT

Employ-ment/

citizenship

TRANSFORMED STUDENT IDENTITY & ATTRIBUTES:

FIT

FIT

FIT

FIT

FIT

FIT

FIT

FIT

Retention/Progression/Positive experience

Key Constructs of the Predictive Modela) Students' inter-personal attributes:

Demographics and past socio-economic status, including educational and family background and exposure to role models;

Current socio-economic status and life circumstances, measured by the constructs of time and opportunity to study and stability in life circumstances and support for study;

b) Students' intra-personal attributes: Academic readiness and ability; Metacognitive skills; Psychological attributes and outcomes of other processes;

c) Institutional services, practices & culture: The quality of academic and administrative services; Institutional culture and practices;

d) Integration, engagement and transformation: Students' effective management of their life circumstances and mitigation of

risks as well as meeting learning expectations and utilising opportunities; The institution's effective management of academic and support processes

and mitigation of risks.

Student as Situated Agent

Student Walk

Institution as Situated Agent

Background:• Demographics• Past SES- Educ.

Background- Family

Background- Role Models

• Satisfaction• Graduatene

ss

Quality of Academic Services

Quality of Admin

Services

Social: Institutional Culture & Practices

Fit: Academic Choices & Activities

Academic Readiness & Ability

Meta-Cognitive

Skills

Psycho-logical

Attributes &

Outcomes

Formative Assessmen

tCourse Success

Intra-Personal

Utilisation of Admin/ Support Services

Fit with Institutional Culture & Practices

Graduation

Success

Inter-Personal

Student’s Effective Management of:• Life Circumstances & Risks• Learning Expectations &

Opportunities

Integration, Engagement & Transformation

Institutional Services, Practices & Culture

Current SES & Life Circumstances:

- Time & Opportunity

- Stability & Support

Institution’s Effective Management of:

• Academic & Support Processes/Risks

• Student Profile/Risk & Communication

TRACKING SYSTEMProfiling, Tracking & Predicting Risk at the level of Student/Module/Qualification/Institution

Senate

School/College TLSC

STLSC

Student Success Forum

DCCADSMPPDLibrary

DSAADSAR

Admin StructuresTSDL

Academic Department

Student Support Coordinator

Lecturer/Supervisor/Online Mentor/Tutors/Regions

Professional Structures

Academic Admin

DISALibrary

Student Information•Applications/Registration•HEMIS•Assessment Performance/Scores•Academic Readiness Self-Assessment•Student Profile Survey•Student Satisfaction Survey•Exit/Tracer Surveys• ICMAs

Operational Processes•Application/Registration•Study Material•Assessment Management•Finance•HR

Communication/Engagement•College/School/Department/

Regions•E-Tutor/F2F Tutor/Online Mentor•Counsellor•Call Centre•Admin Department•Tutorial Attendance•myUnisa/Library•Student Course Evaluation

USGS Dean of Stud.Affective

Student Profile Design Challenges• Considerations were given

specifically to question response formats and scaling

• Initial draft survey questionnaire consisting of over 100 questions based on key constructs identified in conceptual and predictive models

• Throughout design process, imperative to ensure alignment between questionnaire items, measurements and constructs

• Final version comprising approximately 50 questions derived

• Methodological and practical issues had considered in operationalising the instrument

Data Analysis Challenges• All questions were designed within

demands of data• Develop single continuous scale

measure for each construct that is uni-dimensional, can discriminate across full spectrum of students and is valid/reliable

• Four steps in the construction of scale measures, namely:

1. Item selection2. Examination of the empirical

relationships of items3. Combining of items into a

scale measure; and4. Validating the scale measure

Risk Categories: Key Element of Segmented Student

Profiling• This involved distilling 3 primary student-related cluster

constructs from the predictive model, namely:– Academic ability– Psychological attributes/metacognitive skills and– Life circumstances– Effective engagement with the institution (construct left

out of the initial risk categorisation, as this involves complex measurement through, for example, student engagement surveys)

• A good example of deriving simplified, but meaningful measurable constructs out of the complexity of the full predictive model

• The challenge was to define risk categories which could be measured on appropriate scales. Three approaches were explored

Hypotheses• If sufficient engagement, integration & transformation

is achieved, this will generate:

– Greater utilisation of support services– Sufficient fit between students' choices, behaviours,

transforming attributes & performance and institutional communications, practices, expectations and culture

• In turn, this will generate greater success in:

– Formative assessment, course success, graduation, student satisfaction and required graduate attributes

2 Categorical Risk Measures: High/Low (8 Permutations)

Academic Ability Skills/Attributes Circumstances Risks Risk Categories

Able Developed Conducive 3L Very low risk

Able Underdeveloped Conducive 2L 1H Low risk

Able Developed Obstructive 2L 1H  

Challenged Developed Conducive 2L 1H  

Able Underdeveloped Obstructive 1L 2H High risk

Challenged Underdeveloped Conducive 1L 2H  

Challenged Developed Obstructive 1L 2H  

Challenged Underdeveloped Obstructive 3H Very high risk

Risk Model3 Categorical Risk Measures: High/Medium/Low (27 Permutations)

Academic Ability

Skills/Attributes

Circumstances Risks Risk

CategoriesH H H 3H Very LowH H M 2H 1M  H M H 2H 1M  M H H 2H 1M  H M M 1H 2M LowM H M 1H 2M  M M H 1H 2M  H H L 2H 1L  L H H 2H 1L  H L H 2H 1L  

H M L 1H 1M 1L Moderate

L H M 1H 1M 1L  

M L H 1H 1M 1L  

L M H 1H 1M 1L  

M H L 1H 1M 1L  

H L M 1H 1M 1L  

M M M 3M  H L L 1H 2L HighL H L 1H 2L  L L H 1H 2L  L M M 2M 1L  M L M 2M 1L  M M L 2M 1L  M L L 1M 2L Very High

5-point Risk Measure (125 Permutations)

Permutations Risk Score

Risk Categories

20 3-6 Very Low

33 7-8 Low

37 9-10 Moderate

25 11-12 High

10 13-15 Very High

• Use of different multivariate techniques

• “While identifying relevant

variables explaining and protecting

success is the point of departure,

the real challenge, in light of the

complexities involved, is

determining the combined effects

of and relationships between

different predictor variables.”

(Subotzky & Prinsloo, Distance

Education 32/2, 2011)

Reflection on process so far

Step 1Project Design

Step 2Data Collection

Quantitative

Research Process

Step 3Analyses,

Interpretation and

Reporting

Challenge 1:Identifying relevant

measures

Challenge 2:Methodological and operational

considerations

Challenge 3:Data analyses

• Translation of conceptual ideas into meaningful questions/variables for profiling, tracking & risk/success prediction

• Defining of measurable constructs & risk/success categories

• Scaling considerations• Definition of risk categories

• Tracking system• Survey design (data gathering

method; timing & frequency; incentives

Questions