ET4 Online Symposium, 2013 Using Text Analytics to Enhance Data-Driven Decision Making Liz Wallace Director, Institutional Research Melissa Layne, Ed.D.

ET4 Online Symposium, 2013

Using Text Analytics to Enhance Data-Driven Decision Making

Liz WallaceDirector, Institutional Research

Melissa Layne, Ed.D.Director, Research and Methodology

Phil Ice, Ed.D.VP, Research & Development

ASSESSMENT AT APUSA D3M Culture

ConnectionsAll analyses and stakeholders

are interrelated

Levels of AnalysisA range of approaches are required to

satisfy stakeholder needs

Descriptive Data

Inferential

Retention Learning Effectiveness Instructional DesignBeta Level Technology Integration

Regression Factor AnalysisDecision Trees

Predictive ModelingFederation of multiple demographic

and transactional data sets

Retention and Causality

Understanding who is likely to dis-enroll is different than why

Quantitative measures are not adequate for implementing systematic change across the enterprise

Explanatory data is needed

End of Course Survey Data

APUS participates in the Community ofInquiry (COI) End of Course survey.

The COI is a validated instrument based onthe research around Social, Cognitive, andTeaching Presence.

More than 500,000 learners have used this instrument and created a strong baseline forfurther research.

Garrison, D. R., Anderson, T., & Archer, W. (2000). Critical inquiry in a text-based environment: Computer conferencing in higher education. The Internet and Higher Education, 2(2-3), 87-105

COMMUNITY OF INQUIRY FRAMEWORK

• A process model of learning in online and blended educational environments

• Grounded in a collaborative constructivist view of higher education

• Assumes effective online learning requires the development of a community of learners that supports meaningful inquiry and deep learning

SOCIAL PRESENCE – ELEMENTS

• Affective expression (expressing emotion, self-projection)

• Open communication (learning climate, risk free expression)

• Group cohesion (group identity, collaboration)

COGNITIVE PRESENCE – ELEMENTS

• Triggering event (sense of puzzlement)

• Exploration (sharing information & ideas)

• Integration (connecting ideas)

• Resolution (synthesizing & applying new ideas)

TEACHING PRESENCE – ELEMENTS

• Design and organization (setting curriculum & activities)

• Facilitation (shaping constructive discourse)

• Direct instruction (focusing & resolving issues)

COMMUNITY OF INQUIRY SURVEY

• 9 social presence items (3 affective expression, 3 open communication, 3 group cohesion)

• 12 cognitive presence items (3 triggering, 3 exploration, 3 integration, 3 resolution)

• 13 teaching presence items (4 design & facilitation, 6 facilitation of discourse, 3 direct instruction)

COMMUNITY OF INQUIRY SURVEY - VALIDATION

• Tested in graduate courses at four institutions in the US and Canada

• Principal component factor analysis

• Three factor model predicted by CoI framework confirmed

• Arbaugh, Cleveland-Innes, Diaz, Garrison, Ice, Richardson, Shea & Swan – 2008

• Subsequent validation and cumulative n over 1 million – used at at least 50 institutions

Faculty EvaluationCombining descriptive, regression

and factor analysis

Anomalies

• Factor patterns can be an indication of problems

• Two factor solution found to produce lower level learning outcomes

• Four factor pattern associated with higher levels of disenrollment

• Factor analytics indicate that there is a problem, not what it is

Explanatory Data

• Understanding trends on end of course survey is useful but not adequate for implementing change

• Explanatory data must be utilized for clarification

• Utilization of qualitative survey data

• Large volume of data makes traditional qualitative methods impractical

• Exploration needs to go beyond branch / node analysis

• Text analytics utilized at APUS

Text Analysis• Utilized IBM/SPSS Text Analytics for Surveys

• Large volume of data - started with a 2-month sample to “train” the model

• Libraries existed in Text Analytics, but they were not specific to CoI or Higher Education

• Existing Opinions library helped identify Positive and Negative comments

• Needed to define CoI categories and identify the terms that were related to each category

• Decided to focus on Teaching Presence and Cognitive Presence

Teaching Presence

• Teaching presence • Design & Organization• Facilitation of Discourse• Direct instruction

• Used Community of Inquiry questions to begin to define the categories.

• Coded each category and a Positive (+) and a Negative (-) grouping for each

• Example: Direct Instruction, Direct Instruction-Positive, Direct Instruction-Negative

Teaching Presence

Cognitive Presence

• Cognitive presence items • Triggering Event• Exploration• Integration• Resolution

• Used Community of Inquiry questions to begin to define the categories.

• Coded each category and a Positive and a Negative grouping for each

• Example: Exploration, Exploration-Positive, Exploration-Negative

Cognitive Presence

Testing

• Tested/trained the model by having someone who knows the CoI well (Phil Ice & Melissa Layne) code samples and compared to the results from the model

• Repeated with new samples to improve accuracy

• Final test with new data sample (not used for training) to most-effectively measure accuracy

• Testing based upon 2011 data was approximately 80% accurate

Considerations

• Categories under Teaching Presence were more commonly and accurately assigned

• Fewer responses related to Triggering Event, Resolution and Integration than to the category of Exploration

Considerations

• Model will never be 100% accurate because of phrasing or sarcasm

• “I don’t think anything was exceptional during this class.” -- may be flagged as positive

• “That professor was so dynamic – as dynamic as an old shoe.” -- may be flagged as positive

• “Not exceptional” or “not dynamic” would be coded correctly

Application of Text Analysis

• Included courses identified as 3 or 4 factor courses

• Included only survey responses with comments

• Applied to surveys with at least one of the categories (such as Exploration) with a mean average score that fell below 3 (on a scale of 1-5)

• This data was loaded into the IBM/SPSS Text Analytics for Surveys model

• Results were exported to Excel

Exploration

• Random sampling of data with thematic extraction

• Validation through second round random pulls

• Consistency with conceptual framework of the CoI

• Does it pass the sniff test?

Examples

• Lack of clarity in instructional design and organization – bad directions

• No value given to discussion based activities

• Disconnect with group

• Triggering event poorly structured

• Lack of perceived applicability

ET4 Online Symposium, 2013

Thank You!Liz Wallace

Director, Institutional [email protected] Melissa Layne, Ed.D.

Director, Research and [email protected]

Phil Ice, Ed.D.VP, Research & Development

[email protected]