Evaluating Online Learning: Issues and Strategies Mark Hawkes Dakota State University Innovations, Educating New Generations March 1, 2002
Dec 31, 2015
Evaluating Online Learning: Issues and Strategies
Mark HawkesDakota State UniversityInnovations, Educating New GenerationsMarch 1, 2002
Presentation Objective
Discuss online learning evaluation approaches in graduate programs at two universities Identify criteria/indicators suitable for the evaluation of online learning environments
Distance Learning Literature
Evaluation . . . . . . . No Yes(79%)
(21%)
Focus . . . . . . . . . Training Education
Impact on Learning . . . . . . . No Yes
(87%)(13%)
(58%) (42%)
Instructor Student Student Student Student Student
Internet
Login Interface
Content
Resources
AssessmentCommunication
Modes
Support Services System Resources
Managementand Organizational
Information
OnlineLearning Architecture
Institutional
Familiar Online Learning Evaluation Targets . . .
interface designinstructional designstudent satisfactiontechnology accessfaculty satisfactioneconomic viabilitydepartmental capacityinterdepartmental collaboration
Common Dimensions of Effective Online Learning:
Relevant and challenging assignments Providing adequate and timely feedback through teacher-student interactionFlexibility in teaching and learning Constructing coordinated learning environments Constructing rich environments for student to student interaction
DSU’s Educational Technology Program
Students:36 Credit hour MS program80% Education (K12, Technical, Higher Ed.)20% Business/industry90% Online; 10% On campusFemale 68%; Male 32%Project-based curriculum
DSU’s ET EnvironmentPervasive technological culture Consistency between program goals and the state/region-wide initiatives Campus-wide faculty support Institutional experience in Web-based instruction delivery Multi-delivery methodsClient: teachers, teacher developers, instructors, trainers, technology coordinators, etc.Predominantly web-based delivery
An Evaluation Model . . .
IlluminativeOperation of Components And Subcomponents
IntegrativeHolistic perspective onThe learning experience
Course &Program Design
Components
Infra-structure/System
Work Flow
Interaction Impact
Process Product
Observing and Detecting Focused on Performance Functional Problems Outcomes
Infrastructure/System
Input/output devices Network speed and connectivity Network design/Topology Technical support systems and maintenance
Course and Program Design
Nature of the Design Situation Based Role of State and National Standards Sequencing/Instructional Strategies Assessment Motivation: Learning vs. Performance
Visualization Tools and MediaUser InterfaceCourse Management
Work Flow
Use of discussion toolsSoftware usageMessage redundancy (audio, video, web pages, emails).Progression Do learners progress through their
work tasks in a linear fashion? (novice-like)
Nonlinear opportunistic fashion (expert-like)
InteractionSocial and instructionalMust account for all of the following relationships:
Instructor Learner
Learners
Content
Learners
Technology
Content
Technology
Learner
Instructor
Content Assessment
Learning Resources(Technology)
LearnerRecords
Catalog info
Query Preferences
Performance
Interaction Example
(Metcalf, Snitzer, Austin, 2001)
Learner
Online Course InteractionAnnouncementsEmailDiscussion Board (WebBoard Conferencing)Synchronous text chatStreaming media using Real PlayerDesktop VideoFile LoadingOnline assessmentAudio/video clipsAudio-narrated PowerPoint'sRoom-based Video
ImpactCourse performanceCollaborative learningRetention/attrition (course and program)Professional relevance and utilityLearner productivity
Evaluation Attributes
Multi-sourced data (students, server log files, etc) Internal and external Performance based Comparison and criterion based
Student Course Ratings
3.16
3.26
3.27
3.29
3.32
3.32
3.42
3.45
3.45
1 2 3 4
1
1=Strongly Disagree; 4=Strongly Agree
The instructor wasaccessible
The instructor wasresponsive
Course was relevant todegree
Communicationtechnologies were effective
Course activities wereapplicable
Ample opportunity fordiscussion
Course format appropriatefor content
I enjoyed the course
Confident in conductingdesign
Helpfulness of Technology
3.13
3.19
3.52
3.58
3.61
3.74
1 2 3 4
1
1=Not at all helpful; 4=Very helpful
Personal communicationwith classmates orinstructor
Informational course emails
Electronic chat sessionswith instructor
Electronic chat sessionswith classmates
Course readings on website
Course texts
The breadth of this course was:
1 2 3 4 5 6 7 Not nearly enough The right amount Way too much
Compared to a traditional course
1 2 3 4 5 6 7A much narrower range About the same range A much wider range ofof material was covered of material was covered material was covered
Online: 4.61Compared to traditional 4.65 n=32
The depth of this course was:
1 2 3 4 5 6 7 Not nearly enough The right amount Way too much
Compared to a traditional course
1 2 3 4 5 6 7Material was covered in Material was covered in Material was covered in much less depth about the same depth much more depth
Online: 4.48Compared to traditional 4.42 n=32
The extent of critical thinking required:
1 2 3 4 5 6 7 Not nearly enough The right amount Way too much
Compared to a traditional course
1 2 3 4 5 6 7 Much less About the same Much more
Online: 4.61Compared to traditional 4.94 n=32
The amount of effort put into the course:
1 2 3 4 5 6 7 Much less About the same Much more
Compared to a traditional course
1 2 3 4 5 6 7 Much less About the same Much more
Online: 5.65Compared to traditional: 5.26 n=32
Evaluation Processes
Mostly formativeMixed data collection and analysis methods (document analysis, student artifact analysis, survey, interview, text analysis, etc.)Course and program evaluation
Typical Problems with Online Courses
Facilitating and encouraging collaboration Time managementStudent proficiency with course toolsAmbiguous directionsTimeliness of feedback
Factors Beyond ID Control
Student sophistication with technology tools System capacity Learner availability/accessibility Enthusiastic, responsive instructorGood learner supportMotivated learners
How to Design and Effective Online Course?
Follow basic ID principalsBuild a climate of disclosure and full participation Institute informal student evaluation and check-in mechanisms Active and intensive instructor participation Build in as much interactivity as possibleCreate visually interesting screens/pagesEnsure instructions are very clearMulti-mode interaction is criticalRedundancy