Slide 1 of 51 Course BIONF 2201 February 20, 2004 Evaluating educational software Evaluating educational software Instructor: Heiko Spallek, DMD, PhD Based on papers/presentations by: T. Schleyer, DMD, PhD, University of Pittsburgh L. Johnson, PhD, University of Iowa D. Rubright, MFA, MA, University of Iowa H. Spallek, DMD, PhD, University of Pittsburgh February 20th, 2004 Dental Information Systems #2201
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Based on papers/presentations by: T. Schleyer, DMD, PhD, University of PittsburghL. Johnson, PhD, University of IowaD. Rubright, MFA, MA, University of IowaH. Spallek, DMD, PhD, University of Pittsburgh
February 20th, 2004Dental Information Systems #2201
BIONF 2201
Evaluating educational software
February 20, 2004Slide 2 of 51
A framework for evaluating educational software
Asking the learners … Guidelines for the Design of Educational
Author not indicated: 71% No goals and objectives: 23% No references: 85%
BIONF 2201
Evaluating educational software
February 20, 2004Slide 18 of 51
Navigation
Direct indication of progress: 23% Indirect indication of progress: 45% Progress actively obscured: 32% Navigation: approx. 60% easy or very
easy to navigate
BIONF 2201
Evaluating educational software
February 20, 2004Slide 19 of 51
Interaction
No e-mail contact possible: 47% Author’s e-mail listed: 24% Other e-mail: 29%
BIONF 2201
Evaluating educational software
February 20, 2004Slide 20 of 51
Discussion Limitations
– design guidelines are preliminary only– study used only a subset of design criteria– some criteria subjective (navigation, length)– not certain that all online courses in
dentistry were found– password-protected courses not reviewed
BIONF 2201
Evaluating educational software
February 20, 2004Slide 21 of 51
Discussion Variation of credit/hours vs course length Dearth of true multimedia courses Testing and feedback uses Internet
capabilities only marginally Low compliance with accepted
standards for educational materials
BIONF 2201
Evaluating educational software
February 20, 2004Slide 22 of 51
Discussion (cont.)
Poor use of navigational design Interaction obviously not desired in
most courses Advanced functions of educational
software not used (e.g. customization after pretest)
BIONF 2201
Evaluating educational software
February 20, 2004Slide 23 of 51
Recommendations
Disseminate Guidelines widely Online CDE should be peer-reviewed Develop valid instruments for assessing
courses Insert TITLE and KEYWORD tags into
HTML Establish central index of courses
Slide 24 of 51Course BIONF 2201 February 20, 2004
Evaluating educational software
Evaluation of Web-based Dental CE-CoursesHeiko Spallek DMD PhD*, Elizabeth Pilcher DMD**, Ji-Young Lee***, Titus Schleyer DMD PhD*
*Center for Dental Informatics, University of Pittsburgh, School of Dental Medicine, Pittsburgh, PA **Medical University of South Carolina, College of Dental Medicine, Charleston, SC***Temple University School of Dentistry, Philadelphia, PA
Study goals
evaluate the outcomes of online CDE courses through analysis of 6 organizations focused on
how the participants of online CDE can be characterized whether the participants' expectations were met by the courses how the participants evaluated the content of the courses why they enrolled how they experienced the online environment
→ develop recommendations for the design of future courses
Slide 25 of 51Course BIONF 2201 February 20, 2004
Evaluating educational software
Study design exploratory study
survey of 436 past course participants from 9 online CDE courses from 6 organizations (health care schools and commercial CE providers)
courses varied in content, length, type of provider, tuition
inclusion criteria were continuing education courses in dentistry that granted continuing education credits online for at least a year
Slide 26 of 51Course BIONF 2201 February 20, 2004
Evaluating educational software
Course title Organization
Features Online since Number of former participants
CE credit hours / Accreditation
Nitrous Oxide Conscious Sedation, University of Texas Health Science Center, San Antonio, Dental School
Preview section, quizzes, can download in pdf
Oct. 1998
61 9
Top 40 Drugs, Medical University of South Carolina, College of Dental Medicine
Quiz, e-mail instructor, outside links Sept. 1998
161 2
Dentistry on the Internet, Temple University School of Dentistry
pre- and post-test, class listserve, quiz, communication with the course instructor
Oct. 1997 113 3
Submitting an Invention to a Dental Manufacturer, University of Michigan, School of Dentistry
Post test June 1997
21 1
Asthma Procter & Gamble
Course test Not Avail.
40 2
Introduction to Composite Dentistry DentalXchange.com
Course test Nov. 1999
10 2
Treating the Unscheduled Dental Emergency DentalXchange.com
Course test Nov. 1999
10 3
Tooth Bleaching DentalXchange.com
Course Test Mar. 1999
10 2
Tricks of the Trade in Endodontics DentalXchange.com
Course test Jan. 1999
10 2
Evaluated CDE courses
Slide 27 of 51Course BIONF 2201 February 20, 2004
Evaluating educational software
Study design: survey"The Tailored Design Method" by Don Dillman
self-administered, dual-mode (e-mail and postal mail), partially branched survey (concise, limited the number of open-ended questions)
→ instrument available at http://di.dental.pitt.edu/cesurvey/
Slide 28 of 51Course BIONF 2201 February 20, 2004
Evaluating educational software
Results/discussion: response rate
Response by Course Solicitations Respondents Rate
completed with CE certification
Asthma 40 20 50.00% 20 Dentistry on the Internet 113 42 37.17% 12 Nitrous Oxide Conscious Sedation 61 44 72.13% 5 Submitting an Invention to a Dental Manufacturer 21 10 47.62%
10
Top 40 Drugs 161 41 25.47% 6 Tricks of the Trade in Endodontics 10 5 50.00% 5 Treating the Unscheduled Dental Emergency 10 2 20.00%
1
Introduction to Composite Dentistry 10 4 40.00% 3 Tooth Bleaching 10 1 10.00% 1 Total 436 169 38.76% 63
Slide 29 of 51Course BIONF 2201 February 20, 2004
Evaluating educational software
Results/discussion: demographics
Responses to the question “On the whole, how sophisticated a computer user do you consider yourself?”
Self Assessment Category Number of Reponses Percentage very unsophisticated 18 14% unsophisticated 14 11% neither sophisticated nor unsophisticated 48 37% sophisticated 32 25% very sophisticated 18 14% Total 130 100%
Slide 30 of 51Course BIONF 2201 February 20, 2004
Evaluating educational software
Results/discussion: marketing
Responses to the question “How did you learn about this particular course?
19% Internet search engine 15% course provider's homepage 15% personal recommendation 9% professional journal 2% alumni journal 10% other sources
Slide 31 of 51Course BIONF 2201 February 20, 2004
Evaluating educational software
Results/discussion: marketing
Participants perceived respectability of an online CDE course:
Agreement with the statement "I am more likely to tell others about my participation in an online course than my participation in a traditional classroom-based lecture."
19% strongly agreed with this statement 11% somewhat agreed 24% neither agreed nor disagreed 6% somewhat disagreed 1% strongly disagreed 8% no opinion
Slide 32 of 51Course BIONF 2201 February 20, 2004
Evaluating educational software
Results/discussion: online environmentTime spent working online for the course?
"Dentistry on the Internet" => 14.5 hours"Submitting an Invention to a Dental Manufacturer" => 1 hour
When?27% accessed the course material during work hours79% after work hours 6% specified both
From where?31% home64% their office3% a library
Slide 33 of 51Course BIONF 2201 February 20, 2004
Evaluating educational software
Results/discussion: online environment What was the single most important reason that attracted you to an online course
“convenience” 47%
What was the biggest disadvantage to the online format?“lack of human interaction” 13% “cannot ask questions” 12%
The lack of face-to-face contact with a teacher was a stumbling block for your learning
agreed 18% disagreed 65% neither agreed nor disagreed 17%
Slide 34 of 51Course BIONF 2201 February 20, 2004
Evaluating educational software
Results/discussion: meeting initial expectations
courses ranked equally well in most categories– "exploit the convenience of online learning" – "fit the course into my schedule"
problem spots– cannot "communicate with peers online" – cannot "interact one-to-one with the instructor"
authors' experiences suggest– participants seldom raise content-related questions – since inception: the instructor of the course “Top 40 Drugs” received a total of four content-related questions
Slide 35 of 51Course BIONF 2201 February 20, 2004
Evaluating educational software
Conclusions Evaluated online CDE courses do meet some of the needs and expectations of dental professionals.
generally limited number of participants → no ROI *
participants mainly originated in the United States
recommendations for online course development →→→→→→→→→
Carr, Sarah. Is anyone making money on distance education? The Chronicle of Higher Education 2-16-2001
Slide 36 of 51Course BIONF 2201 February 20, 2004
Evaluating educational software
Recommendations
Online CDE courses need to:
be current cover the subject matter in-depth be guided by an experienced instructor define average time necessary to complete the entire course be marketed among dental professionals
BIONF 2201
Evaluating educational software
February 20, 2004Slide 37 of 51
Pedagogical Issues
Benefits of the computer Instructional techniques
match the audience Instructional techniques
match the content
Assessment strategy Customizable content Content is reinforced Interactions vary
How well does it teach? Are activities appropriate for the audience, objectives and content?
How well does it teach? Are activities appropriate for the audience, objectives and content?
BIONF 2201
Evaluating educational software
February 20, 2004Slide 38 of 51
Pedagogical Issues
Benefits of the computer
www.lib.uiowa.edu/commons/skullvr/index.html
BIONF 2201
Evaluating educational software
February 20, 2004Slide 39 of 51
Pedagogical Issues
Instructional techniques match audience and content
D4: Dx & Tx PlanningD4: Dx & Tx Planning
DiagnosticBytesDiagnosticBytes
D1 & D2: AssessmentD1 & D2: Assessment
Assessment of Geriatric Patients
Assessment of Geriatric Patients
BIONF 2201
Evaluating educational software
February 20, 2004Slide 40 of 51
Pedagogical Issues
Assessment Strategy
Diagnosis of Head and Neck PainDiagnosis of Head and Neck Pain
BIONF 2201
Evaluating educational software
February 20, 2004Slide 41 of 51
Pedagogical Issues
Interactions vary
Diagnosis of Head and Neck PainDiagnosis of Head and Neck Pain
BIONF 2201
Evaluating educational software
February 20, 2004Slide 42 of 51
Information is complete, accurate, & logically organized
Subject MatterIs the content accurate and appropriate for the audience?
Is the content accurate and appropriate for the audience?
Method: Heuristics Evaluation = systematic inspection of a user interface design for usability
most popular usability inspection method goal: find usability problems ( fix them as part of an iterative design process) small set of evaluators examine the interface evaluator judge compliance with recognized usability principles (the "heuristics") general heuristics + category-specific heuristics
Slide 80 of 51Course BIONF 2201 February 20, 2004
Evaluating educational software
Expert Review
Ten Usability Heuristics by Jakob Nielsen
Visibility of system status Match between system and the real world User control and freedom Consistency and standards Error prevention Recognition rather than recall Flexibility and efficiency of use Aesthetic and minimalist design Help users recognize, diagnose, and recover from errors (bad, good example) Help and documentation
Nielsen, J. (1994). Heuristic evaluation.
In Nielsen, J., and Mack, R.L. (Eds.), Usability Inspection Methods, John Wiley & Sons, New York, NY.