Page 1
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Tips for Providing Fair, Accessible, and Efficient
Online AssessmentsJeff Place, Questionmark
email: [email protected] www.questionmark.com
Page 2
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Role of assessments – IN Learning and OF Learning
Page 3
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Types of Assessments
… IN learning
Formative Assessments◦ Strengthen memory recall◦ Examples:
Quizzes
Diagnostic Assessments◦ Route you to learning◦ Learn what you know◦ Learn what you need to
know◦ Examples:
Learn Quizzes Tests
… OF Learning
Summative Assessments◦ Measure knowledge, skills
and competencies◦ Examples:
Exams and Certifications Compliance tests Pre-employment tests
Reaction Assessments◦ How did you feel
about..?◦ Examples:
Course evaluations Level 1s
Slide 3
Page 4
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.Slide 4
Assessments IN Learning
Results Database – Aggregating results for all interactionsNeeds Analysis, Opinion/attitude surveys, Transcripts, Skills Gap Analysis, Certification Results
AppropriateLearningEvents
Pre-Course
Formative
Post-Course
Page 5
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.Slide 5
Assessments IN Learning
Results Database – Aggregating results for all interactionsNeeds Analysis, Opinion/attitude surveys, Transcripts, Skills Gap Analysis, Certification Results
AppropriateLearningEvents
Pre-Course
Formative
Post-Course
Purpose:• Create intrigue• Route to course• Establish benchmarks
Page 6
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.Slide 6
Assessments IN Learning
Results Database – Aggregating results for all interactionsNeeds Analysis, Opinion/attitude surveys, Transcripts, Skills Gap Analysis, Certification Results
AppropriateLearningEvents
Pre-Course
Formative
Post-CoursePurpose:• Stimulate and strengthen
memory• Correct misconceptions
Page 7
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.Slide 7
Assessments IN Learning
Results Database – Aggregating results for all interactionsNeeds Analysis, Opinion/attitude surveys, Transcripts, Skills Gap Analysis, Certification Results
AppropriateLearningEvents
Pre-Course
Formative
Post-Course
Purpose:• Determine if
knowledge transfer occurred
• Reduce Forgetting Curve
Page 8
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Learning Curve
Slide 8
Know
ledg
e/Sk
ill
Time
Trai
ning Trai
ning
Ouch! The Forgetting Curve
Page 9
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Questions IN the Learning Process
Slide 9
Know
ledg
e/Sk
ill
Time
Redu
ce F
orge
tting
Redu
ce F
orge
tting
Crea
tes
Intri
gue
Stre
ngth
ens M
emor
y
Questions….
Page 10
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Assessments
… IN learning
Formative Assessments◦ Strengthen memory recall◦ Examples:
Quizzes
Diagnostic Assessments◦ Route you to learning◦ Learn what you know◦ Learn what you need to
know◦ Examples:
Learn Quizzes Tests
… OF Learning
Summative Assessments◦ Measure knowledge, skills
and competencies◦ Examples:
Exams and Certifications Compliance tests Pre-employment tests
Reaction Assessments◦ How did you feel
about..?◦ Examples:
Course evaluations Level 1s
Slide 10
Page 11
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.Slide 11
Assessments After LearningKirkpatrick’s Levels of assessment
Level 1 - ReactionWas the environment suitable for learning?
Level 2 - KnowledgeDid they learn anything
Level 3 - BehaviorKSA being used on the job?
Level 4 - ResultsWas it worth it?
Assessment
Assessment
Assessment
Page 12
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Assessments must: Provide the Right Stimulus to Gather Data
...then Provide the Right Analysis
with Right Presentationto the Right Person at the Right Time
Be Valid and Reliable Be Efficient and Cost Effective
Page 13
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Tip: Accommodating Test Takers with
Disabilities Pay particular attention to the authoring and delivery of assessments
Page 14
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
People with disabilities deserve to be assessed fairly and sometimes Accommodations must be made
Following good practice in web accessibility not only helps those with disabilities but many others◦ Workers not used to computers can use touch
screens◦ Temporary injuries incapacitating participants
Increasingly important for government, academic and corporations to empower and harness the intellect and energy of those with disabilities
Accommodations and Accessibility
Page 15
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
International definitions of blindness and low vision / partial sightedness varies, but there are tens of 1,000s of people of working age impacted
Accommodations include:◦ Screens to be compatible
with text to speech screen readers
◦ Allow participant to: Select large text Change colors and contrast
Blindness and Partially Sighted
Page 16
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Many types of color “blindness” People see colors differently to
the norm ~7% of men and ~0.5% of
women are color blind Accommodations include:
◦ Not using only color to distinguish information or navigation
◦ Not relying on color within the question or choices to get the right answer
Color Blindness
Colors of rainbow as viewed by a person with no color vision deficiencies.
Colors of rainbow as viewed by a person with protanopia.
Colors of rainbow as viewed by a person with deuteranopia.
Colors of rainbow as viewed by a person with tritanopia.
Page 17
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Dyslexia challenges people to read and spell ~10% of people are impacted; 4% severely Most sufferers muddle through Accommodations include:
◦ Allow participant to: Select larger text Change colors and contrast Use (text to speech) screen readers Have extra time
◦ Not using only color to distinguish information or navigation
◦ Not relying on color within the question or choices to get the right answer
Dyslexia
Page 18
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Some disabilities (multiple sclerosis, paralysis, muscle and joint problems) prevent use of mouse/keyboard and require special input devices
Number of people affected:◦ Quadriplegia (often caused by
vehicle or sports injury) ~100,000 people in US ~25,000 people in UK
Accommodations include:◦ Ensure assessment system
can be used by keyboard (not mouse) as such special input devices emulate keyboard
Motor Disabilities
Page 19
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
W3C (WCAG)http://www.w3.org/TR/WCAG10/ (WCAG 1, 1999)http://www.w3.org/TR/WCAG20/ (WCAG 2, Dec 2008)
Section 508 from the US government◦ Main document (web applications):
http://www.access-board.gov/sec508/guide/1194.22.htm
◦ Secondary document (software): http://www.access-board.gov/sec508/guide/1194.21.htm
◦ Section 508 similar to WCAG but not identicalwww.jimthatcher.com/sidebyside.htm
Guidance and Standards
Page 20
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Optimize delivery for screen-reader compatibility
Provide text-sizing and contrast controls
Enable navigation without mouse Keyboard shortcuts Alternate pointing devices
Accessibility and Accommodations
Page 21
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Text-to-Speech Screen readers Market leader text-to-speech screen reader
is JAWS (75% of market) www.freedomscientific.com/jaws-hq.asp
◦ Costs about US$1,000/seat◦ Demo version available at
http://www.freedomscientific.com/downloads/jaws/jaws-downloads.asp allows 40 mins use before restarting
Page 22
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Allowing Extra Time
Page 23
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Provide Accessibility Review
Workflow feature ensure review of accessibility issues
Page 24
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Provide scribes and proctors to help those with disabilities answer assessments
Page 25
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Expand “Clickable” Regions
Select by clickingform field or choice text region
Page 26
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Text Sizing Controls
Page 27
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Contrast controls
Page 28
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Tip: Consider how you plan to score a test or exam before
you begin writingRecommended reading:
Criterion Referenced Test Development: Technical and Legal Guidelines for Corporate Training
Sharon A. Shrock and William C. Coscarelli
Slide 28
Page 29
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
CRT vs. NRT
Criterion ReferencedHow well did the
participant master the content and/or skill
Typically used for certification tests,
licensing exams, end of course exams, etc.
Norm ReferencedHow well did the
participant perform compared to others in a
group
Typically used for standardized college
placement tests, ‘competition’ tests (e.g. advancement test for
limited number of openings)
Page 30
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Informed Judgment◦ Poll SMEs to determine criteria for passing (no. of correct
answers, no. of attempts)◦ Not recommended for high-stakes assessments
Angoff Method◦ Poll SMEs◦ Consider marginal candidates and the probability of getting specific
questions right (0-100%)◦ Make a mathematical calculation to determine cut score
Borderline Group Method◦ Candidates are interviewed prior to an exam and borderline cases are
identified◦ After the exam, the scores of borderline candidates can be used to
determine the cut score Contrasting Group
◦ Compare SME to others
Setting cut scores for Criterion Referenced Tests
Slide 30
Page 31
Angoff MethodWhat is the expected possibility of a marginal student getting question right?
Expert A
Expert B
Expert C Agreement
Question 1 70% 75% 60% 68%Question 2 55% 65% 60% 60%Question 3 60% 65% 60% 62%Question 4 70% 65% 60% 65%Question 5 25% 75% 80% 60%Question 6 60% 75% 75% 70%Question 7 55% 45% 80% 60%Question 8 80% 80% 85% 82%Question 9 85% 75% 75% 78%Question 10 70% 65% 70% 78%
63% 69% 71% 67%
Slide 31
Page 32
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Norm Reference Testing (NRT): The “Normal Curve”
Slide 32
Page 33
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
NRT: Test designed to “spread” and rank participants within a group; so to easily identify top performers
Slide 33
Top 10% of scorers
Page 34
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.Slide 34
Setting a Pass/Fail score?
Norm ReferencedIf used to determine a ‘cut’ or ‘pass/fail’,
the pass/fail score is determined after test
Criterion ReferencedPass/fail score determined before test
175 people
Only 50 peoplecan pass
175 people
1st group
55 pass
2nd group
35 pass
3rd group
71 pass
Page 35
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Tip: Outline a “Test Blue Print” before you begin writing your
test or exam questionsEnsure your test meets your objectives
Improve efficiency of test writing
Slide 35
Page 36
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Assessment Blue Print A detailed, written plan for an assessment Typically includes
descriptions of the purpose and target audience the topics and performance areas it will cover the types of items and number to be written for each
topic or performance area scoring timing the administration method (paper, browser, CD) desired reports, distribution of results decisions made based on results
Page 37
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Blueprint for Nursing Licensure Exam
Page 38
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Tip: Consider options for gathering content from
globally dispersed SMEs
Page 39
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Using a distributed authoring and centralized item management with globally dispersed authors
Internet
Page 40
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Can you provide multilingual interfaces to SMEs?
Page 41
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Tip: When delivering localized/multilingual
assessments, consider the workflow of your translation
process.
Page 42
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Efficient Models for Translating Items, Quizzes and Tests
Translated Item
Base Language
Item
English
FrenchGerma
nDutch
Translated Test
Base Language
Test
English
FrenchGerma
nDutch
Page 43
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Interactive Translations
3rd-party localisation using industry-standard format
Two Translation Methods
Export to XLIFF (XML)
Translate using 3rd
Party tool or service
Import from XLIFF (XML)
Translate withinan assessment authoring tool
Define translation
project
On-The-Fly
Page 44
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Multilingual Item Banking
Page 45
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Example of localization software: Trados Studio
Slide 45
Page 46
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Tip: Align delivery environment to the stakes of an assessment
Slide 46
Page 47
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Higher Stakes
High
Medium
Low
Medium Stakes
Low Stakes
Low/High to High/High Stakes Tests
Slide 47
What’s at Stake?
Life and Limb
Promotion & Jobs & Legal Concern
Educational Exams
Tests
Elearning & Surveys
Page 48
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Issues to Consider
Fraud Triangle Threats
Rationalization Opportunity Motivation
Impersonation Content Theft Cheating
Page 49
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.Slide 49
High stakes: Secure Browser Mitigates risks of:
◦ Cheating◦ Items being leaked/shared with
future test participants
How it works…◦ No menu or icons◦ Cannot switch to another
program◦ Disables printing◦ Can be customized to permit
certain functions, i.e. printing feedback page Secure Browser
Regular Browser
Page 50
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.Slide 50
Other risks to mitigate Power Outage? Loss of Internet connection?
Ensure there is an “auto save” or “save as you go” mechanism:◦Answers auto-saved as the
participant takes test◦In the event of technical failure,
participants can resume an assessment where they left off
Page 51
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Tip: True “Blended” delivery of assessments should
improve more than your delivery options
Blended delivery should also yield more efficiency in authoring, administration and reporting
Page 52
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.Slide 52
Secure Browser
Blended Delivery
Web Browser Disconnected Delivery
LMS
Integration Windows / CDTest Centers
Printing andScanning
Mobile Devices
Social Networking Platforms Embedded in Wikis, Blogs, and Web pages
Page 53
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
What is “disconnected delivery?
Slide 53
Page 54
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Disconnected delivery – content downloaded before event, results synched after test
Participants
Thumb Drive
Assessment Content
AssessmentsResults
CD
SD Card
Synch with Central Assessment Server
Synch with Central Assessment Server
Page 55
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Participant Answers on
Paper
Printing and Scanning – An option when workstations, Internet not available
Slide 55
Print Test Form
Assessment Preparation
Paper Scanned
Paper
Responses Uploaded to Central Server for
Scoring and Reporting
Page 56
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
What is “auto sensing”?
Slide 56
Page 57
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Auto Sense Device/Browser
Senses Device/Brows
er
Server packages the Assessment for Device/Browser/iFrame
Sends HTML
On-the-fly auto-sizing of skin and controls
Page 58
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Inexpensive, very portable Internet-connected mobile devices
Page 59
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
The benefits?
Slide 59
Deliver to any device or context
Schedule once
Author once
Single results set
Page 60
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Investigate which tools support development/delivery for mobile devices
Review W3C’s Mobile Web Best Practices: Consider potential device limitations – e.g. screen size,
plug-in support (e.g. Flash) Avoid pop-ups, frames, image maps and use of tables
for formatting Ensure simple, consistent navigation See full best practices guide at:
http://www.w3.org/TR/mobile-bp/ Consider the benefits of “auto sensing” of
participant’s device/browser
Delivering assessments to mobile devices?
Slide 60
Page 61
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Tip: Make effective use of corrective feedback
Enhances learningReduces forgetting
Slide 61
Page 62
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
What is feedback? Stimulus is provided to a participant
according to their responses within an assessment.
Feedback can be provided…◦ Based on how a participant scores and/or
answers… A single question A group or “topic” of questions
◦ Or, based on how a participant scores on an assessment
Slide 62
Page 63
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Simple corrective question feedback
Slide 63
Question and
participant’s answer
Corrective feedback
given
Page 64
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Feedback based topics of questions
Slide 64
Corrective feedback points participant to appropriate
learning resources
“Topics” of questions
answered in assessment
Score on
topic
Page 65
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
General guidelines on feedback… Improving performance
Give the participant guidance for improving their skill or knowledge
PrecisionBe precise enough that the participant knows what needs to be improved
Learning resourcesTell the participant how and where to go to acquire the information
Slide 65
Page 66
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
When is feedback useful? Dealing with difficult questions
Helps participant understand, learn from error
Correcting major errorsErrors that demonstrate a fundamental misunderstanding of a topic
Slide 66
Page 67
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Times you might avoid giving feedback… When security is important
Where doing so may compromise the integrity of the test as a whole
When questions are relatedWhen it may provide clues to answer other questions
Slide 67
Page 68
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Tip: Use item analysis and other techniques to ensure
valid, reliable measurementsHow do you know it’s a good test?
Slide 68
Page 69
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Reliability and Validity Assessments must be reliable and valid
◦ Reliable = They work consistently
◦ Valid = They ‘look like the job’
If an assessment is not reliable it cannot be valid
Page 70
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Reliable = ConsistentNot Valid = Not like the job
Page 71
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Not Reliable = Not ConsistentNot Valid = Not like the job
Page 72
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Reliable = ConsistentValid = Like the job
Page 73
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Types of Validity Face Validity
◦ Do test-takers think the test measures what it’s supposed to measure?
Content Validity◦ Do SMEs agree that the test measures what it’s supposed
to measure? ◦ Based on review of objectives and items as specified in
assessment blueprint Concurrent Validity
◦ Does the test accurately classify known experts and known novices?
◦ Correlation calculation
Page 74
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Assessment analytics for reliabilityType of analysis
What you aim to learn…
Analysis of Items (Questions)
•Is the question too easy? Too difficult?•How many results for this question?•Are there potential problems with this question (such as ambiguous wording)?•Has this question been over-exposed?
Test analysis Is the test reliably measuring what is was designed to measure? Are these assessment results “defensible” (will they withstand an appeal or legal challenge?)
Slide 74
Page 75
Slide 75
Item Statistics- ChoicesChoices
# of Responden
ts%
Selected
Difficulty(also p-value)
0 = Too difficult1 = Too easy
Participant Mean
OutcomeDiscrimin
-ation
Outcome Correlatio
n
A
B
C
D
Page 76
Slide 76
Item Statistics- # of respondentsChoices
# of Responden
ts%
Selected
Difficulty(also p-value)
0 = Too difficult1 = Too easy
Participant Mean
OutcomeDiscrimin
-ation
Outcome Correlatio
n
A 21
B 52
C 323
D 112
Page 77
Slide 77
Item Statistics- % of respondentsChoices
# of Responden
ts%
Selected
Difficulty(also p-value)
0 = Too difficult1 = Too easy
Participant Mean
OutcomeDiscrimin
-ation
Outcome Correlatio
n
A 21 4%
B 52 10%
C 323 64%
D 112 22%
Page 78
Slide 78
Item Statistics- Difficulty (also P-value)Choices
# of Responden
ts%
Selected
Difficulty(also p-value)
0 = Too difficult1 = Too easy
Participant Mean
OutcomeDiscrimin
-ation
Outcome Correlatio
n
A 21 4%
B 52 10%
C 323 64% .64
D 112 22%
Page 79
Slide 79
Item Statistics- Participant MeanChoices
# of Responden
ts%
Selected
Difficulty(also p-value)
0 = Too difficult1 = Too easy
Participant Mean
OutcomeDiscrimin
-ation
Outcome Correlatio
n
A 21 4% 36%
B 52 10% 47%
C 323 64% .64 66%
D 112 22% 52%
Page 80
Slide 80
Item Statistics- Outcome DiscriminationChoices
# of Responden
ts%
Selected
Difficulty(also p-value)
0 = Too difficult1 = Too easy
Participant Mean
OutcomeDiscrimin
-ation
Outcome Correlatio
n
A 21 4% 36%
B 52 10% 47%
C 323 64% .64 66% 0.49
D 112 22% 52%
Page 81
Slide 81
Item Statistics- Outcome CorrelationChoices
# of Responden
ts%
Selected
Difficulty(also p-value)
0 = Too difficult1 = Too easy
Participant Mean
OutcomeDiscrimin
-ation
Outcome Correlatio
n
A 21 4% 36% -0.32
B 52 10% 47% -0.22
C 323 64% .64 66% 0.49 0.24
D 112 22% 52% -0.15
Page 82
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.Slide 82
Other question and item statistics… Number of participants who missed a question
◦ Are an unexpected number of participants are missing a question (i.e., not responding)? Could be a problem with the question.
The number of responses for each question◦ The more results for a question, the more stable the
information Better assurance that the statistics generated are trustable
# Times item has been presented◦ The more times that a question is presented, the more
the question is “exposed.” ◦ Tracking item exposure vital to retiring questions to avoid
questions from being “over exposed”
Page 83
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Item Analysis - Summary Items with zero or negative discrimination
are suspect Items with zero or negative correlation are
suspect Pay attention to distractor pattern
◦ Distractors with low p-values are probably unrealistic and are therefore discounted
Page 84
Test Reliability: Cronbach’s Alpha Internal consistency test
reliability: A measure of how well the assessment questions relate to one another to measure the same topic
Crucial information for legal defensibility – is the test reliably measuring what is was designed to measure?
Speaks to the quality of assessment questions: Are they of high statistical quality?
)(1 2
22
T
pT
SSS
kk
Page 85
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.Slide 85
Learn about assessments… Questionmark White Papers
◦ Assessments through the Learning Process◦ Delivering Assessments Safely and Securely ◦ Using Questionmark Perception to Help Ensure Compliance
with Regulatory Principles for e-Assessment ◦ Measuring Learning Results◦ Providing Feedback to Learners◦ Using Online Assessment for Compliance◦ The Learning Benefits of Questions
◦ www.questionmark.com/whitepapers
Questionmark Blog◦ blog.questionmark.com
Page 86
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
What questions do you have?Jeff Place, Questionmark
email: [email protected] www.questionmark.com
Page 87
Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Thank you!Jeff Place, Questionmark
email: [email protected] www.questionmark.com