Testing to Evaluate Learning New Faculty Orientation 2005 Dr. Diane Wink Professor School of Nursing College of Health and Public Affairs ©Diane Wink 2005.

Testing to Evaluate Testing to Evaluate LearningLearning

New Faculty Orientation New Faculty Orientation 20052005Dr. Diane Wink

ProfessorSchool of Nursing

College of Health and Public Affairs

©Diane Wink 2005

WelcomeWelcome

To UCF

Why test??Why test??

Achievement of objectives of learning measured with

evaluation



evaluation



evaluationClass activities must reflect objectives and what will be

testedClass

Experience

and Activities

ObjectivesObjectives

• All objectives should lead directly to evaluation through the learning experience itself.

• Evaluation should be of the objectives.

Class

Experience

and

Activities

Questions to askQuestions to ask

• What is the best evaluation method to determine student mastery and application of this knowledge?

Questions to ask - Test Questions to ask - Test AlternativesAlternatives

• Term paper• Class activity

– Debate and bib development

• Problem set completion• Service Learning activity• Course project

– Case Study– Development of tool

• Non written evaluation– Performance of the

competency being developed

For Testing: Questions to For Testing: Questions to askask

• How will this test help me know if learning objectives are being met?

• Is the test addressing essential information?

• Does the test reflect the readings, class activities and/or web discussion?

Questions to askQuestions to ask

• How many questions will it take to determine if student has met objectives/ has the knowledge?

Do a test blueprint!

Test BlueprintTest Blueprint

Written TestsWritten Tests

• Multiple Choice• True-False• Matching• Essay and Short answer essay

For all types of examsFor all types of exams

• Have a colleague review your items

• Consider the possibility of cheating– Alternative forms of tests– Random seating– 3X5 key points card– New exams– No cell phones, PDAs– Open book if on-line– In person for on-line

Multiple Choice ItemsMultiple Choice Items

• The stem should:– Contain a self-contained question or

problem– Present as much of the item's content

as possible– Be precise - no extraneous

information


• The stem should:– Be independent of all other stems.

If part of a situation, say this clearlyDo not make answer dependent on

getting a prior item correct

– Be a positive rather than a negative statement or question

Should not be a reading test!


• The answer choices should:– Contain only one best or correct answer– Be grammatically consistent with stem– Be similar in complexity, detail, length

and plausibility– Avoid "all of the above" or "none of the

above"


• The answer choices should:– All be plausible– Be arranged logically

Alphabetical, numerical, length

– Randomly use each alternative position for correct answers

– Contain action verbs from Bloom's Taxonomy


• The answer choices should:– Be on the same page as the stem – Not have obscure or esoteric

language

Sample Item- PoorSample Item- Poor

If all of the following heart valves are defective but only one can be repaired, which should it be?

a. Aorticb. Mitralc. Pulmonicd. Tricuspid

Sample ItemSample Item

Narrowing and tightness of which heart valve is most likely to result in left ventricular enlargement?

a. Aorticb. Mitralc. Pulmonicd. Tricuspid

True False ItemsTrue False Items

• State the item as simply and clearly as possible

• Keep statements equal in length• Write each item so that it is clearly

true or false

True False ItemsTrue False Items

• Avoid qualifications such as "often" or "usually"

• Randomize the true and false items

• Test important information and facts

Sample T/F Item Sample T/F Item (Hard to score)(Hard to score)

15. For each of the following, choose A if true and B if false.

a. The major pump chamber of the heart is the ventricle

b. Valves control blood flow into the atriac. The pulmonary artery carries fully

oxygenated blood to the lungs

Sample T/F Item Sample T/F Item (Easy to score)(Easy to score)

For each of the following, choose A if true and B if false.

15. The major pump chamber of the heart is the left ventricle

16. Valves control blood flow into the atria17. The pulmonary artery carries fully

oxygenated blood

Matching ItemsMatching Items

• Put the basis of relationship for matching in the directions

• Put the entire set of matching items on a single page

• Arrange items in a systematic order• Number premises on the left and

letter the responses on the right a-eAllows computer grading

Matching ItemsMatching Items

• Use a larger or smaller set of responses and allow responses to be used more than once or not at all

• Use homogeneous lists that concern the same content

• Number each item separately to allow computer grading

Sample Matching Item - Sample Matching Item - PoorPoor

31. Match each item in Column A with an historical event listed in Column B. Events in Column B can be used more than once.

Hoover a. Implemented emancipation proclamation

Roosevelt b. Established the CCCLincoln c. Major war ends Clinton d. Stock market crash

f. Was impeached

Sample Matching Item - Sample Matching Item - PoorPoor

31. Match each item in Column A with an historical event listed in Column B. Events in Column B can be used more than once.

Hoover a. Implemented emancipation proclamation

Roosevelt b. Established the CCCLincoln c. Major war ends Clinton d. Stock market crash

f. Was impeached

All questions (there are actually four) have same item number!

Confusion over which Roosevelt, which stock market crash?

Sample Matching - BetterSample Matching - BetterMatch each of the presidents listed in Column A with the historical

event listed in Column B. Events in Column B can be used more than once.

Column A Column B31. Hoover a. Post civil way reconstruction in

progress in US32. F.D. Roosevelt b. Establishment of the

CCC33. Lincoln c. WWII ends34. Clinton d. Stock market crashed starting depression

f. Impeached

Exam grading support at Exam grading support at UCFUCF

University testing service support for multiple choice/true false/matching exams

WebCTDisability

Services

Grading issuesGrading issues

What if the item is bad??!!

Essay/Short Answer ItemsEssay/Short Answer Items

What do they test?Higher cognitive levels

ApplicationAnalysisSynthesisEvaluation

Not for simple recall of facts

Essay/Short Answer ItemsEssay/Short Answer Items

• First ask– Is item clearly written? – If extended response needed, is it too

broad? – If restricted response needed, would

the content be assessed better with an objective item?


• Poor: List the components of a persuasive letter to a government official.

• Better: Write a letter to a government official in support of or opposed to gun control.


• Poor: What are the themes in War and Peace?

• Better: Relate a theme in War and Peace to world events in 2005

Grading essay/short essay Grading essay/short essay itemsitems

Write out the answer Does your answer

match the intent of the item?

What problems did you encounter in answering the item?

Can the student answer in the time allowed?

Grading essay/short answer Grading essay/short answer itemsitems

• Analytical method• Points for each essential element

• Holistic method• Look for essential elements• Evaluate overall quality of response• Little feedback

Analytical Grading Method Analytical Grading Method

Identify criteria for evaluating the content of the response and assign points for each:– Essential elements required– Accuracy of content– Logical organization– Clarity– Depth and complexity of analysis– Use of references and specific citations ** Usually only used in formal papers

Analytical Grading MethodAnalytical Grading Method

Decide on the extent to which writing ability and use of style format will be included in the scoring– Spelling– Grammar– Sentence structure– Writing style– APA citation and format ** Usually only used in formal papers

Holistic Grading MethodHolistic Grading Method

Sample “A” answer:

Covers all key points with citations to major resources. Clearly written with analysis of problem clear and persuasive. Minimal to no grammatical or spelling errors.


Sample “B” answer:Covers most key points with citations to most major resources. Clearly written with good analysis of problem. Few grammatical or spelling errors.

Sample “C” answer:Covers few key points with few references to major resources. Written poorly with limited analysis of problem and a non persuasive argument Multiple grammatical or spelling errors.


Sample “D” Answer:Omits most key points with few/no citations to resources. Writing difficult to follow with consistent grammatical and spelling errors

Sample “F” answer:Omits all key points, does not answer question

Grading essay and short Grading essay and short answer itemsanswer items

Before scoring Read a random sample of papers Get a sense of how students answered Revise evaluation criteria if needed Review grading criteria with any other graders -

practice!!! Do key for commonly made comments (give to

students and all graders)1. Great job. Shows excellent analysis of your main points 2. Great ideas but needed to be developed more3. Unclear what you mean 4. Needs citation


Ideal grading technique:– Score answers to one question at a

time for all students– Read each answer twice

First for omissions of essential elementsSecond reread and score


• Ideal grading technique:– Cover up prior scores to items or

record them on a separate answer sheet

– Read papers anonymously; have students use a code number


• If fatigued, stop and rest!

Grading essay/short answer Grading essay/short answer itemsitems

•GTA•Help from peers

Consider time for grading when designing exams

Questions???

Bloom’s TaxonomyBloom’s Taxonomy

CognitiveKnowledge

Identification and recall of information

ComprehensionOrganization and selection of facts and

ideas

ApplicationUse of facts, rules principles


Cognitive (continued)Analysis

Separation of a whole into component parts

SynthesisCombination of ideas to form a new whole

EvaluationDevelopment of opinions, judgments, or

decisions


Affective DomainFeeling toneEmotionDegree of acceptance or rejection

Psychomotor Domain

Resources UsedResources UsedOermann, M. (1999). Developing and scoring essay tests. Nurse

Educator, 24(2), 29-32.Tomey, A. (1999). Preparing a testing blueprint from levels of

course objectives. Nurse Educator, 24(4), 11-12.Tomey, A. (1999). Selected response test items. Nurse Educator,

24(5), 9-10.Tomey, A. (1999). Completion testing items. Nurse Educator,

24(6), 6-7.

Testing to Evaluate Learning New Faculty Orientation 2005 Dr. Diane Wink Professor School of Nursing College of Health and Public Affairs ©Diane Wink 2005.

Documents

evaluation slide

online slide

test blueprint slide

tricuspid slide

ucf slide

learning objectives

extraneous information

blooms taxonomy slide