FOR STANDARDS-BASED ASSESSMENTS TEST ITEM WRITING
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
F O R S TA N D A R D S - B A S E D A S S E S S M E N T S
TEST ITEM WRITING
AGENDA
8:00 – 10:00 Item writing guidelines
10:00 – 10:30 Write items
10:30 – 11:30 Vet items as a group
11:30 – 12:30 Lunch
12:30 – 1:00 Item rewrites
1:00 – 2:30 Write items
2:30 – 3:30 Vet items in like groups
*We have included slides in this presentation for you to use with your teachers, but we will not go in great detail on some of the slides with you.
ITEM WRITING GUIDELINES
3
ANATOMY OF AN ITEM
4
Use the excerpt below to answer the question. (Direction Line)
“We hold these truths to be self-evident, that all men are created equal, that they are endowed by their Creator with certain unalienable Rights, that among these are Life, Liberty and the pursuit of Happiness.”(Stimulus – prompt/background)
Which document includes these words? (Stem - question)
A.The Magna Carta
B.The Mayflower Compact (Distractors)
C.The Articles of Confederation
D.The Declaration of Independence* (Correct Answer)
Cognitive Complexity, Item Difficulty, Sensitivity, Bias, Copyright
TEST ITEM WRITING PROCESSOVERVIEW – FOR MULTIPLE CHOICE
5
1. Look at the standard and item specifications
2. Develop a stimulus (if required)
3. Write the question (item stem)
4. Write the correct answer
5. Write distractors that are reasonable but clearly incorrect
6. Identify the cognitive complexity level for your item
7. Check the item for any flaws in logic, construction, or grammar and mechanics, bias, sensitivity, etc.
COGNITIVE COMPLEXITY
6
UNDERSTANDING COGNITIVE COMPLEXITY
• Florida measures the cognitive demand of the task that students are asked to perform by using Norman Webb’s Depth of Knowledge (DOK) levels
• Items that you develop will fall into one of the four DOK levels:
- Level 1: Recall
- Level 2: Basic Application of Concepts & Skills
- Level 3: Strategic Thinking & Complex Reasoning
- Level 4: Extended Thinking & Complex Reasoning
Reference material is available as a supplemental resource.
*Florida combines Level 3 and 4 and should be indicated as Level 3
8
EXAMPLES OF COMPLEXITY LEVEL QUESTIONS FOR U.S. HISTORY
9
MULTIPLE CHOICE QUESTIONS FOR U.S. HISTORY- LOW COGNITIVE COMPLEXITY
10
Which social issue is addressed in the cartoon?A. Industrial safetyB. The training of teachersC. Compulsory public educationD. Organization of immigrant laborers
MULTIPLE CHOICE QUESTIONS FOR U.S. HISTORY-MODERATE COGNITIVE COMPLEXITY
11
Which problem was the state action shown in the cartoon meant to resolve?
A. The influence of labor unionsB. The employment of child laborersC. The low literacy rates for immigrantsD. The poor working conditions in industry
EOC MULTIPLE CHOICE QUESTIONS FOR U.S. HISTORY- HIGH COGNITIVE COMPLEXITY
12
Based on the cartoon, what conclusion can be made about the US in the early 1900s?
A. The government assumed an active role in suppressing social activists.
B. The government assumed an active role in legislating social reforms.
C. Business monopolies controlled the means of production.D. Business owners monitored the activities of laborers.
ITEM WRITING TRAINING
13
UNIFY ITEM TYPES
•Multiple Choice
•Mutliselect
•GRID with
Action Buttons
•Hot Text
•Open Response
•Matching
•& more…
ITEM TYPES
15
Selected response (multiple choice)
Selected Response item (multiple choice):
•Should take the student approx. one minute per item to answer
•Has a question with four answer choices – three distractors and one correct answer
•The stem and answer choices can include text or images
WHAT IS A STIMULUS?
The stimulus is what prompts the stem (or question).
Stimuli can be:
• Text
• Charts and graphs
• Diagrams
• Illustrations
• Photographs
• Maps
• Numeric equations
• Tables and charts
• Timelines
• Graphic organizers
Not every question will have a stimulus. (Stem & answer choices only.)
WHAT IS THE STEM?
17
• The stem is the question being asked and the foundation of the
item.
• After reading the stem, the student should know exactly what the
problem is and what he or she is expected to do to solve it.
• If the student has to infer what the problem is, the item will likely
measure the student’s ability to draw inferences from vague
descriptions rather than his or her achievement of a course
objective.
ACTIVITY
18
Please complete the
activity and then you may
take a 5 minute break
THE STEM: CLEAR AND CONCISE
19
The problem should be stated clearly in the stem.
THE STEM: CLEAR AND CONCISE
20
From the article, the reader can tell that:A.Turtles like to hide under the rocks
B.Dogs like to play in the snow
C.Squirrels like to eat mulberries
D.Cats like to chase mice
From the article, what can the reader tell about what that turtles like to do?
A. Hide under the rocksB. Play in the snowC. Eat mulberriesD. Chase mice
Poor Example
Better Example
Does this stem provide the test taker with enough information?
THE STEM: RELEVANT MATERIAL
21
Include relevant material; avoid irrelevant material.
The local community theater in Conroe, Texas is performing a play in the Conroe High School auditorium. There are 15 rows of seats. Each row contains 28 seats.
What is the largest number of tickets that the community theater can sell to fill the auditorium for one performance?
A. 400 tickets
B. 420 tickets*
C. 450 tickets
D. 470 tickets
Answer – The stem is too lengthy, contains too much unnecessary information, and has no relation to Florida. The excessive information may confuse students as to what is actually being asked.
THE STEM: AVOID REDUNDANCY
22
Rather than repeating redundant words or phrases in the
distractors, place such material in the stem.
STEM: CLUEING
23
Lisa is writing a report on different types of ecosystems. Which source would most likely contain this information?
A. Cooking with Native Plants
B. A Guide to Rock Collecting
C. Ecosystems of North America*
D. The Big Book of Animal Habits
Key words in the stem that are also found in the answer choices may be clues to the correct answer.
This measures students cleverness, not knowledge of content.
24
THE STEM: NEGATIVE
Student achievement is more effectively measured by having students identify correct rather than an incorrect answer. Words like “is not”, “except”, “least”, etc. should be used sparingly.
25
WHAT IS A DISTRACTOR?
A distractor is an answer choice that leads a student away from the correct answer.
A good distractor has the following characteristics:
• Correct grammar
• Professional tone (avoid humor or slang)
• Unique scenario (one size does not fit all items)
• Direct and concise
• Test-savvy students easily eliminate implausible choices and improve their chances of guessing the correct answer.
• If a writer cannot develop three plausible distractors, then the question itself may be flawed.
26
STEM AND DISTRACTORS: VERBATIM
Avoid stems and distractors that are word-for-word from the pages of the textbook. This measures student’s memory rather than understanding.
27
DISTRACTORS: CLEAR AND CONCISE
Clear wording
reduces student
confusion, and
concise wording
reduces the
reading burden
placed on the
student.
28
DISTRACTORS: OVERLAPPING
If the overlap
includes the correct
answer, there may
be confusion about
which is the correct
answer and which is
the distractor.
29
DISTRACTORS: HOMOGENEOUS
Keep the
distractors
homogeneous
in content.
30
DISTRACTORS: GRAMMAR
Keep the grammar of each distractor consistent with
the stem.
31
DISTRACTORS: PARALLEL FORM
The stem and distractors should be worded similarly.
In this food chain, what will happen if there are fewer crickets in one year?
A.There will be more hawks
B. There will be fewer cattails
C.There will be fewer shrews*
D.The shrews will eat cattail
MORE ABOUT PARALLELISM
32
If the correct answer contains:Then the distractors should
NOT contain:
A long phrase or sentence Short phrases or sentences
A general rule Specific instances
A complete sentence A phrase
A means to an end End products
Simple, common language Professional jargon
Unqualified words or statements Qualifying words
Negative descriptions Positive descriptions
33
DISTRACTORS: FLAGS
Avoid words such as never, always, only, etc.; they serve as flags to alert students.
34
DISTRACTORS: PLAUSIBLE
Avoid unrealistic or humorous distractors. They are nonfunctional and increase student’s chance of guessing the correct answer.
35
DISTRACTORS: SINGLE CORRECT ANSWER
Avoid “all of the above”, “none of the above”, “both a and b”, etc. (multiple responses or combined responses)
36
TEST ITEM: CONVENTIONS
• Use proper grammar, spelling and punctuation.• Avoid using unnecessarily difficult vocabulary.• Use grade appropriate language – all items should
be written to the lower grade band for the standard.• Ex. LACC.1112.W.1.1 should be written to an 11th
grade level.• Ex. MACC.912.MP.5.1 should be written to a 9th grade
level.
TEST ITEM: SENSITIVE TOPICS
37
Be mindful of how you approach sensitive topics, such as:
• Alcoholic beverages, drugs, or tobacco
• Death and/or funerals
• Gambling
• Name brands and trademarked products
• Occult (witchcraft, etc.)
• Parapsychology
• Politics
• Religion
There may be exceptions to this rule (such as history courses).
TEST ITEM: BIAS
38
Read the following statement:
Firemen are important to the community.
They keep all of us safe.
• Which word contains bias?
• How can it be changed to eliminate the bias?
Answer: Change firemen to firefighters
CHECK ITEMS FOR BIAS
39
To ensure that test items are unbiased, include:‒Precisely defined ideas‒Language appropriateness
But avoid:‒Sensitive topics‒Gender stereotypes‒Ethnic stereotypes‒Socioeconomic bias‒Geographic bias
COPYRIGHT ISSUES
40
To avoid copyright issues:• Go to the original source and provide full source information
• Use public domain sites, such as government sites (e.g., EPA, US Census Bureau, or Library of Congress) and check for copyright information
• Avoid using permissioned work
Avoid using copyrighted names/brand names, such as:• Kleenex• Miami Dolphins• Adidas• Jell-O• Q-Tip
WEBSITES FOR ITEM INSPIRATION
41
• http://achievethecore.org/
• http://www.problem-attic.com/
• http://www.engageny.org/resource/regents-
exams-mathematics
• Florida Achieves Grade 5 and Bio (Shared by
Heidi in Google Drive)
42
QUESTIONS??
FSA MATHEMATICS ITEM TYPES
• Multiple Choice
• Mutliselect
• Equation
• GRID with Action
Buttons
• Hot Text
• Open Response
• Matching
• Table
• Editing Task
Choice
MULTIPLE CHOICE
Multiple Choice items require the student to
select a single answer option. (worth 1
point)Multiple Choice Items consist
of:
A context (mathematical or
real-world);
A question;
A single correct answer;
and
Three plausible distractors.
MULTISELECT ITEMS
Multi-select items allow the student to select more than one answer option.
Some items may ask the student to select a specific number of responses,
while others will ask them to select all correct responses. No partial credit
will be given. (worth 1 point)
Multiselect Items consist of:
A context (mathematical or
real-world);
A question;
Two or more correct
answers; and
Several plausible distractors.
Select all the statements that correctly
compare the two numbers.
Click the
checkbox next
to each option
you want to
select as a
response. You
may select
more than one
option.
EQUATION ITEMS
Equation items require the student to enter a valid statement that
answers the question. The response may be a number, an
expression, or an equation. No partial credit will be given. (worth 1
point)
Equation Items consist of:
A context (mathematical or
real-world);
A question; and
A response area.
Equivalent fractions and decimals will
be accepted except in items that are
assessing a standard where a certain
form is the expectation of the standard.
A zero in the ones place for answers
that fall between -1 and 1 (excluding 0
itself) is not required.
Students can type in an answer using a
mixed number or a fraction greater than
one.
EQUATION ITEM
GRID ITEM WITH ACTION BUTTONS
GRID items require the student to use the point, line, and/or
arrow buttons to create a response on a grid. No partial credit
will be given. (worth 1 point)
GRID Items consist
of:
A context
(mathematical or
real-world);
A question; and
A response area.
HOT TEXT ITEMS
Hot text items require the student to either click on a
response option or drag a response option to another
location.
Hot text Items
consist of:
A context
(mathematical or
real-world);
A question; and
A response
area.
In the ordered pair (0,4), what
does the 4 represent in terms of
its location on the coordinate
plane? Type your answer in the
space provided.
OPEN RESPONSE ITEMS
Open response items require the student use the
keyboard to enter a response into a text field.
Open Response Items
consist of:
A context (mathematical
or real-world);
A question; and
A response area.Click on the text box
and type your answer
using the keyboard.
Select the value of each decimal
number when it is rounded to the
nearest whole number.
MATCHING ITEMS
Matching items require the student to check a box to indicate if
information from a column header matches information from a row. No
partial credit will be given. (worth 1 point)
Matching Items consist of:
A context (mathematical
or real-world);
A question; and
A response area.
Click in the
boxes to
highlight the
desired
options.
The table shows the height of two containers, in feet..
Complete the table to show the height of each container, in
inches..
TABLE ITEMS
Table items require the student to type numeric values into a
given table. The student may complete the entire table or
portions of the table depending on the question. No partial
credit will be given. (worth 1 point)
Table Items consist
of:
A context
(mathematical or
real-world);
A question; and
A response area.
EDITING TASK CHOICE ITEMS
Editing task items require the student to click on the highlighted word
or phrase and select the replacement word or phrase from a menu. No
partial credit will be given. (worth 1 point)
Table Items consist
of:
A context
(mathematical or
real-world);
A question; and
A response area.
COMBINATION ITEMS
Items could combine more than one item type. These
may be worth more than 1 point, in which case, points
may be awarded by type.
Related Documents
