User’s Manual for Iteman Classical Item and Test Analysis Version 4.3 June 2013
User’s Manual for
Iteman
Classical Item and Test Analysis
Version 4.3
June 2013
Contact Information Assessment Systems Corporation
6053 Hudson Road, Suite 345
Woodbury, MN. 55125
Voice: (651) 647-9220
Fax: (651) 647-0412
www.assess.com
Bookmarks To view PDF Bookmarks for this manual, select the Bookmark tab on the left side of the Acrobat
window. The bookmark entries are hyperlinks that will take you directly to any section of the
manual that you select.
License Unless you have purchased multiple licenses for Iteman 4.3, your license is a single-user license.
Instructions for transferring your license between computers are in Appendix E.
Technical Assistance and support If you need technical assistance using Iteman 4.3, please visit the Support section of our Web site,
http://www.assess.com/xcart/pages.php?pageid=8, which contains FAQs and updated versions of
our software. Technical assistance for Iteman 4.3 is provided for as long as you maintain the
current version. Please provide us with the invoice number for your license purchase (typically 5
digits, like #22111) when you request technical assistance.
Citation Guyer, R., & Thompson, N.A., (2013). User’s Manual for Iteman 4.3. Woodbury, MN:
Assessment Systems Corporation.
No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any
form or by any means—electronic, mechanical, photocopying, recording, or otherwise—without
the prior written consent of the publisher.
Copyright © 2013 by Assessment Systems Corporation
All Rights Reserved
Iteman is the trademark of Assessment Systems Corporation
Table of Contents
1. Introduction ...................................................................................................................... 1
Your Iteman 4 License and Unlocking Your Copy ............................................................ 1
2. Input Files ......................................................................................................................... 4
The Data Matrix File........................................................................................................... 4
Iteman 3 Data Format ......................................................................................................... 4
Delimited Data Matrix File ................................................................................................. 5
The Item Control File ......................................................................................................... 5
3. Running the Program ...................................................................................................... 7
Using the Menus ................................................................................................................. 7
The File Menu ................................................................................................................. 7
Using the Main Program ..................................................................................................... 8
The Files Tab .................................................................................................................. 8
The Input Format Tab ..................................................................................................... 9
The Scoring Options Tab .............................................................................................. 11
The Output Options Tab ............................................................................................... 13
Using Multiple Runs Files ................................................................................................ 15
Creating a Multiple Runs File ....................................................................................... 15
Opening a Multiple Runs File ....................................................................................... 17
A Sample MRF File ...................................................................................................... 17
Running the sample files................................................................................................... 19
4. Interpreting the Output ................................................................................................. 22
Test-Level Output (Examinee Scores) .............................................................................. 22
Test-Level Output (Reliability Analysis) ......................................................................... 23
Test-Level Output (Graphics) ........................................................................................... 24
Conditional Standard Error of Measurement (CSEM) ..................................................... 24
Item-Level Output............................................................................................................. 25
What to Look For .............................................................................................................. 29
Item Difficulty .............................................................................................................. 29
The P value (Multiple Choice) ........................................................................... 29
The Item Mean (Polytomous) ............................................................................ 30
Item Correlations .......................................................................................................... 30
Multiple Choice Items ........................................................................................ 30
Polytomous Items ............................................................................................... 30
DIF Statistics ................................................................................................................. 31
Mantel-Haenszel ................................................................................................ 31
z-test Statistic ..................................................................................................... 31
p .......................................................................................................................... 32
Bias Against ....................................................................................................... 32
Option statistics ............................................................................................................. 32
Scores Output File ............................................................................................................ 32
Collusion Index (Bellezza & Bellezza, 1989) .................................................................. 33
References ........................................................................................................................... 34
Appendix A: The Iteman 3 Header ................................................................................... 35
The Control Line ............................................................................................................... 35
The Keyed Responses ....................................................................................................... 36
The Number of Alternatives ............................................................................................. 36
The Scale Inclusion Codes ................................................................................................ 36
Appendix B: Troubleshooting ........................................................................................... 37
Please check the data file format specifications ............................................................... 37
Please check the number of items or number of ID columns specified in the Iteman 3
Header ............................................................................................................................... 37
Please select an input file with an Iteman 3 Header ......................................................... 38
Valid item responses of 0 were identified. The Iteman 3 Header does not support item
responses that begin at 0. .................................................................................................. 38
At least one valid item response of 0 was identified. ....................................................... 39
At least one unidentified response character was identified and will be scored as
incorrect. ........................................................................................................................... 39
Check the data matrix file, examinee XXX did not respond to all XXX items................ 40
Appendix C: Formulas ....................................................................................................... 41
Conditional Standard Error of Measurement Formulas .................................................... 41
Livingston Classification Consistency Index ................................................................... 41
Differential Item Functioning ........................................................................................... 41
Item Collusion Index ........................................................................................................ 42
Appendix D: Program Defaults File ................................................................................. 43
Appendix E: License Transfer .......................................................................................... 46
Summary ........................................................................................................................... 46
Step 1 – Demo/Trial Program ........................................................................................... 46
Step 2 – Licensed Program ............................................................................................... 47
Step 3 – Demo/Trial Program ........................................................................................... 49
Iteman 4.3 Manual Page 1
1. Introduction
Iteman™ is a Windows® application designed to provide detailed item and test analysis reports
using classical test theory (CTT). The purpose of these reports is to help testing programs
evaluate the quality of test items by examining their psychometric characteristics.
Iteman has a friendly graphical user interface (GUI) that makes it easy to run the program, even
if you are not familiar with psychometrics. The GUI is organized into five tabs: Settings, Files,
Input Format, Scoring Options, and Output Options. These are discussed in detail in Chapter 3:
Running the Program.
Iteman 4.3 offers several substantial advantages over Iteman 3:
1. The most important advantage is the addition of graphics. It is now possible to produce
an item quantile plot for each item. Moreover, you control the number of points in the
plot.
2. Iteman 4 is able to handle pretest (trial or unscored) items—items that are not included in
the final score but for which statistics are still desired.
3. More statistics are calculated, including the alpha (KR-20) reliability coefficient with
each item deleted, several split-half reliability coefficients (both with and without
Spearman-Brown correction), conditional standard error of measurement, and subgroup P
(proportion correct) statistics for up to seven ordered groups.
4. Instead of simple ASCII text files, the output is now rich text file (RTF) format prepared
as a formal report, and also in a comma-separated value (CSV) format that is able to be
manipulated (sorted, highlighted, etc.) in spreadsheet software. It additionally produces a
CSV file of examinee scores.
5. Scaled scores and subscores can be added to the output.
6. Scores can be classified into two groups at a specified cut score, and the two groups can
use your labels.
7. Items can be analyzed relative to an external score rather than the total score on a test.
8. The maximum number of items that can be analyzed has been increased to 10,000.
9. A “batch” type of capability, using a “Multiple Runs File” has, been added to allow you
to run multiple data sets without having to use the graphic user interface for each run.
Multiple Runs files can be created outside Iteman in a text editor or interactively within
Iteman.
Your Iteman 4 License and Unlocking Your Copy Unless you have purchased a network or multiple-computer license, your license for Iteman 4 is
a single-user license. Under this license you may install Iteman 4 on two computers (e.g., a
Iteman 4.3 Manual Page 2
desktop and a laptop) so long as there is no possibility that the two copies of the software will be
in use simultaneously. If you would like to use Iteman 4 on a network or by more than one user,
please contact us to arrange for the appropriate number of additional licenses.
Iteman 4 is shipped as a functionally-limited demonstration copy. It is limited to no more than 50
items and 50 examinees, but has no expiration date. We can permanently convert your demo
copy to the fully functioning software by email, phone, or fax once you have completed the
license purchase. To unlock Iteman 4, please email/phone/fax to ASC:
1. Your name and email address.
2. Your organization or affiliation.
3. Your invoice number (in the top right corner of your invoice). You should make a
record of your invoice number since you might be asked for it if you request technical
support.
4. The “unlock codes,” which are two numeric codes that are unique to the installation of
Iteman 4 on any given computer. To obtain these two codes, click on the “Unlock
Program” button when Iteman 4 starts (Figure 1.1) This license window can also be
reached by clicking on the License button and selecting “Unlock” when Iteman 4 is
running in demo mode.
Figure 1.1: Screen Visible When Iteman 4 is Locked
From the unlock screen you will need to send us the two blue Computer ID and Session ID
numbers (Figure 1.2). For your convenience we have provided a “Copy IDs to Clipboard”
button. This will copy both IDs to the Windows clipboard along with a brief message and the
email address to which to send your payment information. This can then be pasted into an email
message, filled in, and sent to [email protected]. If you have already paid for your Iteman 4
license, be sure to add your invoice number to this message.
When we receive these codes from you, we will respond with a single numeric Activation Code
(if you have purchased a permanent license) or two codes (if you have purchased an annual
subscription license) that you will need to enter into this same window from which you obtained
Iteman 4.3 Manual Page 3
your Activation Codes (the red labels in Figure 1.2). Once you enter the code(s) that we send
you, your copy will be unlocked and fully functional.
Figure 1.2: The Unlock Screen
Note that if you install Iteman 4 on a second computer, you will need to repeat this process for
that computer since the unlock codes are specific to a given computer.
• Iteman 4 is permanently unlocked for academic use, but is an annual subscription for
non-academic use. The license status box (see Figure 3.1) will display the current
license status, including the number of days remaining for your subscription. As the
subscription nears the end, the background color of the box will change to alert you to the
need to renew your subscription for another year (red if you have less than 30 days
remaining, yellow if 30-90 days, and green if more than 90 days).
.
Iteman 4.3 Manual Page 4
2. Input Files
Iteman 4 requires two input files: the Data Matrix File and an Item Control File. The formats for
these files are described in the following sections.
The Data Matrix File The Data Matrix File is the file that contains examinee identification (ID) and the responses to
each item. Responses can be alphabetical (A,B,C,D… or a,b,c,d…) or numerical (1,2,3,4…),
where A = a = 1, etc. Each line presents the information for one person. An example of this is
shown in Figure 2.1 for 10 items and 5 examinees. In this file, there are 9 columns of ID (the
last two are blank) and 10 columns of responses.
Figure 2.1: Example of an Input Data File (No Ignored Columns)
Additional columns can be ignored, so it is not necessary to delete any data if your data file has
information other than ID and responses. For example, your file might contain exam dates,
locations, education level, or sensitive personal data that you do not want included in the output.
In the example shown in Figure 2.2, group membership information is included in column 19.
An example of this is shown in Figure 2.2; you might want to include examinee ID numbers (the
first six columns) in your output but not names. Chapter 3: Running the Program describes
how to skip these columns.
Figure 2.2: Example of an Input Data File (Columns to Ignore)
Iteman 3 Data Format
Iteman 4 permits the analysis of a Data Matrix File in the format used with Iteman 3 (and other
programs in the Item and Test Analysis package) , which includes four header lines of control
information in the data file rather than in a separate control file. If the Iteman 3 header is
included in the Data Matrix File, then the user should specify this with the checkbox on the
“Files” tab of Iteman 4. Note that this method does not allow the use of Item IDs. See
Appendix A for a description of the Iteman 3 header.
Person1 4213323412
Person2 1213323410
Person3 3323123413
Person4 1223323414
Person5 2214323411
6153425 John Doe M 4213323412
5947824 Jane Doe F 1213323410
5976281 Jack Hall M 3323123413
1359687 Jim Hill M 1223323414
9778236 Jen Smith F 2214323411
Iteman 4.3 Manual Page 5
Delimited Data Matrix File Iteman 4.3 now permits the use of a data matrix file that is delimited by either a comma or a tab.
As shown in Figure 2.3, this now permits the inclusion of variable length examinee IDs.
Chapter 3: Running the Program describes how to specify a delimited input file.
Figure 2.3: Example of a Comma-Delimited Data Matrix File
If a differential item functioning (DIF) analysis was requested the DIF group membership code
should follow the examinee ID as shown by Figure 2.4. It is important to note that the DIF
membership codes (M and F) will not be recognized if they are included as part of the examinee
ID (e.g., Person9M).
Figure 2.4: Example of a Comma-Delimited Data Matrix with DIF Codes
The Item Control File The previous version of Iteman required that the specifications for the test be provided on the
first four lines of the data file, with all the data itself moved down. Iteman 4 provides the
specifications as a separate Item Control File. This makes it easier to produce, as well as allows
for the handling of a greater amount of information. This file is tab-delimited, which means that
you can construct it in a spreadsheet program and then “Save As” a tab-delimited text file.
There are six columns of information in the control file for each item. Begin each item on a new
line:
1. Item ID: cannot contain a tab character, but length can vary across items;
2. Key(s):
• Correct answer(s) as A,B,C,D if unscored multiple-choice.
• 1 if items are scored dichotomous.
• For polytomous items: + if positively scored or – if negatively or reverse scored.
3. Number of alternatives (maximum is 15). For multiple-choice items that are scored 0
or 1, the number of options is 2.;
4. Domain or content area (unique domain label, maximum is 50);
5. Inclusion status:
• Y = Yes (included in the analysis),
Person9,4,2,1,3,3,2,3,4,1,2
Person10,1,2,1,3,3,2,3,4,1,0
Person11,3,3,2,3,1,2,3,4,1,3
Person12,1,2,2,3,3,2,3,4,1,4
Person13,2,2,1,4,3,2,3,4,1,1
Person9,M,4,2,1,3,3,2,3,4,1,2
Person10,F,1,2,1,3,3,2,3,4,1,0
Person11,M,3,3,2,3,1,2,3,4,1,3
Person12,F,1,2,2,3,3,2,3,4,1,4
Person13,F,2,2,1,4,3,2,3,4,1,1
Iteman 4.3 Manual Page 6
• N = No (not included),
• P = Pretest;
6. Item type:
• M = Unscored multiple-choice items with responses that begin at 1 or A. For
scored multiple-choice items, see P below.
• R = Rating scale items: polytomous items with responses that begin at 1 or A).
• P = Items with numeric responses that begin at 0 (e.g., 0, 1, 2, 3). This includes
multiple-category partial credit items, and dichotomously scored multiple-choice
items (scored 0 or 1);
An example of the control file is shown in Figure 2.5. There are ten items, with nine multiple
choice items and one partial credit item. The first five are in Domain 1, while the latter five are
in Domain 2. The first four items in each domain are scored, while the fifth item in domain 2 is
a pretest item. The keyed answers are either 1, 2, 3, or 4 for each multiple choice item since each
item has 4 alternatives. Keys can be alphabetical or numeric. Item 7 has two keyed responses (3
and 1). For Item 7, item responses will be scored as correct if the examinee answers either 3 or
1.
If an item is polytomously scored, the key should be “+” if positively scored and “-” if negatively
(reverse) scored. Item 10 is a positively scored (+) partial credit item with item responses that
begin at 0. For item 10, the item responses will be 0, 1, 2, 3, and 4, since the item has five
options.
The control file should have as many lines as there are items in the test. The program counts the
lines of information in the control file, and that serves as the total number of items in the test.
There is a maximum of 10,000 items (lines) in Iteman 4.3.
Figure 2.5: Example of an Item Control File
Item01 1 4 Science Y M
Item02 2 4 Science Y M
Item03 3 4 Science Y M
Item04 4 4 Science Y M
Item05 1 4 Science P M
Item06 2 4 Reading Y M
Item07 31 4 Reading Y M
Item08 4 4 Reading Y M
Item09 1 4 Reading Y M
Item10 + 5 Reading P P