Using the Many-Faceted Rasch Model to Evaluate Standard Setting Judgments: IllustrationWith the Advanced Placement Environmental Science Exam Pamela K. Kaliski, Stefanie A. Wind, George Engelhard Jr., Deanna L.Morgan, Barbara S. Plake, and Rosemary. Reshetar psychology and applied statistics ://ncaase.com/about/bio?id=4 http://des.emory.edu/home/ people/faculty/Engelhard.html
22
Embed
Pamela K. Kaliski, Stefanie A. Wind, George Engelhard Jr., Deanna L.Morgan,
Using the Many-Faceted Rasch Model to Evaluate Standard Setting Judgments: An IllustrationWith the Advanced Placement Environmental Science Exam. http://des.emory.edu/home/people/faculty/Engelhard.html. Pamela K. Kaliski, Stefanie A. Wind, George Engelhard Jr., Deanna L.Morgan, - PowerPoint PPT Presentation
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Using the Many-FacetedRasch Model to Evaluate
Standard Setting Judgments:An IllustrationWith the Advanced Placement
Standard setting ‘‘. . . standard setting refers to the process of establishing one or
more cut scores on examinations. The cut scores divide the distribution of examinees’ test performances in two or more categories’’ Cizek and Bunch (2007)
criteria The criteria for evaluating panelist judgments Procedural validity : implementation issues and
documentation Internal validity : interpanelist and intrapanelist consistency External validity : comparisons with other methods
33
Many-Faceted Rasch Model
n: panelist k: a standard setting modified Angoff rating i :item ; j : round n is the judged severity for panelist n, i is the average judged item difficulty for item i, j is the judged average performance level for round j jk is the cut score, or threshold coefficient, from round j for
standard setting ratings of k. ( rating k relative to k − 1)
separation statistics/chi-square statistic(b) model–data fit : Outfit MSE
(c) the creation of a visual display for comparing panelist judgments on the latent variable
55
Multiple Yes-No (MYN) method
MYN requires panelists to consider the borderline examinee at each cut score and to identify at which level the borderline examinee would be able to answer each item correctly.
panelists considered each item and decided whether or not the borderline examinee in
each category would be able to identify the correct answer
66
PLDs
77
Would a borderline-1/2 student be able to answer this item correctly? If yes, then the panelist would circle the 1/2 cut score on the rating form and move on to the next item.
If no, then the panelists would consider the next question about the same item: Would a borderline-2/3 student be able to answer this item correctly?
If yes, the 2/3 cut score would be circled for that item and the panelist would move on to the next item. If no, the panelist would consider the next question about the same item: Would a borderline-3/4 student be able to answer this item correctly?
If yes, the 3/4 cut score would be circled for that item and the panelist would move on to the next item.
If no, then the panelists would consider the next question about the same item: Would a borderline-4/5 student be able to answer this item correctly? If yes, the 4/5 cut score would be circled for that item and the panelist would move on to the next item. If no, then the panelist would consider the final question about the same item: Would the above borderline-5 student be able to answer this item correctly?
If yes (which is likely given that all other possible borderline students have
been considered), then the Above 5 score would be circled for that item.
88
99
instuments
The Advanced Placement (AP) program (Advanced Placement Environmental Science (APES) examination) is composed of 34 courses and corresponding examinations in 22 subject areas.
Data used in this study come from the 2011 administration of the APES exam and the standard setting for this examination.
100 MC items and four CR itemsData used in this study are the ratings that resulted
from two rounds of item-level judgments provided by the 15 APES panelists
1010
Research Purpose
the MFR model is used to evaluate the quality of judgments on MC items provided by panelists who participated in a modified Angoff standard setting that used the MYN method for MC items, the 2011 APES exam.
panelist characteristics (gender and level of teaching) are incorporated into the MFR model to determine whether or not these are explanatory variables that account for differences in panelist ratings