This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
• NAEP (“The Nation’s Report Card) provides a NAEP (“The Nation’s Report Card) provides a snapshot of snapshot of student achievement overall, by student achievement overall, by state and by various state and by various subgroupssubgroups
• Increasing reliance on low-stakes large-scale assessment surveys Increasing reliance on low-stakes large-scale assessment surveys for education policyfor education policy
• Issues relate to both national and international LSASsIssues relate to both national and international LSASs
• In the U.S., NAEP is the only source of nationally comparable data In the U.S., NAEP is the only source of nationally comparable data on student achievement that can be used for state-level on student achievement that can be used for state-level comparisonscomparisons
• Under NCLB, 4Under NCLB, 4thth and 8 and 8thth grade NAEP play an expanded role in grade NAEP play an expanded role in monitoring state-level resultsmonitoring state-level results
• Strong interest in expanding role of 12Strong interest in expanding role of 12thth grade NAEP grade NAEP
• National Commission on 12National Commission on 12thth grade NAEP (2004) grade NAEP (2004) recommendationsrecommendations– Redesign to report on student readinessRedesign to report on student readiness– Expand to state levelExpand to state level– Increase participation and motivationIncrease participation and motivation
Goal:Goal: To estimate the effects of different To estimate the effects of different monetary incentives monetary incentives on student on student performance on 12performance on 12thth grade NAEP grade NAEP
• ExperimentsExperiments– Focus on mathematicsFocus on mathematics– O’Neil et al (NAEP items)O’Neil et al (NAEP items)– Baumert et al (PISA items)Baumert et al (PISA items)
• PsychologyPsychology– Intrinsic vs. extrinsic motivationIntrinsic vs. extrinsic motivation
• Behavioral EconomicsBehavioral Economics– Monetary incentives can workMonetary incentives can work– Participants must be cognizant of incentivesParticipants must be cognizant of incentives
• External validity (i.e., link directly to NAEP)External validity (i.e., link directly to NAEP)– Used released NAEP materialsUsed released NAEP materials– Followed NAEP administrative and data processing Followed NAEP administrative and data processing
proceduresprocedures– Carried out NAEP-like psychometric and statistical Carried out NAEP-like psychometric and statistical
analysesanalyses• Heterogeneous school sampleHeterogeneous school sample• Large study for sufficient power to detect effectsLarge study for sufficient power to detect effects
• Control: Standard NAEP instructionsControl: Standard NAEP instructions
• Incentive 1: Standard NAEP instructions + Incentive 1: Standard NAEP instructions + Promise of a $20 gift card at conclusion of Promise of a $20 gift card at conclusion of sessionsession
• Incentive 2: Standard NAEP instructions + $5 Incentive 2: Standard NAEP instructions + $5 Gift card + $15 for a correct answer to each of Gift card + $15 for a correct answer to each of two questions to be chosen at random at the two questions to be chosen at random at the conclusion of sessionconclusion of session
Study Design: Incentives (2)Study Design: Incentives (2)
• All students in both incentive conditions were All students in both incentive conditions were asked to select Target or Barnes & Noble for the asked to select Target or Barnes & Noble for the gift card and to indicate their preference on a gift card and to indicate their preference on a sign-up sheetsign-up sheet
• Students in all three conditions actually received Students in all three conditions actually received $35 gift cards at the end of the sessions$35 gift cards at the end of the sessions
• Students were informally debriefed before Students were informally debriefed before leavingleaving
Study Design: InstrumentationStudy Design: Instrumentation
• Mapping to the NAEP Reading Framework (3 Mapping to the NAEP Reading Framework (3 contexts)contexts)– * * Reading for literary experience (35%)Reading for literary experience (35%)– * Reading for information (45%)* Reading for information (45%)– Reading to perform a task (20%)Reading to perform a task (20%)
• Assembling test bookletsAssembling test booklets– 2 reading blocks + background questionnaire2 reading blocks + background questionnaire– Each reading block consists of a passage and a set Each reading block consists of a passage and a set
of associated questionsof associated questions– Each block is expected to take 25 minutesEach block is expected to take 25 minutes– Blocks vary with respect to the total number of Blocks vary with respect to the total number of
questions and the proportions of multiple choice, questions and the proportions of multiple choice, short answer and extended response questionsshort answer and extended response questions
Study Design: Sample SelectionStudy Design: Sample Selection
• Power analysis indicated need for a sample of 60 Power analysis indicated need for a sample of 60 schools with 60 students per school (20 per schools with 60 students per school (20 per condition in each school)condition in each school)
• Worked with NAEP state coordinators and Westat Worked with NAEP state coordinators and Westat to obtain a (final) convenience sample of 59 schoolsto obtain a (final) convenience sample of 59 schools
• Student recruitment was carried out using Student recruitment was carried out using standard NAEP methods (but no special incentives)standard NAEP methods (but no special incentives)
• Number of participating students was lower than Number of participating students was lower than targettarget
• Random samples of 12Random samples of 12thth graders invited to graders invited to participateparticipate
• In each school students randomly allocated In each school students randomly allocated to the three to the three conditionsconditions
• Fall (not spring) administrationFall (not spring) administration• Sessions in a school were simultaneous or Sessions in a school were simultaneous or
consecutive to consecutive to eliminate possibility of eliminate possibility of contaminationcontamination
• Limited accommodationsLimited accommodations• No make-up sessionsNo make-up sessions
Data Preparation: Data Preparation: Scoring and Item AnalysisScoring and Item Analysis
• Scoring was conducted by NCS/PearsonScoring was conducted by NCS/Pearson• Preliminary item analysis held no surprises: Preliminary item analysis held no surprises:
Differences by condition inDifferences by condition in– Proportions correct Proportions correct – Percentage of omitted itemsPercentage of omitted items
•Highest for extended CR items Highest for extended CR items – Percentage of off-task responsesPercentage of off-task responses
•Generally very small (<< 1%)Generally very small (<< 1%)– Percentage of items not reachedPercentage of items not reached
•Particularly high for last CR itemParticularly high for last CR item
Data Preparation: Data Preparation: Scaling, Conditioning and LinkingScaling, Conditioning and Linking
• Scaling by subscaleScaling by subscale– Fit item characteristic curves to dataFit item characteristic curves to data– Compare to archival resultsCompare to archival results– Estimate three-group modelEstimate three-group model– Reasonable fitReasonable fit
• ConditioningConditioning– Combine cognitive data with ancillary data from Combine cognitive data with ancillary data from
questionnairesquestionnaires– Obtain posterior score distribution for each Obtain posterior score distribution for each
• LinkingLinking– Linear transformation to the NAEP scaleLinear transformation to the NAEP scale– Construct composite reporting scaleConstruct composite reporting scale
• White-Black and White-Hispanic differences White-Black and White-Hispanic differences grow grow somewhat larger under incentivessomewhat larger under incentives
• Effects of incentives generally positive for Effects of incentives generally positive for subgroupssubgroups
N 155 122 98 COL Percent 19.0 14.1 11.1 Not very important MEAN 287.0 292.4 288.7 N 369 348 345 COL Percent 45.3 40.3 39.2 Somewhat important MEAN 291.6 293.0 296.3 N 205 286 294 COL Percent 25.2 33.1 33.4 Important MEAN 289.9 294.2 297.7 N 86 108 144 COL Percent 10.6 12.5 16.4 Very important MEAN 282.6 291.2 295.1
• Although treatment groups were determined Although treatment groups were determined randomly, there were differences in various randomly, there were differences in various characteristics that characteristics that might have contributed might have contributed to the estimated treatment effects.to the estimated treatment effects.
• We ran an ANOVA adjusting NAEP scores for We ran an ANOVA adjusting NAEP scores for a number of demographic and home a number of demographic and home environment characteristics, as well as environment characteristics, as well as students’ reading habits.students’ reading habits.
• The ANOVAs were run separately for males The ANOVAs were run separately for males and females.and females.
• They yield adjusted least squares means that They yield adjusted least squares means that can be compared to the raw means.can be compared to the raw means.
Impact of “leverage” groups was examined by identifying those Impact of “leverage” groups was examined by identifying those subgroups with the largest positive effect (Incentive 2) and a subgroups with the largest positive effect (Incentive 2) and a large enough sample size to rule out sampling fluctuations.large enough sample size to rule out sampling fluctuations.
(i) Male, White, Absent more than 3 days in the last month(i) Male, White, Absent more than 3 days in the last month [Effect ~3x larger than overall effect for males] [Effect ~3x larger than overall effect for males]
[95/802][95/802] Removing this group would reduce effect of Incentive 2 Removing this group would reduce effect of Incentive 2 by ~25%.by ~25%.
(ii) Female, Hispanic, Not ELL(ii) Female, Hispanic, Not ELL [Effect ~3x larger than overall effect for females] [Effect ~3x larger than overall effect for females]
[82/919][82/919] Removing this group would reduce effect of Incentive 2Removing this group would reduce effect of Incentive 2
• 1212thth grade NAEP results should be interpreted cautiously grade NAEP results should be interpreted cautiously
• Expansion of 12Expansion of 12thth grade NAEP ought to wait on policy grade NAEP ought to wait on policy action on incentivesaction on incentives
• Measuring reading as NAEP does may be problematic in the Measuring reading as NAEP does may be problematic in the current context current context
• In modifying NAEP cognitive instruments (e.g. for readiness), In modifying NAEP cognitive instruments (e.g. for readiness), the administrative setting should be taken into accountthe administrative setting should be taken into account