Assisting GPRA Report for MSP Xiaodong Zhang, Westat MSP Regional Conference Miami, January 7-9, 2008.
Post on 01-Jan-2016
221 Views
Preview:
Transcript
Assisting GPRA Report Assisting GPRA Report for MSPfor MSP
Xiaodong Zhang, WestatXiaodong Zhang, Westat
MSP Regional ConferenceMSP Regional Conference
Miami, January 7-9, 2008Miami, January 7-9, 2008
GPRAGPRA
GPRA measuresGPRA measuresGPRA Measure—GPRA Measure—the percentage of MSP the percentage of MSP teachers who significantly increase their content teachers who significantly increase their content knowledge, as reflected in project-level pre- and knowledge, as reflected in project-level pre- and post-assessmentspost-assessments
GPRA Measure—GPRA Measure—the percentage of MSP the percentage of MSP projects that use an experimental or quasi-projects that use an experimental or quasi-experimental design for their evaluations that are experimental design for their evaluations that are conducted successfully and that yield conducted successfully and that yield scientifically valid resultsscientifically valid results
DQI and GPRADQI and GPRA
Data Quality Initiative: Data Quality Initiative: technical assistance technical assistance to ED grant programs to improve the quality to ED grant programs to improve the quality of information on program performanceof information on program performance
GPRA Measure: MSP Teacher Content GPRA Measure: MSP Teacher Content Knowledge (MSPTCK) softwareKnowledge (MSPTCK) software
GPRA Measure: Evaluation rubricGPRA Measure: Evaluation rubric
MSPTCK objectivesMSPTCK objectives
Designed to assist grantees to determine the number of Designed to assist grantees to determine the number of teachers who have made significant gains in content teachers who have made significant gains in content knowledge (APR Q8b-f)knowledge (APR Q8b-f)
MathematicsMathematics b) Number of teachers with both pretest and posttest in math b) Number of teachers with both pretest and posttest in math
content knowledgecontent knowledge c) Number of teachers who showed significant gains in math c) Number of teachers who showed significant gains in math
content knowledgecontent knowledge
ScienceScience e) Number of teachers with both pretest and posttest in science e) Number of teachers with both pretest and posttest in science
content knowledgecontent knowledge f)f) Number of teachers who showed significant gains in science Number of teachers who showed significant gains in science
content knowledgecontent knowledge
MSPTCK processMSPTCK process
Two-step processTwo-step process
Step 1. Determine if overall posttest scores are Step 1. Determine if overall posttest scores are significantly higher than pretest (paired-samples significantly higher than pretest (paired-samples t-test)t-test)
Step 2. Determine the number of teachers in the Step 2. Determine the number of teachers in the tested group who made significant gainstested group who made significant gains If more than one test used for different groups, run If more than one test used for different groups, run
application separately for each testapplication separately for each test If more than one test used for the same group, select If more than one test used for the same group, select
the most relevant onethe most relevant one
MSPTCK tutorialMSPTCK tutorial
Access the spreadsheet onlineAccess the spreadsheet online
MSPTCK tutorialMSPTCK tutorial
MSPTCK tutorialMSPTCK tutorial
Enter data hereEnter data here tab tab Blank rows not allowed Blank rows not allowed
between databetween data
Import data from Import data from another Excel fileanother Excel file
Perform range check Perform range check
MSPTCK tutorialMSPTCK tutorial
MSP tutorialMSP tutorial
View results here View results here tabtab
MSPTCK tutorialMSPTCK tutorial
Information Information tabtab
Evaluation rubricEvaluation rubric
The criteria on the rubric are the minimum The criteria on the rubric are the minimum criteria that need to be met for an criteria that need to be met for an evaluation to have been successfully evaluation to have been successfully conducted and yield valid data conducted and yield valid data
An evaluation has to meet each criterion in An evaluation has to meet each criterion in order to meet the GPRA measureorder to meet the GPRA measure
Rubric: designsRubric: designs
Experimental study—Experimental study—the study measures the intervention’s effect the study measures the intervention’s effect by randomly assigning individuals (or other units, such as by randomly assigning individuals (or other units, such as classrooms or schools) to a group that participated in the classrooms or schools) to a group that participated in the intervention, or to a control group that did not, and then compares intervention, or to a control group that did not, and then compares post-intervention outcomes for the two groupspost-intervention outcomes for the two groups
Quasi-experimental study—Quasi-experimental study—the study measures the intervention’s the study measures the intervention’s effect by comparing post-intervention outcomes for treatment effect by comparing post-intervention outcomes for treatment participants with outcomes for a comparison group (that was not participants with outcomes for a comparison group (that was not exposed to the intervention), chosen through methods other than exposed to the intervention), chosen through methods other than random assignment. For example:random assignment. For example:
Comparison-group study with equatingComparison-group study with equating—a study in which statistical —a study in which statistical controls and/or matching techniques are used to make the treatment controls and/or matching techniques are used to make the treatment and comparison groups similar in their pre-intervention characteristicsand comparison groups similar in their pre-intervention characteristics
Rubric: designsRubric: designs Regression-discontinuity studyRegression-discontinuity study—a study in which individuals (or —a study in which individuals (or
other units, such as classrooms or schools) are assigned to other units, such as classrooms or schools) are assigned to treatment or comparison groups on the basis of atreatment or comparison groups on the basis of a “cutoff” score “cutoff” score on a pre-intervention non-dichotomous measureon a pre-intervention non-dichotomous measure
Other studyOther study——The study uses a design other than a The study uses a design other than a randomized controlled trial, comparison-group study with randomized controlled trial, comparison-group study with equating, or regression-discontinuity study, including equating, or regression-discontinuity study, including pre-post pre-post studies, whichstudies, which measure the intervention’s effect measure the intervention’s effect based on the pre-test to post-test differences of a single based on the pre-test to post-test differences of a single group, and comparison-group studies without equating, group, and comparison-group studies without equating, or non-experimental studies that compare outcomes of or non-experimental studies that compare outcomes of groups that vary with respect to implementation fidelity or groups that vary with respect to implementation fidelity or program dosage program dosage
Rubric: aspectsRubric: aspects
For Experimental Designs:For Experimental Designs:Sample SizeSample SizeQuality of Measurement InstrumentsQuality of Measurement InstrumentsQuality of Data Collection MethodsQuality of Data Collection MethodsAttrition Rates, Response RatesAttrition Rates, Response RatesRelevant Statistics ReportedRelevant Statistics Reported
For Quasi-experimental Designs:For Quasi-experimental Designs:All of the above, plusAll of the above, plus
Baseline Equivalence of GroupsBaseline Equivalence of Groups
Rubric: sample sizeRubric: sample size
Met the criterionMet the criterion—sample size was adequate —sample size was adequate (i.e., based on power analysis with (i.e., based on power analysis with recommended significance level=0.05, recommended significance level=0.05, power=0.8, and a minimum detectable effect power=0.8, and a minimum detectable effect informed by the literature or otherwise justified) informed by the literature or otherwise justified)
Did not meet the criterionDid not meet the criterion —the sample size —the sample size was too small was too small
Did not address the criterion Did not address the criterion
Rubric: quality of measurement Rubric: quality of measurement instrumentsinstruments
Met the criterionMet the criterion—the study used existing —the study used existing data collection instruments that had already data collection instruments that had already been deemed valid and reliable to measure been deemed valid and reliable to measure key outcomes, or data collection instruments key outcomes, or data collection instruments developed specifically for the study were developed specifically for the study were sufficiently pre-tested with subjects who were sufficiently pre-tested with subjects who were comparable to the study samplecomparable to the study sample
Did not meet the criterionDid not meet the criterion—the key data —the key data collection instruments used in the evaluation collection instruments used in the evaluation lacked evidence of validity and reliability lacked evidence of validity and reliability
Did not address the criterionDid not address the criterion
Rubric: quality of data collection Rubric: quality of data collection methodsmethods
Met the criterionMet the criterion—the methods, procedures, —the methods, procedures, and timeframes used to collect the key and timeframes used to collect the key outcome data from treatment and control outcome data from treatment and control groups groups were the samewere the same
Did not meet the Did not meet the criterioncriterion—instruments/assessments were —instruments/assessments were administered differently in manner and/or at administered differently in manner and/or at different times to treatment and control group different times to treatment and control group participantsparticipants
Rubric: data reduction ratesRubric: data reduction rates
Met the criterion—Met the criterion—(1) the study measured the key outcome (1) the study measured the key outcome variable(s) in the post-tests for at least 70% of the original study variable(s) in the post-tests for at least 70% of the original study sample (treatment and control groups combined), or there is sample (treatment and control groups combined), or there is evidence that the high rates of data reduction were unrelated to evidence that the high rates of data reduction were unrelated to the intervention, the intervention, ANDAND (2) the proportion of the original study (2) the proportion of the original study sample that was retained in follow-up data collection activities sample that was retained in follow-up data collection activities (e.g., post-intervention surveys) and/or for whom post-intervention (e.g., post-intervention surveys) and/or for whom post-intervention data were provided (e.g., test scores) was similar for both the data were provided (e.g., test scores) was similar for both the treatment and control groups (i.e., less or equal to a 15-percent treatment and control groups (i.e., less or equal to a 15-percent difference), or the proportion of the original study sample that was difference), or the proportion of the original study sample that was retained in the follow-up data collection was different for the retained in the follow-up data collection was different for the treatment and control groups, but sufficient steps were taken to treatment and control groups, but sufficient steps were taken to address this differential attrition in the statistical analysisaddress this differential attrition in the statistical analysis
Rubric: data reduction ratesRubric: data reduction rates
Did not meet the criterion—Did not meet the criterion—(1) the study failed to (1) the study failed to measure the key outcome variable(s) in the post-tests for measure the key outcome variable(s) in the post-tests for 30% or more of the original study sample (treatment and 30% or more of the original study sample (treatment and control groups combined), and there is no evidence that control groups combined), and there is no evidence that the high rates of data reduction were unrelated to the the high rates of data reduction were unrelated to the intervention; intervention; OROR (2) the proportion of study participants (2) the proportion of study participants who participated in follow-up data collection activities who participated in follow-up data collection activities (e.g., post-intervention surveys) and/or for whom post-(e.g., post-intervention surveys) and/or for whom post-intervention data were provided (e.g., test scores) was intervention data were provided (e.g., test scores) was significantly different for the treatment and control groups significantly different for the treatment and control groups (i.e., more than a 15-percent difference) and sufficient (i.e., more than a 15-percent difference) and sufficient steps to address differential attrition were not taken in steps to address differential attrition were not taken in the statistical analysisthe statistical analysis
Did not address the criterionDid not address the criterion
Rubric: relevant statistics reportedRubric: relevant statistics reported
Met the criterion—Met the criterion—the final report includes treatment the final report includes treatment and control group post-test means and tests of and control group post-test means and tests of statistical significancestatistical significance for key outcomes or provides for key outcomes or provides sufficient information for calculation of statistical sufficient information for calculation of statistical significance (e.g., mean, sample size, standard significance (e.g., mean, sample size, standard deviation/standard error)deviation/standard error)Did not meet the criterion—Did not meet the criterion—the final report does not the final report does not include treatment and control group post-test means include treatment and control group post-test means and/or tests of statistical significance for key outcomes and/or tests of statistical significance for key outcomes or provide sufficient information for calculation of or provide sufficient information for calculation of statistical significance (e.g., mean, sample size, statistical significance (e.g., mean, sample size, standard deviation/standard error) standard deviation/standard error) Did not address the criterionDid not address the criterion
Rubric: baseline equivalence of Rubric: baseline equivalence of groups for QEDgroups for QED
Met the criterion—Met the criterion—there were no significant pre-there were no significant pre-intervention differences between treatment and intervention differences between treatment and comparison group participants on variables related to comparison group participants on variables related to the study’s key outcomes, or adequate steps were the study’s key outcomes, or adequate steps were taken to address the lack of baseline equivalence in taken to address the lack of baseline equivalence in the statistical analysisthe statistical analysis
Did not meet the criterion—Did not meet the criterion—there were statistically there were statistically significant pre-intervention differences between significant pre-intervention differences between treatment and comparison group participants on treatment and comparison group participants on variables related to the study’s key outcomes, and no variables related to the study’s key outcomes, and no steps were taken to address lack of baseline steps were taken to address lack of baseline equivalence in the statistical analysisequivalence in the statistical analysis
Did not address the criterionDid not address the criterion
DisclaimerDisclaimer
The instructional practices and assessments The instructional practices and assessments discussed or shown in this presentation are discussed or shown in this presentation are not intended as an endorsement by the U.S. not intended as an endorsement by the U.S. Department of Education. Department of Education.
top related