1 RESEARCH DESIGN RESEARCH DESIGN A Plan for Research A Plan for Research A RESEARCH DESIGN A RESEARCH DESIGN ■ A Plan of Action = A Plan of Action = Design LOGIC & Design LOGIC & A detailed plan for A detailed plan for data collection and data collection and analysis: “Everything analysis: “Everything but cell entries in the but cell entries in the tables!” tables!”
30
Embed
A RESEARCH DESIGN - Indiana University Bloomingtoneducy520/sec6342/week_05/experimental_design… · RESEARCH DESIGN A Plan for Research A RESEARCH DESIGN ... What do I want to explain?
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
1
RESEARCH DESIGNRESEARCH DESIGN
A Plan for ResearchA Plan for Research
A RESEARCH DESIGNA RESEARCH DESIGN
■■ A Plan of Action =A Plan of Action =
Design LOGIC &Design LOGIC &
A detailed plan for A detailed plan for data collection and data collection and analysis: “Everything analysis: “Everything but cell entries in the but cell entries in the tables!” tables!”
2
Choosing a Research Design: Choosing a Research Design: LOGICAL considerationsLOGICAL considerations
■■ What type of question am I asking?What type of question am I asking?–– Purpose?Purpose?
–– Exploratory (a case study)Exploratory (a case study)
■■ changes associated changes associated with prewith pre--test”test”
■■
■■ SOLUTIONSOLUTION
Solutions to avoid TEST Solutions to avoid TEST EFFECTSEFFECTS
■■ No preNo pre--testtest
■■ Solomon 4 Group Solomon 4 Group Experimental Experimental DesignDesign
■■ Use Unobtrusive Use Unobtrusive measurementmeasurement
16
THREATS to INTERNAL THREATS to INTERNAL VALIDITYVALIDITY
■■ ““TEST effects” onlyTEST effects” only
■■ 1 of 10 THREATS1 of 10 THREATS
■■ INTERNAL INTERNAL VALIDITY =VALIDITY =
■■ confounding confounding influences influences associated with associated with experimental experimental (research) (research) conditionsconditions
VERY important RULEVERY important RULE
■■ Evaluating ACCURACY of resultsEvaluating ACCURACY of results
■■ You must be sure there were NO You must be sure there were NO THREATS to INTERNAL VALIDITY THREATS to INTERNAL VALIDITY before you can generalize to other before you can generalize to other contextscontexts
■■ (EXTERNAL VALIDITY)(EXTERNAL VALIDITY)
17
THREATS to INTERNAL THREATS to INTERNAL VALIDITY (10)VALIDITY (10)
■■ No.1: Differences in No.1: Differences in Characteristics Characteristics SUBJECTSSUBJECTS
■■ SOLUTION???SOLUTION???
Threat to Internal Validity No. 1:Threat to Internal Validity No. 1:Differences in Differences in SSs s Characteristics Characteristics
■■ SOLUTION???SOLUTION???
■■ RANDOMIZERANDOMIZE
18
Threat to Internal Validity No. 2Threat to Internal Validity No. 2
■■ HISTORY HISTORY EFFECTS:EFFECTS:
■■ events or events or confounding confounding influences occurring influences occurring between prebetween pre-- and and postpost-- testtest
■■ (changes in (changes in environment)environment)
SOLUTION???SOLUTION???
■■ To Avoid HISTORY EFFECTS???To Avoid HISTORY EFFECTS???
19
Threat to INTERNAL Threat to INTERNAL VALIDITY No. 3VALIDITY No. 3
■■ MATURATION:MATURATION:
changes in subjects changes in subjects between prebetween pre-- and and postpost--teststests
■■ SOLUTION???SOLUTION???
Threat to INTERNAL Threat to INTERNAL VALIDITY No. 4VALIDITY No. 4
■■ TEST EFFECTS:TEST EFFECTS:
changes associated changes associated with prewith pre--testtest
■■ SOLUTION???SOLUTION???
20
THREAT to INTERNAL THREAT to INTERNAL VALIDITY No. 5VALIDITY No. 5
■■ INSTRUMENTATION:INSTRUMENTATION:
Changes associated Changes associated with change or decay in with change or decay in test instrumentstest instruments
■■ SOLUTIONS???SOLUTIONS???
THREAT to INTERNAL THREAT to INTERNAL VALIDITY NO. 6 VALIDITY NO. 6
■■ SUBJECT SUBJECT MORTALITY:MORTALITY:
Loss of subjectsLoss of subjects
■■ SOLUTIONS???SOLUTIONS???
21
Threat to INTERNAL Threat to INTERNAL VALIDITY NO. 7VALIDITY NO. 7
■■ REGRESSION REGRESSION towards the meantowards the mean
■■ unreliable measures unreliable measures (values) due to (values) due to choice of extreme choice of extreme values for subjectsvalues for subjects
■■ SOLUTIONSOLUTION
■■ Scores on testScores on test
■■ ff
■■ 100% x Es#1100% x Es#1
■■ xxxxxxxx
■■ 50% 50% xxxxxxxxxx
■■ xxxxxxxxxxxxxxxx
■■ 0% 0% xxxxxxxxxxxxxxxxxxxxxxxx
SOLUTION: RandomizeSOLUTION: Randomize
■■ Best way to Best way to minimize “atypical minimize “atypical values”values”
■■ Can calculate “odds”Can calculate “odds”or “p value” of or “p value” of getting an extreme getting an extreme scorescore
22
Another SolutionAnother Solution
■■ REPLICATION:REPLICATION:
Repeat same study Repeat same study with AT LEAST ONE with AT LEAST ONE new wrinklenew wrinkle
■■ If sample If sample representative representative should get same should get same resultsresults
THREAT to INTERNAL THREAT to INTERNAL VALIDITY NO. 8VALIDITY NO. 8
■■ REACTIVITY REACTIVITY (HAWTHORNE(HAWTHORNE
Effect):Effect):
■■ artificial change artificial change produced by produced by measurement measurement processprocess
■■ SOLUTION???SOLUTION???
23
THREAT to INTERNAL THREAT to INTERNAL VALDITY NO. 9VALDITY NO. 9
■■ EXPERIMENTEREXPERIMENTER
BIASBIAS
■■ SOLUTION???SOLUTION???
INTERNAL THREAT to INTERNAL THREAT to VALIDITY No. 10VALIDITY No. 10
■■ INTERACTIONINTERACTION
EFFECTS:EFFECTS:
■■ Effect caused by Effect caused by any combination of any combination of the above threatsthe above threats
■■ SOLUTION?????? SOLUTION??????
24
INTERACTION effectsINTERACTION effects
■■ Is the “WORST” Is the “WORST” threat to validity:threat to validity:
■■ can’t controlcan’t control
■■ can’t always predictcan’t always predict
■■ must try to avoidmust try to avoid
Threats to EXTERNAL Threats to EXTERNAL VALIDITYVALIDITY
■■ ““GeneralizabilityGeneralizability”...”...
■■ To what extent will To what extent will your findings apply your findings apply to other to other contexts???? contexts????
25
How Representative was How Representative was experiment?experiment?
■■ CRITERIA: JudgmentalCRITERIA: Judgmental
■■ Are SUBJECTS representative?Are SUBJECTS representative?
■■ Is experimental SETTING representative?Is experimental SETTING representative?
■■ Are MANIPULATIONS &MEASURES of X & Are MANIPULATIONS &MEASURES of X & Y representative of all Xs &Y representative of all Xs &YsYs??
■■ Are there UNIQUE circumstances Are there UNIQUE circumstances associated with the EXPERIMENTAL associated with the EXPERIMENTAL CONDITION NOT present in other contexts?CONDITION NOT present in other contexts?
GENERALIZABILITY of GENERALIZABILITY of resultsresults
■■ Internal/External Internal/External Validity are often … Validity are often …
■■ a subjective a subjective judgment… Are you judgment… Are you measure what you measure what you claim to be claim to be measuring???measuring???
26
EXTERNAL VALIDITYEXTERNAL VALIDITY
■■ To what extent can To what extent can your findings be your findings be applied to other applied to other contexts?contexts?
Choose Data Collection Choose Data Collection Method and Analytical Method and Analytical
RoutinesRoutines■■ Data Collection Data Collection
Methods:Methods:
■■ 1+ Case studies1+ Case studies
■■ ExperimentExperiment
■■ Surveys (1 time pt)Surveys (1 time pt)
■■ Content analysisContent analysis
■■ Aggregate data Aggregate data analysis analysis
How detailed should I get in How detailed should I get in writing a Research Design?writing a Research Design?
■■ Provide enough Provide enough details so SOME details so SOME ONE ELSE could ONE ELSE could “execute it”...collect “execute it”...collect the data and the data and analyze it according analyze it according to your plan”to your plan”
■■ “All but the cells in “All but the cells in the table”the table”
30
Congratulations!Congratulations!
■■ The hard creative The hard creative work is overwork is over
■■ on to ........on to ........
■■ Implementation of Implementation of your RESEARCHyour RESEARCH