Day 6: Non-Experimental & Experimental Design

Day 6: Non-Experimental & Day 6: Non-Experimental & Experimental DesignExperimental Design

Where are the beakers??Where are the beakers??

What kind of research is considered What kind of research is considered the “gold standard” by the Institute of the “gold standard” by the Institute of Education Sciences?Education Sciences?

A.A. DescriptiveDescriptiveB.B. Causal-ComparativeCausal-ComparativeC.C. CorrelationalCorrelationalD.D. ExperimentalExperimental

Why?Why?

Why does most educational Why does most educational research use non-experimental research use non-experimental

designs?designs?

What is the purpose of What is the purpose of non-experimental designs?non-experimental designs?

Causal-Comparative ExampleCausal-Comparative Example

Green & Jaquess (1987) Green & Jaquess (1987) – Interested in the effect of high school Interested in the effect of high school

students’ part-time employment on their students’ part-time employment on their academic achievement.academic achievement.

– Sample: 477 high school juniors who were Sample: 477 high school juniors who were unemployed or employed > 10 hours/wk.unemployed or employed > 10 hours/wk.

Causal-Comparative DesignCausal-Comparative Design

A study in which the researcher attempts to A study in which the researcher attempts to determine the cause, or reason, for pre-determine the cause, or reason, for pre-existing differences in groups of individualsexisting differences in groups of individuals

At least two different groups are compared on At least two different groups are compared on a a dependent variabledependent variable or measure of or measure of performance (called the “effect”) because the performance (called the “effect”) because the independent variableindependent variable (called the “cause”) has (called the “cause”) has already occurred or cannot be manipulated already occurred or cannot be manipulated

Causal-Comparative DesignCausal-Comparative Design

Ex-post factoEx-post facto– Causes studied after they have exerted Causes studied after they have exerted

their effect on another variable.their effect on another variable.

Causal-Comparative DesignCausal-Comparative Design

DrawbacksDrawbacks– Difficult to establish causality based on Difficult to establish causality based on

collected data.collected data.– Unmeasured variables (Unmeasured variables (confoundingconfounding

variables) are always a source of potential variables) are always a source of potential alternative causal explanations.alternative causal explanations.

Some Thought Questions…Some Thought Questions…

Correlational DesignCorrelational Design

Determines whether and to what degree Determines whether and to what degree a relationship exists between two or a relationship exists between two or more quantifiable variables.more quantifiable variables.

Correlational DesignCorrelational Design

The degree of the relationship is The degree of the relationship is expressed as a coefficient of correlationexpressed as a coefficient of correlationExamplesExamples– Relationship between math achievement Relationship between math achievement

and math attitudeand math attitude– Relationship between degree of a school’s Relationship between degree of a school’s

racial diversity and student use of racial diversity and student use of stereotypical languagestereotypical language

– Your topics?Your topics?

Correlation coefficient…Correlation coefficient…

-1.00 +1.00

strong negative strong positive

0.00

no relationship

Advantages of Correlational DesignAdvantages of Correlational Design

Analysis of relationships among a large Analysis of relationships among a large number of variables in a single studynumber of variables in a single studyInformation about the Information about the degreedegree of the of the relationship between the variables being relationship between the variables being studiedstudied

CautionsCautions

A relationship between two variables A relationship between two variables does not mean one causes the other does not mean one causes the other (Think about the reading achievement (Think about the reading achievement and body weight correlations)and body weight correlations)Possibility of low reliability of the Possibility of low reliability of the instruments makes it difficult to identify instruments makes it difficult to identify relationshipsrelationships

CautionsCautions

Lack of variability in scores (e.g. Lack of variability in scores (e.g. everyone scoring very, very low; everyone scoring very, very low; everyone scoring very, very high; etc.) everyone scoring very, very high; etc.) makes it difficult to identify relationshipsmakes it difficult to identify relationshipsLarge sample sizes and/or using many Large sample sizes and/or using many variables can identify significant variables can identify significant relationships for statistical reasons and relationships for statistical reasons and not because the relationships really exist not because the relationships really exist (Avoid (Avoid shotgunshotgun approach) approach)

CautionsCautions

Need to identify your sample to know Need to identify your sample to know what is actually being compared.what is actually being compared.If using predictor variables, time interval If using predictor variables, time interval between collecting the predictor and between collecting the predictor and criterion variable data is important.criterion variable data is important.

Correlational DesignsCorrelational Designs

Guidelines for interpreting the size of Guidelines for interpreting the size of correlation coefficientscorrelation coefficients– Much larger correlations are needed for Much larger correlations are needed for

predictions with individuals than with groupspredictions with individuals than with groupsCrude group predictions can be made with Crude group predictions can be made with correlations as low as .40 to .60correlations as low as .40 to .60Predictions for individuals require Predictions for individuals require correlations above .75correlations above .75

Correlational DesignsCorrelational Designs

Guidelines for interpreting the size of Guidelines for interpreting the size of correlation coefficientscorrelation coefficients– Exploratory studiesExploratory studies

Correlations of .25 to .40 indicate the need Correlations of .25 to .40 indicate the need for further researchfor further researchMuch higher correlations are needed to Much higher correlations are needed to confirm or test hypotheses confirm or test hypotheses

Correlational DesignsCorrelational Designs

Criteria for evaluating correlational studiesCriteria for evaluating correlational studies– Causation should not be inferred from Causation should not be inferred from

correlational studiescorrelational studies– Practical significance should not be confused Practical significance should not be confused

with statistical significancewith statistical significance– The size of the correlation should be The size of the correlation should be

sufficient for the use of the results sufficient for the use of the results (individuals vs groups)(individuals vs groups)

Think…Think…

If you were going to take your action If you were going to take your action research topic, and create a causal-research topic, and create a causal-comparative study, what would it look comparative study, what would it look like?like?

--OR----OR--If you were going to take your action If you were going to take your action research project, and create a research project, and create a correlational study, what would it look correlational study, what would it look like?like?

Experimental DesignExperimental Design

The Gold Standard?The Gold Standard?

To ReviewTo Review

Why is most educational research Why is most educational research comprised of comprised of nonnon-experimental research -experimental research designs?designs?

To ReviewTo Review

What is the purpose of What is the purpose of nonnon-experimental -experimental research?research?

To ReviewTo Review

How does the independent variable How does the independent variable function in function in nonnon-experimental research?-experimental research?

To ReviewTo Review

Can non-experimental research claim Can non-experimental research claim causality?causality?

An exampleAn example

Read the example given in class and in Read the example given in class and in pairs respond to the questionspairs respond to the questions

Experimental ResearchExperimental Research

PurposePurpose– To make To make causalcausal inferences about the relationship inferences about the relationship

between the independent and dependent variablesbetween the independent and dependent variables

CharacteristicsCharacteristics– Direct manipulationDirect manipulation of the independent variable of the independent variable– Control of extraneous variablesControl of extraneous variables

Experimental DesignsExperimental Designs

Single Group Post-testSingle Group Post-testSingle Group Pre-test Post-test Single Group Pre-test Post-test Non-Equivalent Groups Post-testNon-Equivalent Groups Post-testQuasi-Experimental DesignQuasi-Experimental DesignRandomized Post-test onlyRandomized Post-test onlyRandomized Pre-test Post-testRandomized Pre-test Post-testFactorialFactorial

Examples

Experimental ValidityExperimental Validity

Internal validityInternal validity– The extent to which the The extent to which the independent independent

variablevariable, and not other , and not other extraneous extraneous variablesvariables , produced the observed effect on , produced the observed effect on the dependent variablethe dependent variable

External validityExternal validity– The extent to which the results are The extent to which the results are

generalizablegeneralizable

Internal ValidityInternal Validity

Threats that reduce the level of confidence in Threats that reduce the level of confidence in any causal conclusionsany causal conclusionsKey Question: Is this a Key Question: Is this a plausibleplausible threat to the threat to the internal validity of the study?internal validity of the study?

Threats to Internal ValidityThreats to Internal ValidityHistoryHistory– Extraneous events have an effect on the subjects’ Extraneous events have an effect on the subjects’

performance on the dependent variableperformance on the dependent variable– Ex - The crash of the stock market, 9-11, the Ex - The crash of the stock market, 9-11, the

invasion of Iraq, etc.invasion of Iraq, etc.

SelectionSelection– Groups that are initially Groups that are initially notnot equal due to equal due to

differences in the subjects in those groupsdifferences in the subjects in those groups– Ex - Positive and negative attitudes, high and low Ex - Positive and negative attitudes, high and low

achievers, etc.achievers, etc.

Threats to Internal ValidityThreats to Internal Validity

MaturationMaturation– Changes experienced within the subject over Changes experienced within the subject over

timetime

PretestingPretesting– The effect of having taken a pretestThe effect of having taken a pretest

InstrumentationInstrumentation– Poor technical quality (i.e. validity, reliability) Poor technical quality (i.e. validity, reliability)

or changes in instrumentationor changes in instrumentation

Threats to Internal ValidityThreats to Internal ValiditySubject attritionSubject attrition– Differential loss of subjects from groupsDifferential loss of subjects from groups

Statistical regressionStatistical regression– The natural movement of extreme scores toward the The natural movement of extreme scores toward the

meanmean

Diffusion of treatmentDiffusion of treatment– The treatment is given to the control groupThe treatment is given to the control group

Experimenter effectsExperimenter effects– Different characteristics or expectations of those Different characteristics or expectations of those

implementing the treatments across groupsimplementing the treatments across groups

Threats to Internal ValidityThreats to Internal Validity

Subject effectsSubject effects– The effects of being aware that one is The effects of being aware that one is

involved in a studyinvolved in a study– Four typesFour types

Hawthorne effectHawthorne effectJohn Henry effectJohn Henry effectResentful demoralizationResentful demoralizationNovelty effectNovelty effect

Internal ValidityInternal Validity

Key Point: Ultimately, validity is a matter Key Point: Ultimately, validity is a matter of judgment. Ask if it is of judgment. Ask if it is reasonablereasonable that that possiblepossible threats are threats are likelylikely to affect the to affect the results.results.

External Validity

The extent to which results can be generalized from a sample to a particular population.Question – Why would really good internal validity often result in poor external validity?

External ValidityExternal Validity

Factors affecting external validityFactors affecting external validity– SubjectsSubjects

Representativeness of the sample in Representativeness of the sample in comparison to the populationcomparison to the populationPersonal characteristics of the subjects Personal characteristics of the subjects

– Situations - characteristics of the settingSituations - characteristics of the settingSpecific environmentSpecific environmentSpecial situationSpecial situationParticular schoolParticular school

External ValidityExternal Validity

Importance of explanation of sampling Importance of explanation of sampling proceduresprocedures

Experimental DesignsExperimental DesignsSingle Group Post-testSingle Group Post-testSingle Group Pre-test Post-test – Single Group Pre-test Post-test – Libby, DebLibby, Deb

Non-Equivalent Groups Post-test – Non-Equivalent Groups Post-test – Mary, CherylMary, Cheryl

Quasi-Experimental Design – Quasi-Experimental Design – Pete, LauraPete, Laura

Randomized Post-test only –Randomized Post-test only – Amanda, Nicole, Tam Amanda, Nicole, Tam

Randomized Pre-test Post-test – Randomized Pre-test Post-test – Karen, Jen, Justin Karen, Jen, Justin

Examples

Your TaskYour Task

Based on the topic of your proposal, Based on the topic of your proposal, design an experimental study using the design an experimental study using the design you were assigned.design you were assigned.– Write a research question and hypothesis.Write a research question and hypothesis.– Sketch out the methods.Sketch out the methods.

Identify strengths and weaknesses of Identify strengths and weaknesses of each design.each design.

Experimental DesignsExperimental Designs

NotationNotation– RR indicates random selection or random indicates random selection or random

assignmentassignment– OO indicates an observation indicates an observation

TestTestObservation scoreObservation scoreScale scoreScale score

– XX indicates a treatment indicates a treatment– AA,, B B, , CC, ... indicates a group, ... indicates a group

Pre-Experimental DesignsPre-Experimental Designs

No pre-experimental design controls internal No pre-experimental design controls internal validity threats well validity threats well Single group pretest onlySingle group pretest only– A X OA X O– Internal validity threatsInternal validity threats

History, maturation, attrition, experimenter effects, subject History, maturation, attrition, experimenter effects, subject effects, and instrumentation are viable threatseffects, and instrumentation are viable threatsUseful only when the research is sure of the status of the Useful only when the research is sure of the status of the knowledge, skill, or attitude being changed knowledge, skill, or attitude being changed andand there are there are no extraneous variables affecting the resultsno extraneous variables affecting the results

Pre-Experimental DesignsPre-Experimental Designs

Single group pretest post-testSingle group pretest post-test– A O X OA O X O– Internal validity threatsInternal validity threats

Maturation and pretesting are threatsMaturation and pretesting are threatsHistory and instrumentation are potential threatsHistory and instrumentation are potential threats

– Useful when subject effects will not influence the Useful when subject effects will not influence the results, history effects can be minimized, and results, history effects can be minimized, and multiple pretests and post-tests are usedmultiple pretests and post-tests are used

Pre-Experimental DesignsPre-Experimental Designs

Non-equivalent groups post-test onlyNon-equivalent groups post-test only– A X O A X O

B OB O– Internal validity threatsInternal validity threats

Definite Threat: Selection Definite Threat: Selection Potential Threats: History, maturation, and Potential Threats: History, maturation, and instrumentationinstrumentation

– Useful when groups are comparable and subjects Useful when groups are comparable and subjects can be assumed to be about the same at the can be assumed to be about the same at the beginning of the studybeginning of the study

Quasi-Experimental DesignsQuasi-Experimental DesignsTypesTypes– Non-equivalent pretest/post-test, experimental Non-equivalent pretest/post-test, experimental

control groupscontrol groupsA O X O A O X O B O OB O O

– Non-equivalent pretest/post-test, multiple treatment Non-equivalent pretest/post-test, multiple treatment groupsgroups

A O XA O X11 O O B O XB O X22 O O

Useful when subjects are in pre-existing Useful when subjects are in pre-existing groups (e.g. classes, schools, teams, etc.)groups (e.g. classes, schools, teams, etc.)

Quasi-Experimental DesignsQuasi-Experimental Designs

Threats to internal validityThreats to internal validity– Selection is the major concernSelection is the major concern– Likely to control for most other threats, Likely to control for most other threats,

provided the groups are not significantly provided the groups are not significantly different from one anotherdifferent from one another

– See Table 9.2 for specific threats related to See Table 9.2 for specific threats related to each designeach design

True Experimental DesignsTrue Experimental Designs

Important terminologyImportant terminology– Random assignmentRandom assignment

Subjects placed into groups by randomSubjects placed into groups by randomEnsures equivalency of the groupsEnsures equivalency of the groups

– Random selection of subjectsRandom selection of subjectsSubjects chosen from population by randomSubjects chosen from population by randomEnsures generalizability to the population from Ensures generalizability to the population from which the subjects were selected (i.e. external which the subjects were selected (i.e. external validity)validity)

True Experimental DesignsTrue Experimental DesignsTypesTypes– Randomized post-test only experimental control Randomized post-test only experimental control

groupsgroupsR A X O R A X O R B OR B O

– Randomized post-test only multiple treatment Randomized post-test only multiple treatment groupsgroups

R A XR A X11 O O R B XR B X22 O O

True Experimental DesignsTrue Experimental Designs

Types (continued)Types (continued)– Randomized pretest/post-test multiple Randomized pretest/post-test multiple

treatment groupstreatment groupsR A O XR A O X11 O O R B O X R B O X22 O O

– Randomized pretest/post-test experimental Randomized pretest/post-test experimental control groupscontrol groups

R A O X O R A O X O R B O O R B O O

True Experimental DesignsTrue Experimental Designs

Threats to internal validityThreats to internal validity– Controls for selection, maturation, and Controls for selection, maturation, and

statistical regressionstatistical regression– Likely to control for most other threatsLikely to control for most other threats– See Table 9.2 for specific threats related to See Table 9.2 for specific threats related to

each designeach design

Evaluating Experimental Evaluating Experimental DesignsDesigns

Criteria for evaluating experimental Criteria for evaluating experimental researchresearch– The primary purpose is to test causal The primary purpose is to test causal

hypotheseshypotheses– There should be direct manipulation of the There should be direct manipulation of the

independent variableindependent variable– There should be clear identification of the There should be clear identification of the

specific research designspecific research design

Evaluating Experimental Evaluating Experimental DesignsDesigns

Criteria for evaluating experimental Criteria for evaluating experimental researchresearch– The design should provide maximum The design should provide maximum

control of extraneous variablescontrol of extraneous variables– Treatments are substantively different from Treatments are substantively different from

one anotherone another– The number of subjects is dependent on or The number of subjects is dependent on or

equal to the number of treatment equal to the number of treatment replicationsreplications