Research design

Research DesignAndrew Martin

PS 372 -- University of Kentucky

What is research design?

•Research design is a plan that shows how a researcher intends to study an empirical question.

•Choice of research design depends on a number of factors.

Research Designs should ...

(1) Establish a relationship between two or more variables. (Comparison)

(2) Demonstrate the results are generally true in the real world. (Generalizability)

(3) Reveal whether one phenomenon precedes another in time. (Manipulation)

(4) Eliminate as many alternative explanations as possible. (Control)

Causal vs. Spurious

Relationships•A causal relationship is a connection between two variables that occurs because one produces, or brings about, the other with complete or great regularity.

•The independent variable is the cause, the dependent variable is the effect.

Causal vs. Spurious

Relationships•A spurious relationship occurs

when the relationship between two variables appears to be valid but is actually explained by variables other than those stated in the hypothesis.

•JRM 124

Downs Voter Paradox (1957)

•In An Economic Theory of Democracy, Anthony Downs concludes that voting is an irrational act in most instances.

•Reward = Probability your vote changes election outcome * benefits - costs

•R = PB - C

Research Design and Causal Inferences

•To explain why phenomena are connected, we must know why they are connected, not simply that they are associated.

Riker and Ordeshook (1968)•Reformulated model, suggesting

Downs’ model omitted an important component of an individual’s choice to vote, the intangible benefits.

•Thus, the model becomes: R = PB - C + D

•D represents non-instrumental or expressive benefits.

Research Design and Causal Inferences•Covariation. The research must

demonstrate that the alleged cause (X) does in fact covary with the supposed effect (Y).

•Time order. The research must show that the cause preceded the effect.

•Eliminates alternatives. All possible joint causes of X and Y must be eliminated.

•JRM 125

•JRM 126

Research Designs•Four major approaches in the

social sciences:

•Experimental design

•Quasi-experimental design

•Cross-sectional design

•Pre-Experimental design

Experiments

•Experiments allow the researcher to control exposure to an experimental variable (or independent variable).

•Theoretically, this allows researchers to make causal inferences with greater confidence than non-experimental approaches.

Classical Experimental

Design•Classical experimental design

involves an experiment with the random assignment of subjects to experimental and control groups with a pre-test and a post-test.

Classical Experimental

Design•Establishes an experimental group

and a control group.

•Each subject is randomly assigned to one group or the other.

•The researcher controls the administration or introduction of the experimental treatment.

Classical Experimental

Design•The researcher establishes and measures a dependent variable both before and after the experimental variable is introduced.

•The experimenter controls the environment -- that is, the time, location and other physical aspects of the of the experiment.

•JRM 129

•NN 91

Internal Validity

Internal validity constitutes the ability to show that manipulation or variation of the independent variable actually causes the dependent variable to change.

Internal Validity

•Internal validity is directly related to the control component of research design.

• Random assignment of participants helps guard against this, but sometimes this is not ethical or practical.

•Extrinsic factors can sometimes produce differences between experimental and control groups.

Internal Validity•Extrinsic factors produce

differences in the two groups prior to the research operation.

•When this happens, it is difficult to distinguish selection effects from effects of the independent variable.

•This causes selection bias, and it threatens a study’s internal validity.

Internal Validity

•Intrinsic factors include changes to individuals or units studies during the study period.

Intrinsic Factors•History

•Maturation

•Experiment mortality

•Instrumentation

•Testing

•Regression artifact

•Interactions with selection

Intrinsic FactorsNachmias-Nachmias (2000)

•History refers to all events that occurred during the time of the study that might affect the individuals studied and provide a rival explanation for the change in the dependent variable.

•Ex: Financial crisis in government during the time of the experiment.

Intrinsic FactorsNachmias-Nachmias (2000)

•Maturation involves biological, psychological or social processes that produce changes in individuals or units studied with the passage of time.

•Ex: Test scores and teaching methods. Perhaps the students just became older and wiser independently of teaching method.

Intrinsic FactorsNachmias-Nachmias (2000)

•Experimental mortality refers to dropout problems that prevent the researcher from obtaining complete information on all cases.

•Ex: Studying the effect of media on prejudice, and most dropouts were prejudiced individuals.

Intrinsic FactorsNachmias-Nachmias (2000)

•Instrumentation designates changes in the measuring instruments between the pre-test and post-test.

•Ex: Changing criteria for evaluating psychological tests.

Intrinsic FactorsNachmias-Nachmias (2000)

•Testing the possible reactivity of measurement is a major problem in social science research. The process of testing may change the phenomena being measured.

•Ex: People aware of the purpose of the survey; taking intelligence tests often

Intrinsic FactorsNachmias-Nachmias (2000)

•Sometimes selection factors and intrinsic factors interact.

•The most common are selection-history and selection-maturation.

Procedures of Control

Nachmias-Nachmias (2000)

•Matching

•Randomization

•The Control Group

MatchingNachmias-Nachmias (2000)

•Matching is a way of equating the experimental and control groups on extrinsic factors known to be related to the research project.

MatchingNachmias-Nachmias (2000)

•Precision matching, for each case in the group, another case with identical characteristics is selected for the control group.

•NN 98

MatchingNachmias-Nachmias (2000)

•Frequency distribution looks at central characteristics of the two groups rather than on-on-one matching.

•Frequency distribution is a more efficient alternative.

•NN 99

RandomizationNachmias-Nachmias (2000)•Randomization is a method of

control that helps to offset the confounding effects of known and unknown factors by randomly assigning cases to experimental and control groups.

•Can be accomplished by flipping a coin, using a table of random digits or any other method that ensure each case has an equal probability of being chosen.

External ValidityNachmias-Nachmias (2000)

•External validity is the extent to which the research findings can be generalized to larger populations and applied to different settings.

•The two main components of external validity are representativeness of the sample and reactive arrangements in the research process.

External ValidityNachmias-Nachmias (2000)

•Reactive arrangements can compromise the experimental setting or situation if they do not reflect the natural setting or situation about which researchers wish to generalize.

•The way the study is conducted could impact its outcome. Ex: Surveys, subject response and question wording.